[Fuego] Error in running the test after latest pull
Bird, Tim
Tim.Bird at sony.com
Tue Dec 15 23:23:21 UTC 2020
This looks like it was caused by a server error ( http error 500), but ftc should have handled
this. I made a modification to ftc to handle this, and pushed the commit to the master branch.
Let me know if you see this again.
-- Tim
> -----Original Message-----
> From: Pooja Sanjay More <pooja.sm at pathpartnertech.com>
>
> Hi Tim,
>
> Issues got fixed with the suggested changes and with recent pull. We tested a case by logging in a request in
>
> fserver for a test to be run on rpi. This test also went smooth, but the results could not be
>
> uploaded to the server due to the fserver issue.
>
> Please find the logs below:
>
> ftc run-request --put-run
> Trying to get request 'request-2020-12-10_05:36:12.67-himanshu_wks:rpi' from server
> TRB: best_local_match=('', u'Benchmark.Dhrystone', '1.1-1')
> TRB: best_remote_match=None
> TRB: candidate=('', u'Benchmark.Dhrystone', '1.1-1')
> Executing test Benchmark.Dhrystone on board rpi (using spec default)
> Running test 'Benchmark.Dhrystone' on board 'rpi' using spec 'default'
> ===== doing fuego phase: pre_test =====
> ===== doing fuego phase: build ========
> The test is already built
> Fuego test_build duration=0 seconds
> ===== doing fuego phase: deploy =====
> ===== doing fuego phase: snapshot =====
> Firmware revision: 5.4.81-v7-40+
> ===== doing fuego phase: run =====
> -------------------------------------------------
>
> Dhrystone Benchmark, Version 2.1 (Language: C)
>
> Program compiled without 'register' attribute
>
> Please give the number of runs through the benchmark:
> Execution starts, 10000000 runs through Dhrystone
> Execution ends
>
> Final values of the variables used in the benchmark:
>
> Int_Glob: 5
> should be: 5
> Bool_Glob: 1
> should be: 1
> Ch_1_Glob: A
> should be: A
> Ch_2_Glob: B
> should be: B
> Arr_1_Glob[8]: 7
> should be: 7
> Arr_2_Glob[8][7]: 10000010
> should be: Number_Of_Runs + 10
> Ptr_Glob->
> Ptr_Comp: 10326352
> should be: (implementation-dependent)
> Discr: 0
> should be: 0
> Enum_Comp: 2
> should be: 2
> Int_Comp: 17
> should be: 17
> Str_Comp: DHRYSTONE PROGRAM, SOME STRING
> should be: DHRYSTONE PROGRAM, SOME STRING
> Next_Ptr_Glob->
> Ptr_Comp: 10326352
> should be: (implementation-dependent), same as above
> Discr: 0
> should be: 0
> Enum_Comp: 1
> should be: 1
> Int_Comp: 18
> should be: 18
> Str_Comp: DHRYSTONE PROGRAM, SOME STRING
> should be: DHRYSTONE PROGRAM, SOME STRING
> Int_1_Loc: 5
> should be: 5
> Int_2_Loc: 13
> should be: 13
> Int_3_Loc: 7
> should be: 7
> Enum_Loc: 1
> should be: 1
> Str_1_Loc: DHRYSTONE PROGRAM, 1'ST STRING
> should be: DHRYSTONE PROGRAM, 1'ST STRING
> Str_2_Loc: DHRYSTONE PROGRAM, 2'ND STRING
> should be: DHRYSTONE PROGRAM, 2'ND STRING
>
> Microseconds for one run through Dhrystone: 1.0
> Dhrystones per Second: 1000000.0
>
> -------------------------------------------------
> ===== doing fuego phase: post_test =====
> ===== doing fuego phase: processing =====
> Going to check 1 fail cases for rpi.default.Benchmark.Dhrystone
> No fail cases detected for rpi.default.Benchmark.Dhrystone
> Fuego: requested test phases complete!
> Packaging run 'Benchmark.Dhrystone-default-16-rpi'
> run/
> run/testlog.txt
> run/machine-snapshot.txt
> run/build.xml
> run/syslog.after.txt
> run/prolog.sh
> run/devlog.txt
> run/spec.json
> run/consolelog.txt
> run/run.json
> run/syslog.before.txt
> Run packaged successfully, and is at: /tmp/run-Benchmark.Dhrystone-default-16-on-himanshu_wks:rpi.frp
> url = http://birdcloud.org/cgi-bin/fserver.py/?action=put_run
> file to send= run-Benchmark.Dhrystone-default-16-on-himanshu_wks:rpi.frp
> response from server = <Response [500]>
> Traceback (most recent call last):
> File "/usr/local/bin/ftc", line 6175, in <module>
> main()
> File "/usr/local/bin/ftc", line 6037, in main
> do_run_request(conf, options)
> File "/usr/local/bin/ftc", line 3553, in do_run_request
> do_put_run(conf, [run_id])
> File "/usr/local/bin/ftc", line 4223, in do_put_run
> result, content = put_run_fuego(conf, run_filepath)
> ValueError: need more than 1 value to unpack
>
> Thanks and regards,
> Pooja More
>
>
>
> On Thu, Dec 10, 2020 at 12:10 AM Bird, Tim <Tim.Bird at sony.com <mailto:Tim.Bird at sony.com> > wrote:
>
>
>
>
> > -----Original Message-----
> > From: Pooja Sanjay More <pooja.sm at pathpartnertech.com <mailto:pooja.sm at pathpartnertech.com> >
> > Sent: Wednesday, December 9, 2020 12:08 AM
> > To: tbird20d at yahoo.com <mailto:tbird20d at yahoo.com> ; Bird, Tim <Tim.Bird at sony.com <mailto:Tim.Bird at sony.com> >;
> fuego at lists.linuxfoundation.org <mailto:fuego at lists.linuxfoundation.org>
> > Subject: Error in running the test after latest pull
> >
> > Hi Tim,
> >
> > While running a Functional.hello_world test on the rpi board of pptlab we are getting the following errors. We are getting this
> after the
> > latest changes were pulled around 8-10 days back.
> >
> >
> > Please find the logs below.
> >
> This looks like 4 different issues:
> 1. cannot change locale
> 2. permission denied on build lock
> 3. no test log on target
> 4. file exists error on console log
>
> One or more of these might be "cascade" errors, which means that some other
> error is causing these, and the error is just a symptom of another error.
>
> I'll comment on each of these inline below.
>
> > Logs:
> >
> > root at cpu-366U:/# ftc run-test -b rpi -t Functional.hello_world
> > Running test 'Functional.hello_world' on board 'rpi' using spec 'default'
> > ===== doing fuego phase: pre_test =====
> > bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
>
> From the frequency of these, this looks like a host-side problem (and not a target-side
> problem). However, I could be wrong.
>
> Can you do the following, and send me the results:
> (inside the container):
>
> ls -l /etc/environment ; cat /etc/environment
> ls -l /etc/locale.gen ; cat /etc/locale.gen
> ls -l /etc/locale.conf ; cat /etc/locale.conf
>
> Mine are basically empty (except for the JENKINS_PORT) in my Docker container.
>
> Please try one of the fixes below (inside the Docker container):
> fix 1:
> perl -pi -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/g' /etc/locale.gen
> locale-gen en_US.UTF-8
> update-locale en_US.UTF-8
>
> fix 2:
> Add support for en_US.UTF-8 inside the container.
>
> echo "LC_ALL=en_US.UTF-8" >> /etc/environment
> echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
> echo "LANG=en_US.UTF-8" > /etc/locale.conf
> locale-gen en_US.UTF-8
>
>
>
> This worked for me.
>
>
> (if you don't have locale-gen in the container, try "apt-get install locales")
>
> If this doesn't fix it, try these steps on the raspberry pi board.
>
> Please report back which of these fixes (if any) worked.
>
> If it still doesn't fix it, then please get a full debug log of the run, and
> send that.
> $ ftc --debug run-test -b rpi -t hello_world
>
> Note that you don't need to use the full test name "Functional.hello_world",
> but only the shortened name.
>
>
> > bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> > bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> > bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> > bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> > bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> > bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> > bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> > ===== doing fuego phase: build =====
> > /fuego-core/scripts/functions.sh: line 403: /fuego-rw/buildzone/rpi.default.Functional.hello_world-/fuego.build.lock: Permission
> denied
>
> This sounds like it was from a test that failed during the build phase and left
> its lock file hanging around.
>
> Try just removing it:
> rm /fuego-rw/buildzone/rpi.default.Functional.hello_world-/fuego.build.lock
>
> Also, there should be a toolchain name in the directory path for the build directory.
> Does your rpi.board file specify a TOOLCHAIN setting?
> # cat /fuego-rw/boards/rpi3-2.board | grep TOOL
> TOOLCHAIN="debian-armhf"
>
> (You can also use ftc to get this:
> # ftc query-board -b rpi3-2 -n TOOLCHAIN
> debian-armhf
>
> It looks like there's a bug in fuego-core/scripts/common.sh where the TOOLCHAIN
> is not set before it is used to set the value of JOB_BUILD_DIR. I made a fix and committed it.
>
> I'm not sure what change I made that would have affect this.
>
> > In Fuego signal_handler (due to error in build phase)
> > ##### doing fuego phase: post_test (from signal handler) #####
> > bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> > bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> > bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> > bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> > scp: /home/pi/test/fuego.Functional.hello_world/Functional.hello_world.log: No such file or directory
>
> I'd like to see the full log to diagnose by the test log is not being created on the target.
> Please run the test with '--debug' and send the log.
>
> > INFO: the test did not produce a test log on the target
> > bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> > bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> > bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> > bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
> > ##### doing fuego phase: processing (from signal handler) #####
> > ### WARNING: Program returned exit code ''
> > ### WARNING: Log evaluation may be invalid
> > !!! ERROR: ERROR: log_compare: pattern 'SUCCESS' found 0 times (expected greater or equal than 1)
> > ERROR: test_processing returned an error
> > chart config not found. Using default values.
> > ERROR: results did not satisfy the threshold
> > Traceback (most recent call last):
> > File "/usr/local/bin/ftc", line 6162, in <module>
> > main()
> > File "/usr/local/bin/ftc", line 6051, in main
> > rcode, run_id = do_run_test(conf, options)
> > File "/usr/local/bin/ftc", line 4710, in do_run_test
> > os.symlink(log_filename, jenkins_log_filename)
> > OSError: [Errno 17] File exists
>
> This looks like an issue with there being an already existing run for a test.
> If so, I need to handle this issue better (or fail more gracefully).
>
> Can you do the following, inside the container:
>
> $ ls -l /fuego-rw/logs/Functional.hello_world
> and
> $ ls -l /var/lib/jenkins/jobs/*hello_world*/builds
>
> and send me the results?
>
> Sometimes this issue clears itself up as you execute more jobs
> (from either Jenkins or using ftc on the command line).
>
> However, ftc is supposed to catch this and avoid this error.
>
> I added some code to try to adjust the build number better
> depending on the state of both Jenkins build numbers and
> Fuego log (/run) numbers.
>
> Please do a 'git pull' on the fuego-core repository, and try again.
>
> Thanks very much for the bug report.
> -- Tim
>
>
>
>
> This message contains confidential information and is intended only for the individual(s) named. If you are not the intended recipient, you
> are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and attached file/s is strictly
> prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be
> secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The
> sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail
> transmission.
More information about the Fuego
mailing list