[Fuego] In-house functional test setup and execution process

Bird, Timothy Tim.Bird at sony.com
Wed Jun 28 18:48:11 UTC 2017



> -----Original Message-----
> From: Dhinakar Kalyanasundaram on Monday, June 26, 2017 10:16 PM
>
> The trick "|| true" worked. The test successfully passed now.
> I will use it based on my need or modify my application to return appropriate
> values.
> May I know what the 'exit code 130' means? what are the other exit codes?

I am not familiar with mfc-encode.  You would have to look in the source,
or search online, for what this exit code means, or for the meaning of other
exit codes.

In general, Unix/Linux programs are pretty random in their use of exit codes.
Sometimes they are documented but often they are not.  Sometimes they
are related to errnos.  So it is sometime helpful to see if the number 
represents something in errno.h (you can google this, or look in 
/usr/include/asm-generic/errno.h)  In that file 130 means: EOWNERDEAD,
which has something to do with mutexes.  It's probably not that.
  -- Tim


> As you mentioned, I will wait for your detailed analysis based on the log I
> sent.
> Thanks much.
> 
> Regards,
> Dhinakar,
> Senior Technical Manager,
> 2-07-526, Phoenix Building,
> +91-9902007650
> Samsung Research Institute, Bangalore.
> 
> 
> 
> 
> 
> --------- Original Message ---------
> Sender : Bird, Timothy <Tim.Bird at sony.com>
> Date   : 2017-06-26 20:51 (GMT+5:30)
> Title  : RE: In-house functional test setup and execution process
> To : Dhinakar Kalyanasundaram<dhinakar.k at samsung.com>
> CC : null<fuego at lists.linuxfoundation.org>
> 
> > -----Original Message-----
> > From: Dhinakar Kalyanasundaram [mailto:dhinakar.k at samsung.com]
> > Sent: Monday, June 26, 2017 8:04 AM
> > To: Bird, Timothy <Tim.Bird at sony.com>
> > Cc: fuego at lists.linuxfoundation.org
> > Subject: RE: In-house functional test setup and execution process
> >
> > Hi Tim,
> >
> > The exit code is 130.
> 
> OK. That's the issue.  I have to work on something else for a few hours, so I c
> an't take a detailed
> look at the log at the moment.  I'll try to get to it later today.
> 
> But the test program (mfc-
> encode) should return 0 for success.  If that program is out of your control
> (you can't adjust what it returns), and it's return code of non-
> zero does not actually indicate failure,
> then you should mask the return code by doing something like the following i
> n fuego_test.sh:
> 
> report "mfc-encode <args> || true"
> 
> That is, add "|| true" to the line that executes mfc-
> encode, forcing the result to be '0' (true, in shell-semantics).
> 
> Then fuego will use the results from log_compare as the result of the test, ra
> ther than the exit code
> from mfc-encode.
>  -- Tim
> 
> 
> > Please find the log and the contents of fuego_test.sh below.
> > I am having some problems in sending pure text based emails from office.
> > Please excuse for that.
> >
> > execution log
> > *************
> > .....................
> > .....................
> > + wait
> > ++ cmd 'cat /tmp/28913-20180'
> > ++ report_devlog 'cmd: cat /tmp/28913-20180'
> > ++ echo 'cmd: cat /tmp/28913-20180'
> > ++ ov_transport_cmd 'cat /tmp/28913-20180'
> > ++ case "$TRANSPORT" in
> > ++ run_python_quiet /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0 'cat
> > /tmp/28913-20180'
> > ++ '[' '!' -z
> > /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> > ++
> > PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/ga
> m
> > es
> > ++ python /usr/local/bin/sersh -T 5 -
> b 115200 /dev/ttyUSB0 'cat /tmp/28913-
> > 20180'
> > ++ wait
> > + RESULT=1
> > + cmd 'rm -f /tmp/28913-20180'
> > + report_devlog 'cmd: rm -f /tmp/28913-20180'
> > + echo 'cmd: rm -f /tmp/28913-20180'
> > + ov_transport_cmd 'rm -f /tmp/28913-20180'
> > + case "$TRANSPORT" in
> > + run_python_quiet /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0 'rm -f
> > /tmp/28913-20180'
> > + '[' '!' -z
> > /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> > +
> > PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/ga
> m
> > es
> > + python /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0 'rm -
> f /tmp/28913-
> > 20180'
> > + wait
> > + return 1
> > + signal_handler
> > + echo 'in signal_handler'
> > in signal_handler
> > + post_test
> > + trap post_term_handler SIGTERM
> > + trap - SIGHUP SIGALRM SIGINT ERR EXIT
> > + fetch_results
> > + local fuego_test_tmp=/tmp/fuego.Functional.mfc
> > + firmware
> > + ov_get_firmware
> > ++ cmd 'uname -r | xargs echo '
> > ++ report_devlog 'cmd: uname -r | xargs echo '
> > ++ echo 'cmd: uname -r | xargs echo '
> > ++ ov_transport_cmd 'uname -r | xargs echo '
> > ++ case "$TRANSPORT" in
> > ++ run_python_quiet /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0
> > 'uname -r | xargs echo '
> > ++ '[' '!' -z
> > /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> > ++
> > PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/ga
> m
> > es
> > ++ python /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0 'uname -
> r | xargs
> > echo '
> > ++ wait
> > + FW=4.9.0-00056-gae2de6e-dirty
> > + export FWVER=4.9.0-00056-gae2de6e-dirty
> > + FWVER=4.9.0-00056-gae2de6e-dirty
> > + get /home/a/fuego.Functional.mfc/Functional.mfc.log /fuego-
> > rw/logs/Functional.mfc/TRAV-Serial.default.8.8/testlog.txt
> > + report_devlog 'get: /home/a/fuego.Functional.mfc/Functional.mfc.log'
> > /fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/testlog.txt
> > + echo 'get: /home/a/fuego.Functional.mfc/Functional.mfc.log' /fuego-
> > rw/logs/Functional.mfc/TRAV-Serial.default.8.8/testlog.txt
> > + ov_transport_get /home/a/fuego.Functional.mfc/Functional.mfc.log
> > /fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/testlog.txt
> > + case "$TRANSPORT" in
> > + startswith /dev/ttyUSB0 /
> > + [[ / = \/ ]]
> > + return 0
> > + run_python_quiet /usr/local/bin/sercp -T 5 -b 115200 -r -d /dev/ttyUSB0
> > serial:/home/a/fuego.Functional.mfc/Functional.mfc.log /fuego-
> > rw/logs/Functional.mfc/TRAV-Serial.default.8.8/testlog.txt
> > + '[' '!' -z
> > /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> > +
> > PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/ga
> m
> > es
> > + python /usr/local/bin/sercp -T 5 -b 115200 -r -d /dev/ttyUSB0
> > serial:/home/a/fuego.Functional.mfc/Functional.mfc.log /fuego-
> > rw/logs/Functional.mfc/TRAV-Serial.default.8.8/testlog.txt
> > + wait
> > + dump_syslogs /tmp/fuego.Functional.mfc after
> > + is_empty /tmp/fuego.Functional.mfc
> > + '[' -z /tmp/fuego.Functional.mfc ']'
> > + is_empty after
> > + '[' -z after ']'
> > + ov_rootfs_logread /tmp/fuego.Functional.mfc after
> > + local fake_log=/tmp/fuego.Functional.mfc/fake-syslog
> > + cmd 'cd /tmp/fuego.Functional.mfc && cat
> > /tmp/fuego.Functional.mfc/fake-syslog > TRAV-Serial.8.8.after'
> > + report_devlog 'cmd: cd /tmp/fuego.Functional.mfc && cat
> > /tmp/fuego.Functional.mfc/fake-syslog > TRAV-Serial.8.8.after'
> > + echo 'cmd: cd /tmp/fuego.Functional.mfc && cat
> > /tmp/fuego.Functional.mfc/fake-syslog > TRAV-Serial.8.8.after'
> > + ov_transport_cmd 'cd /tmp/fuego.Functional.mfc && cat
> > /tmp/fuego.Functional.mfc/fake-syslog > TRAV-Serial.8.8.after'
> > + case "$TRANSPORT" in
> > + run_python_quiet /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0 'cd
> > /tmp/fuego.Functional.mfc && cat /tmp/fuego.Functional.mfc/fake-
> syslog >
> > TRAV-Serial.8.8.after'
> > + '[' '!' -z
> > /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> > +
> > PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/ga
> m
> > es
> > + python /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0 'cd
> > /tmp/fuego.Functional.mfc && cat /tmp/fuego.Functional.mfc/fake-
> syslog >
> > TRAV-Serial.8.8.after'
> > + wait
> > + get /tmp/fuego.Functional.mfc/TRAV-Serial.8.8.before /fuego-
> > rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.before.txt
> > + report_devlog 'get: /tmp/fuego.Functional.mfc/TRAV-Serial.8.8.before'
> > /fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.before.txt
> > + echo 'get: /tmp/fuego.Functional.mfc/TRAV-Serial.8.8.before' /fuego-
> > rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.before.txt
> > + ov_transport_get /tmp/fuego.Functional.mfc/TRAV-Serial.8.8.before
> > /fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.before.txt
> > + case "$TRANSPORT" in
> > + startswith /dev/ttyUSB0 /
> > + [[ / = \/ ]]
> > + return 0
> > + run_python_quiet /usr/local/bin/sercp -T 5 -b 115200 -r -d /dev/ttyUSB0
> > serial:/tmp/fuego.Functional.mfc/TRAV-Serial.8.8.before /fuego-
> > rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.before.txt
> > + '[' '!' -z
> > /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> > +
> > PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/ga
> m
> > es
> > + python /usr/local/bin/sercp -T 5 -b 115200 -r -d /dev/ttyUSB0
> > serial:/tmp/fuego.Functional.mfc/TRAV-Serial.8.8.before /fuego-
> > rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.before.txt
> > + wait
> > + '[' 0 -ne 0 ']'
> > + get /tmp/fuego.Functional.mfc/TRAV-Serial.8.8.after /fuego-
> > rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.after.txt
> > + report_devlog 'get: /tmp/fuego.Functional.mfc/TRAV-Serial.8.8.after'
> > /fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.after.txt
> > + echo 'get: /tmp/fuego.Functional.mfc/TRAV-Serial.8.8.after' /fuego-
> > rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.after.txt
> > + ov_transport_get /tmp/fuego.Functional.mfc/TRAV-Serial.8.8.after
> > /fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.after.txt
> > + case "$TRANSPORT" in
> > + startswith /dev/ttyUSB0 /
> > + [[ / = \/ ]]
> > + return 0
> > + run_python_quiet /usr/local/bin/sercp -T 5 -b 115200 -r -d /dev/ttyUSB0
> > serial:/tmp/fuego.Functional.mfc/TRAV-Serial.8.8.after /fuego-
> > rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.after.txt
> > + '[' '!' -z
> > /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> > +
> > PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/ga
> m
> > es
> > + python /usr/local/bin/sercp -T 5 -b 115200 -r -d /dev/ttyUSB0
> > serial:/tmp/fuego.Functional.mfc/TRAV-Serial.8.8.after /fuego-
> > rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.after.txt
> > + wait
> > + call_if_present test_fetch_results
> > + declare -f -F test_fetch_results
> > + return 0
> > + cleanup
> > + call_if_present test_cleanup
> > + declare -f -F test_cleanup
> > + return 0
> > + '[' true = true ']'
> > + target_cleanup Functional.mfc
> > + local fuego_test_tmp=/tmp/fuego.Functional.mfc
> > + cmd 'rm -rf /home/a/fuego.Functional.mfc /tmp/fuego.Functional.mfc'
> > + report_devlog 'cmd: rm -rf /home/a/fuego.Functional.mfc
> > /tmp/fuego.Functional.mfc'
> > + echo 'cmd: rm -rf /home/a/fuego.Functional.mfc
> > /tmp/fuego.Functional.mfc'
> > + ov_transport_cmd 'rm -rf /home/a/fuego.Functional.mfc
> > /tmp/fuego.Functional.mfc'
> > + case "$TRANSPORT" in
> > + run_python_quiet /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0 'rm -
> rf
> > /home/a/fuego.Functional.mfc /tmp/fuego.Functional.mfc'
> > + '[' '!' -z
> > /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> > +
> > PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/ga
> m
> > es
> > + python /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0 'rm -rf
> > /home/a/fuego.Functional.mfc /tmp/fuego.Functional.mfc'
> > + wait
> > + echo 'Teardown board link'
> > Teardown board link
> > + '[' -n '' ']'
> > + true
> > + ov_transport_disconnect
> > + case "$TRANSPORT" in
> > + return 0
> > + processing
> > + local RETURN_VALUE=0
> > + call_if_present test_processing
> > + declare -f -F test_processing
> > + test_processing
> > + log_compare Functional.mfc 1 mfc.c p
> > + local RETURN_VALUE=0
> > + local PARSED_LOGFILE=testlog.p.txt
> > + '[' -f /fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/testlog.txt ']'
> > ++ cat /fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/testlog.txt
> > ++ grep -E mfc.c
> > ++ wc -l
> > + current_count=2
> > + '[' p = p ']'
> > + '[' 2 -ge 1 ']'
> > + echo 'log_compare: pattern '\''mfc.c'\'' found 2 times (expected greater o
> r
> > equal than 1)'
> > log_compare: pattern 'mfc.c' found 2 times (expected greater or equal tha
> n
> > 1)
> > + '[' p = n ']'
> > + return 0
> > + '[' 0 -ne 0 ']'
> > + fail_check_cases
> > + testlog=/fuego-rw/logs/Functional.mfc/TRAV-
> Serial.default.8.8/testlog.txt
> > + slog_prefix=/fuego-rw/logs/Functional.mfc/TRAV-
> Serial.default.8.8/syslog
> > ++ echo FUNCTIONAL.MFC
> > ++ tr . _
> > + upName=FUNCTIONAL_MFC
> > + fcname=FUNCTIONAL_MFC_FAIL_CASE_COUNT
> > + fcc=
> > + '[' -z '' ']'
> > + return 0
> > + '[' 0 -ne 0 ']'
> > + syslog_cmp
> > + PREFIX=/fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog
> > + rc=0
> > + '[' -f /fuego-rw/logs/Functional.mfc/TRAV-
> > Serial.default.8.8/syslog.before.txt ']'
> > + diff -ua /fuego-rw/logs/Functional.mfc/TRAV-
> > Serial.default.8.8/syslog.before.txt /fuego-rw/logs/Functional.mfc/TRAV-
> > Serial.default.8.8/syslog.after.txt
> > + grep -vEf /fuego-core/engine/scripts/syslog.ignore
> > + grep -E -e '\.(Bug:|Oops)'
> > + '[' 0 -eq 1 ']'
> > + return 0
> > + '[' 0 -ne 0 ']'
> > + PYTHON_ARGS='-W ignore::DeprecationWarning -
> W ignore::UserWarning'
> > + '[' -e /fuego-core/engine/tests/Functional.mfc/parser.py ']'
> > + run_python -W ignore::DeprecationWarning -W ignore::UserWarning
> > /fuego-core/engine/scripts/generic_parser.py 0
> > + '[' '!' -z
> > /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> > + echo 'running python with
> > PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/ga
> m
> > es'
> > running python with
> > PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/ga
> m
> > es
> > +
> > PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/ga
> m
> > es
> > + python -W ignore::DeprecationWarning -W ignore::UserWarning /fuego-
> > core/engine/scripts/generic_parser.py 0
> > Received: 0
> > Test section fail_or_pass: test result 0 satisfies (eq) threshold 0.
> > Data file /fuego-rw/logs/Functional.mfc/plot.data was updated.
> > Reading plot data from /fuego-rw/logs/Functional.mfc/plot.data
> > Saving plot file to /fuego-rw/logs/Functional.mfc/plot.png
> > + rc=0
> > + '[' 0 -eq 0 ']'
> > + run_python -W ignore::DeprecationWarning -W ignore::UserWarning
> > /fuego-core/engine/scripts/parser/dataload.py
> > + '[' '!' -z
> > /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> > + echo 'running python with
> > PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/ga
> m
> > es'
> > running python with
> > PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/ga
> m
> > es
> > +
> > PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/ga
> m
> > es
> > + python -W ignore::DeprecationWarning -W ignore::UserWarning /fuego-
> > core/engine/scripts/parser/dataload.py
> > Writing results to  /fuego-rw/logs/Functional.mfc/results.json
> > + rc=0
> > + '[' 0 -eq 0 ']'
> > + ln -s ../plot.png /fuego-rw/logs/Functional.mfc/TRAV-
> > Serial.default.8.8/plot.png
> > + '[' '!' -e /fuego-rw/logs/Functional.mfc/TRAV-
> > Serial.default.8.8/consolelog.txt ']'
> > + ln -s /var/lib/jenkins/jobs/TRAV-
> Serial.default.Functional.mfc/builds/8/log
> > /fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/consolelog.txt
> > + make_run_file 0
> > + end_arr=($(date +"%s %N"))
> > ++ date '+%s %N'
> > + FUEGO_END_TIME=1498489009729
> > + DURATION=19713
> > + '[' 0 = 0 ']'
> > + FUEGO_RESULT_STR=SUCCESS
> > + cat
> > + return 0
> > + exit 130
> > Build step 'Execute shell' marked build as failure
> > [description-setter] Description set: <a
> > href="/fuego/userContent/fuego.logs/Functional.mfc/TRAV-
> > Serial.default.8.8/plot.png">plot</a> <a
> > href="/fuego/userContent/fuego.logs/Functional.mfc/TRAV-
> > Serial.default.8.8/testlog.txt">log</a>
> > Finished: FAILURE
> >
> >
> > fuego_test.sh
> > *************
> >
> > function test_run {
> >     report "cd /opt/ltp/"
> >     report "mfc-encode -m /dev/video1 -o test1.264 -c h264 -d 120 -r 60 -b
> > 40000000 -s 1920x1080 -t mfcencode_tc1 -g
> > /usr/streams/encode/encodetest1_h264_1920x1080.26"
> > }
> >
> > function test_processing {
> >     log_compare "$TESTDIR" "1" "mfc.c" "p"
> > }
> >
> >
> > Regards,
> > Dhinakar
> >
> >
> >
> > > -----Original Message-----
> > Dhinakar,
> >
> >  Do you know what the exit code is from the test program?
> >  Is it non-zero?  That is what Jenkins uses to determine test failure.
> >
> >  Can you turn on FUEGO_DEBUG (in the job configuration), and
> >  send the full log?  Also, can you send a copy of the fuego_test.sh
> >  for this test?
> >
> >  Thanks,
> >   -- Tim
> >
> >
> >  > -----Original Message-----
> >  > From: dhinakar k [mailto:dhinakar.k at gmail.com]
> >  > Sent: Monday, June 26, 2017 7:03 AM
> >  > To: fuego at lists.linuxfoundation.org; Bird, Timothy <Tim.Bird at sony.com
> >
> >  > Subject: In-house functional test setup and execution process
> >  >
> >  > Hi,
> >  >
> >  > I have an in-house test for MFC codec application.
> >  > I have already kept the executable on target board and created
> >  > fuego_test.sh with test_run and test_processing functions only. spec.jso
> n
> > is
> >  > just default, nothing specific in that.
> >  >
> >  > I use log_compare to look for some specific keyword occurance (like pass
> ,
> >  > success, mfc.c) in the log file.
> >  >
> >  > The test executes correctly and log_compare is able to find keywords but
> > still
> >  > the test result is a failure. What could be the issue? What iam missing her
> e.
> >  >
> >  > Please find a portion of the log below for reference. It says 'Build step
> >  > 'Execute shell' marked build as failure'.
> >  >
> >  > Teardown board link
> >  > log_compare: pattern 'mfc.c' found 2 times (expected greater than or
> > equal
> >  > than 1)
> >  > running python...
> >  > Received: 0
> >  > Test section fail_or_pass: test result 0 satisfies (eq) threshold 0.
> >  > Data file /fuego-rw/.... was updated.
> >  > Reading plot....
> >  > Saving plot...
> >  > running python...
> >  > Writing results to ... /results.json
> >  > Build step 'Execute shell' marked build as failure
> >  > [description setter] ...
> >  > Finished: FAILURE
> >  >
> >  > Regards,
> >  > Dhinakar
> 
> 



More information about the Fuego mailing list