[Fuego] In-house functional test setup and execution process
Dhinakar Kalyanasundaram
dhinakar.k at samsung.com
Tue Jun 27 05:16:12 UTC 2017
Thanks Tim,
The trick "|| true" worked. The test successfully passed now.
I will use it based on my need or modify my application to return appropriate values.
May I know what the 'exit code 130' means? what are the other exit codes?
As you mentioned, I will wait for your detailed analysis based on the log I sent.
Thanks much.
Regards,
Dhinakar,
Senior Technical Manager,
2-07-526, Phoenix Building,
+91-9902007650
Samsung Research Institute, Bangalore.
--------- Original Message ---------
Sender : Bird, Timothy <Tim.Bird at sony.com>
Date : 2017-06-26 20:51 (GMT+5:30)
Title : RE: In-house functional test setup and execution process
To : Dhinakar Kalyanasundaram<dhinakar.k at samsung.com>
CC : null<fuego at lists.linuxfoundation.org>
> -----Original Message-----
> From: Dhinakar Kalyanasundaram [mailto:dhinakar.k at samsung.com]
> Sent: Monday, June 26, 2017 8:04 AM
> To: Bird, Timothy <Tim.Bird at sony.com>
> Cc: fuego at lists.linuxfoundation.org
> Subject: RE: In-house functional test setup and execution process
>
> Hi Tim,
>
> The exit code is 130.
OK. That's the issue. I have to work on something else for a few hours, so I can't take a detailed
look at the log at the moment. I'll try to get to it later today.
But the test program (mfc-encode) should return 0 for success. If that program is out of your control
(you can't adjust what it returns), and it's return code of non-zero does not actually indicate failure,
then you should mask the return code by doing something like the following in fuego_test.sh:
report "mfc-encode <args> || true"
That is, add "|| true" to the line that executes mfc-encode, forcing the result to be '0' (true, in shell-semantics).
Then fuego will use the results from log_compare as the result of the test, rather than the exit code
from mfc-encode.
-- Tim
> Please find the log and the contents of fuego_test.sh below.
> I am having some problems in sending pure text based emails from office.
> Please excuse for that.
>
> execution log
> *************
> .....................
> .....................
> + wait
> ++ cmd 'cat /tmp/28913-20180'
> ++ report_devlog 'cmd: cat /tmp/28913-20180'
> ++ echo 'cmd: cat /tmp/28913-20180'
> ++ ov_transport_cmd 'cat /tmp/28913-20180'
> ++ case "$TRANSPORT" in
> ++ run_python_quiet /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0 'cat
> /tmp/28913-20180'
> ++ '[' '!' -z
> /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> ++
> PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/gam
> es
> ++ python /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0 'cat /tmp/28913-
> 20180'
> ++ wait
> + RESULT=1
> + cmd 'rm -f /tmp/28913-20180'
> + report_devlog 'cmd: rm -f /tmp/28913-20180'
> + echo 'cmd: rm -f /tmp/28913-20180'
> + ov_transport_cmd 'rm -f /tmp/28913-20180'
> + case "$TRANSPORT" in
> + run_python_quiet /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0 'rm -f
> /tmp/28913-20180'
> + '[' '!' -z
> /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> +
> PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/gam
> es
> + python /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0 'rm -f /tmp/28913-
> 20180'
> + wait
> + return 1
> + signal_handler
> + echo 'in signal_handler'
> in signal_handler
> + post_test
> + trap post_term_handler SIGTERM
> + trap - SIGHUP SIGALRM SIGINT ERR EXIT
> + fetch_results
> + local fuego_test_tmp=/tmp/fuego.Functional.mfc
> + firmware
> + ov_get_firmware
> ++ cmd 'uname -r | xargs echo '
> ++ report_devlog 'cmd: uname -r | xargs echo '
> ++ echo 'cmd: uname -r | xargs echo '
> ++ ov_transport_cmd 'uname -r | xargs echo '
> ++ case "$TRANSPORT" in
> ++ run_python_quiet /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0
> 'uname -r | xargs echo '
> ++ '[' '!' -z
> /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> ++
> PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/gam
> es
> ++ python /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0 'uname -r | xargs
> echo '
> ++ wait
> + FW=4.9.0-00056-gae2de6e-dirty
> + export FWVER=4.9.0-00056-gae2de6e-dirty
> + FWVER=4.9.0-00056-gae2de6e-dirty
> + get /home/a/fuego.Functional.mfc/Functional.mfc.log /fuego-
> rw/logs/Functional.mfc/TRAV-Serial.default.8.8/testlog.txt
> + report_devlog 'get: /home/a/fuego.Functional.mfc/Functional.mfc.log'
> /fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/testlog.txt
> + echo 'get: /home/a/fuego.Functional.mfc/Functional.mfc.log' /fuego-
> rw/logs/Functional.mfc/TRAV-Serial.default.8.8/testlog.txt
> + ov_transport_get /home/a/fuego.Functional.mfc/Functional.mfc.log
> /fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/testlog.txt
> + case "$TRANSPORT" in
> + startswith /dev/ttyUSB0 /
> + [[ / = \/ ]]
> + return 0
> + run_python_quiet /usr/local/bin/sercp -T 5 -b 115200 -r -d /dev/ttyUSB0
> serial:/home/a/fuego.Functional.mfc/Functional.mfc.log /fuego-
> rw/logs/Functional.mfc/TRAV-Serial.default.8.8/testlog.txt
> + '[' '!' -z
> /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> +
> PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/gam
> es
> + python /usr/local/bin/sercp -T 5 -b 115200 -r -d /dev/ttyUSB0
> serial:/home/a/fuego.Functional.mfc/Functional.mfc.log /fuego-
> rw/logs/Functional.mfc/TRAV-Serial.default.8.8/testlog.txt
> + wait
> + dump_syslogs /tmp/fuego.Functional.mfc after
> + is_empty /tmp/fuego.Functional.mfc
> + '[' -z /tmp/fuego.Functional.mfc ']'
> + is_empty after
> + '[' -z after ']'
> + ov_rootfs_logread /tmp/fuego.Functional.mfc after
> + local fake_log=/tmp/fuego.Functional.mfc/fake-syslog
> + cmd 'cd /tmp/fuego.Functional.mfc && cat
> /tmp/fuego.Functional.mfc/fake-syslog > TRAV-Serial.8.8.after'
> + report_devlog 'cmd: cd /tmp/fuego.Functional.mfc && cat
> /tmp/fuego.Functional.mfc/fake-syslog > TRAV-Serial.8.8.after'
> + echo 'cmd: cd /tmp/fuego.Functional.mfc && cat
> /tmp/fuego.Functional.mfc/fake-syslog > TRAV-Serial.8.8.after'
> + ov_transport_cmd 'cd /tmp/fuego.Functional.mfc && cat
> /tmp/fuego.Functional.mfc/fake-syslog > TRAV-Serial.8.8.after'
> + case "$TRANSPORT" in
> + run_python_quiet /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0 'cd
> /tmp/fuego.Functional.mfc && cat /tmp/fuego.Functional.mfc/fake-syslog >
> TRAV-Serial.8.8.after'
> + '[' '!' -z
> /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> +
> PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/gam
> es
> + python /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0 'cd
> /tmp/fuego.Functional.mfc && cat /tmp/fuego.Functional.mfc/fake-syslog >
> TRAV-Serial.8.8.after'
> + wait
> + get /tmp/fuego.Functional.mfc/TRAV-Serial.8.8.before /fuego-
> rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.before.txt
> + report_devlog 'get: /tmp/fuego.Functional.mfc/TRAV-Serial.8.8.before'
> /fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.before.txt
> + echo 'get: /tmp/fuego.Functional.mfc/TRAV-Serial.8.8.before' /fuego-
> rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.before.txt
> + ov_transport_get /tmp/fuego.Functional.mfc/TRAV-Serial.8.8.before
> /fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.before.txt
> + case "$TRANSPORT" in
> + startswith /dev/ttyUSB0 /
> + [[ / = \/ ]]
> + return 0
> + run_python_quiet /usr/local/bin/sercp -T 5 -b 115200 -r -d /dev/ttyUSB0
> serial:/tmp/fuego.Functional.mfc/TRAV-Serial.8.8.before /fuego-
> rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.before.txt
> + '[' '!' -z
> /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> +
> PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/gam
> es
> + python /usr/local/bin/sercp -T 5 -b 115200 -r -d /dev/ttyUSB0
> serial:/tmp/fuego.Functional.mfc/TRAV-Serial.8.8.before /fuego-
> rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.before.txt
> + wait
> + '[' 0 -ne 0 ']'
> + get /tmp/fuego.Functional.mfc/TRAV-Serial.8.8.after /fuego-
> rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.after.txt
> + report_devlog 'get: /tmp/fuego.Functional.mfc/TRAV-Serial.8.8.after'
> /fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.after.txt
> + echo 'get: /tmp/fuego.Functional.mfc/TRAV-Serial.8.8.after' /fuego-
> rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.after.txt
> + ov_transport_get /tmp/fuego.Functional.mfc/TRAV-Serial.8.8.after
> /fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.after.txt
> + case "$TRANSPORT" in
> + startswith /dev/ttyUSB0 /
> + [[ / = \/ ]]
> + return 0
> + run_python_quiet /usr/local/bin/sercp -T 5 -b 115200 -r -d /dev/ttyUSB0
> serial:/tmp/fuego.Functional.mfc/TRAV-Serial.8.8.after /fuego-
> rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.after.txt
> + '[' '!' -z
> /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> +
> PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/gam
> es
> + python /usr/local/bin/sercp -T 5 -b 115200 -r -d /dev/ttyUSB0
> serial:/tmp/fuego.Functional.mfc/TRAV-Serial.8.8.after /fuego-
> rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog.after.txt
> + wait
> + call_if_present test_fetch_results
> + declare -f -F test_fetch_results
> + return 0
> + cleanup
> + call_if_present test_cleanup
> + declare -f -F test_cleanup
> + return 0
> + '[' true = true ']'
> + target_cleanup Functional.mfc
> + local fuego_test_tmp=/tmp/fuego.Functional.mfc
> + cmd 'rm -rf /home/a/fuego.Functional.mfc /tmp/fuego.Functional.mfc'
> + report_devlog 'cmd: rm -rf /home/a/fuego.Functional.mfc
> /tmp/fuego.Functional.mfc'
> + echo 'cmd: rm -rf /home/a/fuego.Functional.mfc
> /tmp/fuego.Functional.mfc'
> + ov_transport_cmd 'rm -rf /home/a/fuego.Functional.mfc
> /tmp/fuego.Functional.mfc'
> + case "$TRANSPORT" in
> + run_python_quiet /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0 'rm -rf
> /home/a/fuego.Functional.mfc /tmp/fuego.Functional.mfc'
> + '[' '!' -z
> /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> +
> PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/gam
> es
> + python /usr/local/bin/sersh -T 5 -b 115200 /dev/ttyUSB0 'rm -rf
> /home/a/fuego.Functional.mfc /tmp/fuego.Functional.mfc'
> + wait
> + echo 'Teardown board link'
> Teardown board link
> + '[' -n '' ']'
> + true
> + ov_transport_disconnect
> + case "$TRANSPORT" in
> + return 0
> + processing
> + local RETURN_VALUE=0
> + call_if_present test_processing
> + declare -f -F test_processing
> + test_processing
> + log_compare Functional.mfc 1 mfc.c p
> + local RETURN_VALUE=0
> + local PARSED_LOGFILE=testlog.p.txt
> + '[' -f /fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/testlog.txt ']'
> ++ cat /fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/testlog.txt
> ++ grep -E mfc.c
> ++ wc -l
> + current_count=2
> + '[' p = p ']'
> + '[' 2 -ge 1 ']'
> + echo 'log_compare: pattern '\''mfc.c'\'' found 2 times (expected greater or
> equal than 1)'
> log_compare: pattern 'mfc.c' found 2 times (expected greater or equal than
> 1)
> + '[' p = n ']'
> + return 0
> + '[' 0 -ne 0 ']'
> + fail_check_cases
> + testlog=/fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/testlog.txt
> + slog_prefix=/fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog
> ++ echo FUNCTIONAL.MFC
> ++ tr . _
> + upName=FUNCTIONAL_MFC
> + fcname=FUNCTIONAL_MFC_FAIL_CASE_COUNT
> + fcc=
> + '[' -z '' ']'
> + return 0
> + '[' 0 -ne 0 ']'
> + syslog_cmp
> + PREFIX=/fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/syslog
> + rc=0
> + '[' -f /fuego-rw/logs/Functional.mfc/TRAV-
> Serial.default.8.8/syslog.before.txt ']'
> + diff -ua /fuego-rw/logs/Functional.mfc/TRAV-
> Serial.default.8.8/syslog.before.txt /fuego-rw/logs/Functional.mfc/TRAV-
> Serial.default.8.8/syslog.after.txt
> + grep -vEf /fuego-core/engine/scripts/syslog.ignore
> + grep -E -e '\.(Bug:|Oops)'
> + '[' 0 -eq 1 ']'
> + return 0
> + '[' 0 -ne 0 ']'
> + PYTHON_ARGS='-W ignore::DeprecationWarning -W ignore::UserWarning'
> + '[' -e /fuego-core/engine/tests/Functional.mfc/parser.py ']'
> + run_python -W ignore::DeprecationWarning -W ignore::UserWarning
> /fuego-core/engine/scripts/generic_parser.py 0
> + '[' '!' -z
> /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> + echo 'running python with
> PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/gam
> es'
> running python with
> PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/gam
> es
> +
> PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/gam
> es
> + python -W ignore::DeprecationWarning -W ignore::UserWarning /fuego-
> core/engine/scripts/generic_parser.py 0
> Received: 0
> Test section fail_or_pass: test result 0 satisfies (eq) threshold 0.
> Data file /fuego-rw/logs/Functional.mfc/plot.data was updated.
> Reading plot data from /fuego-rw/logs/Functional.mfc/plot.data
> Saving plot file to /fuego-rw/logs/Functional.mfc/plot.png
> + rc=0
> + '[' 0 -eq 0 ']'
> + run_python -W ignore::DeprecationWarning -W ignore::UserWarning
> /fuego-core/engine/scripts/parser/dataload.py
> + '[' '!' -z
> /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games ']'
> + echo 'running python with
> PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/gam
> es'
> running python with
> PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/gam
> es
> +
> PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/gam
> es
> + python -W ignore::DeprecationWarning -W ignore::UserWarning /fuego-
> core/engine/scripts/parser/dataload.py
> Writing results to /fuego-rw/logs/Functional.mfc/results.json
> + rc=0
> + '[' 0 -eq 0 ']'
> + ln -s ../plot.png /fuego-rw/logs/Functional.mfc/TRAV-
> Serial.default.8.8/plot.png
> + '[' '!' -e /fuego-rw/logs/Functional.mfc/TRAV-
> Serial.default.8.8/consolelog.txt ']'
> + ln -s /var/lib/jenkins/jobs/TRAV-Serial.default.Functional.mfc/builds/8/log
> /fuego-rw/logs/Functional.mfc/TRAV-Serial.default.8.8/consolelog.txt
> + make_run_file 0
> + end_arr=($(date +"%s %N"))
> ++ date '+%s %N'
> + FUEGO_END_TIME=1498489009729
> + DURATION=19713
> + '[' 0 = 0 ']'
> + FUEGO_RESULT_STR=SUCCESS
> + cat
> + return 0
> + exit 130
> Build step 'Execute shell' marked build as failure
> [description-setter] Description set: <a
> href="/fuego/userContent/fuego.logs/Functional.mfc/TRAV-
> Serial.default.8.8/plot.png">plot</a> <a
> href="/fuego/userContent/fuego.logs/Functional.mfc/TRAV-
> Serial.default.8.8/testlog.txt">log</a>
> Finished: FAILURE
>
>
> fuego_test.sh
> *************
>
> function test_run {
> report "cd /opt/ltp/"
> report "mfc-encode -m /dev/video1 -o test1.264 -c h264 -d 120 -r 60 -b
> 40000000 -s 1920x1080 -t mfcencode_tc1 -g
> /usr/streams/encode/encodetest1_h264_1920x1080.26"
> }
>
> function test_processing {
> log_compare "$TESTDIR" "1" "mfc.c" "p"
> }
>
>
> Regards,
> Dhinakar
>
>
>
> > -----Original Message-----
> Dhinakar,
>
> Do you know what the exit code is from the test program?
> Is it non-zero? That is what Jenkins uses to determine test failure.
>
> Can you turn on FUEGO_DEBUG (in the job configuration), and
> send the full log? Also, can you send a copy of the fuego_test.sh
> for this test?
>
> Thanks,
> -- Tim
>
>
> > -----Original Message-----
> > From: dhinakar k [mailto:dhinakar.k at gmail.com]
> > Sent: Monday, June 26, 2017 7:03 AM
> > To: fuego at lists.linuxfoundation.org; Bird, Timothy <Tim.Bird at sony.com>
> > Subject: In-house functional test setup and execution process
> >
> > Hi,
> >
> > I have an in-house test for MFC codec application.
> > I have already kept the executable on target board and created
> > fuego_test.sh with test_run and test_processing functions only. spec.json
> is
> > just default, nothing specific in that.
> >
> > I use log_compare to look for some specific keyword occurance (like pass,
> > success, mfc.c) in the log file.
> >
> > The test executes correctly and log_compare is able to find keywords but
> still
> > the test result is a failure. What could be the issue? What iam missing here.
> >
> > Please find a portion of the log below for reference. It says 'Build step
> > 'Execute shell' marked build as failure'.
> >
> > Teardown board link
> > log_compare: pattern 'mfc.c' found 2 times (expected greater than or
> equal
> > than 1)
> > running python...
> > Received: 0
> > Test section fail_or_pass: test result 0 satisfies (eq) threshold 0.
> > Data file /fuego-rw/.... was updated.
> > Reading plot....
> > Saving plot...
> > running python...
> > Writing results to ... /results.json
> > Build step 'Execute shell' marked build as failure
> > [description setter] ...
> > Finished: FAILURE
> >
> > Regards,
> > Dhinakar
More information about the Fuego
mailing list