[Fuego] Problem with /fuego-rw/tests in your lab

Tim.Bird at sony.com Tim.Bird at sony.com
Tue Mar 16 17:52:22 UTC 2021


Actually a slight correction below...

> -----Original Message-----
> From: Fuego <fuego-bounces at lists.linuxfoundation.org> On Behalf Of Tim.Bird at sony.com
> 
> OK.  thanks for this update.  This is very useful.
> 
> Based on your experience here, I believe I should change the logic for test selection
> for 'ftc run-test'.

That should be: "for 'ftc run-request'".

> Right now, if the test is defined in both /fuego-rw/tests and /fuego-core/tests,
> then Fuego runs the one from /fuego-rw/tests.  However, I think that it would be more
> correct to run the one with the latest version (if the test request specifies 'any' or 'latest', for
> the request test version).
> 
> Looking at the code, it appears I'll have to do some refactoring to support this behavior.
> 
> I try to work on that this week.  Thanks again for the report.
>  -- Tim
> 
> > -----Original Message-----
> > From: Pavan Arun Deshpande <pavan.ad at pathpartnertech.com>
> >
> > Hi Tim,
> >
> > Pulled latest fuego-core sources from the repository
> >
> > fuego_lab_check in fuego-rw/tests directory belonged to an older version and it was picking that for testing as ftc checks for its presence
> in
> > fuego-rw/tests directory.
> >
> > when I removed fuego_lab_check from fuego-rw/tests directory and ran the fuego_lab_check , then it ran the testcase from fuego-
> > core/tests directory which is of the latest version.
> >
> > Please find a copy of log messages generated while running Functional.fuego_lab_check.
> >
> > ===== doing fuego phase: pre_test =====
> > ===== doing fuego phase: build =====
> > The test is already built
> > Fuego test_build duration=0 seconds
> > ===== doing fuego phase: deploy =====
> > ===== doing fuego phase: snapshot =====
> > Firmware revision: 5.4.83-v8-93349cdff-2+
> > ===== doing fuego phase: run =====
> > -------------------------------------------------
> > TAP version 13
> > 1..1
> > # Version of this test=1.4-2
> > # Version of this test (hardcoded)=1.4-2
> > # Location of this test=/fuego-core/tests/Functional.fuego_lab_check
> > ######### Process list of docker container:
> > USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> > root         1  0.0  0.0  19456  3836 pts/0    Ss+  Mar15   0:00 /bin/bash
> > jenkins     58  0.0  0.0  18708     0 ?        S    Mar15   0:00 /usr/bin/daemon --name=jenkins --inherit --
> env=JENKINS_HOME=/var/lib/jenkins
> > --output=/var/log/jenkins/jenkins.log --pidfile=/var/run/jenkins/jenkins.pid -- /usr/bin/java -Djava.awt.headless=true -
> > Djenkins.install.runSetupWizard=false -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar
> > /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8090 --prefix=/fuego
> > jenkins     60  0.7  8.7 9950392 1422876 ?     Sl   Mar15  10:01 /usr/bin/java -Djava.awt.headless=true -
> > Djenkins.install.runSetupWizard=false -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar
> > /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8090 --prefix=/fuego
> > root       110  0.0  0.0   9588     0 ?        Ss   Mar15   0:00 /usr/bin/netserver
> > root       129  0.0  0.0   9440     8 ?        Ss   Mar15   0:00 iperf3 -V -s -D -f M
> > jenkins    176  0.5  1.0 8487928 166808 ?      Sl   Mar15   7:09 java -jar /fuego-core/slave.jar
> > jenkins    177  0.5  1.1 8425932 190836 ?      Sl   Mar15   7:13 java -jar /fuego-core/slave.jar
> > jenkins  13870  0.9  0.3  99016 55856 ?        S    10:50   0:00 /usr/bin/python /usr/local/bin/ftc run-request --put-run --allow-upgrade
> > request-2021-03-16_10:48:31.97-pptlab:rpi
> > jenkins  13873  0.4  0.0  12416  4248 ?        S    10:50   0:00 /bin/bash -e /fuego-core/scripts/main.sh
> > jenkins  13962  0.0  0.0  11192  3008 ?        S    10:50   0:00 bash -c { ps -aux; echo $? > /tmp/13873-4538 ; } 2>&1 | tee -a /fuego-
> > rw/logs/Functional.fuego_lab_check/rpi.default.11.11/hostlog.txt
> > jenkins  13963  0.0  0.0  11192   272 ?        S    10:50   0:00 bash -c { ps -aux; echo $? > /tmp/13873-4538 ; } 2>&1 | tee -a /fuego-
> > rw/logs/Functional.fuego_lab_check/rpi.default.11.11/hostlog.txt
> > jenkins  13964  0.0  0.0   5832   708 ?        S    10:50   0:00 tee -a /fuego-
> rw/logs/Functional.fuego_lab_check/rpi.default.11.11/hostlog.txt
> > jenkins  13965  0.0  0.0  38304  3368 ?        R    10:50   0:00 ps -aux
> > jenkins  18240  0.0  0.0  11180  2960 ?        S    02:05   0:00 /bin/sh -xe /tmp/jenkins4864556993423806461.sh
> > jenkins  18243  0.0  0.0  11256  3104 ?        S    02:05   0:12 /bin/bash /fuego-core/scripts/poll_requests.sh -u -d 12h -w 120
> > ok 1 - process list of docker container
> > ######### git describe of fuego-core
> > v1.5.9-16-gcb0dd1f
> > ok 2 - git describe of fuego-core
> > ######### Installed tests - outside of fuego-core
> > ==== ls -lad /fuego-rw
> > drwxr-xr-x 8 root root 4096 Mar 16 06:42 /fuego-rw
> > ==== ls -lad /fuego-rw/tests
> > drwxr-xr-x 2 root root 4096 Mar 16 06:58 /fuego-rw/tests
> > ==== ls -laR /fuego-rw/tests
> > /fuego-rw/tests:
> > total 8
> > drwxr-xr-x 2 root root 4096 Mar 16 06:58 .
> > drwxr-xr-x 8 root root 4096 Mar 16 06:42 ..
> > ==== ls -lad /fuego-core/tests
> > drwxr-xr-x 197 root root 12288 Jan 12 10:17 /fuego-core/tests
> > ok 3 - remotely installed tests
> > ######### Console log of the last run
> > # last run=/fuego-rw/logs/Functional.hello_world/MINNOW.default.10.10
> > # Log:
> > # ===== doing fuego phase: pre_test =====
> > # ===== doing fuego phase: build =====
> > # The test is already built
> > # Fuego test_build duration=0 seconds
> > # ===== doing fuego phase: deploy =====
> > # ===== doing fuego phase: snapshot =====
> > # Firmware revision: 5.4.81-intel-pk-standard
> > # ===== doing fuego phase: run =====
> > # -------------------------------------------------
> > # Hello World!
> > #
> > # This is a test program, to demonstrate test success and failure
> > # Test result: SUCCESS
> > # -------------------------------------------------
> > # ===== doing fuego phase: post_test =====
> > # ===== doing fuego phase: processing =====
> > # log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal than 1)
> > # chart config not found. Using default values.
> > # Fuego: requested test phases complete!
> > ######### end of log
> > ok 4 - console log of last test run
> > -------------------------------------------------
> > ===== doing fuego phase: post_test =====
> > scp: /home/pi/test/fuego.Functional.fuego_lab_check/Functional.fuego_lab_check.log: No such file or directory
> > INFO: the test did not produce a test log on the target
> > ===== doing fuego phase: processing =====
> > chart config not found. Using default values.
> > Fuego: requested test phases complete!
> > Packaging run 'Functional.fuego_lab_check-default-11-rpi'
> > Run packaged successfully, and is at: /tmp/run-Functional.fuego_lab_check-default-11-on-pptlab:rpi.frp
> > Run package run-Functional.fuego_lab_check-default-11-on-pptlab:rpi.frp was accepted by the server.
> > Run Functional.fuego_lab_check-default-11-rpi put to server
> >
> > Thanks and regards
> >
> > Pavan Arun Deshpande
> >
> >
> >
> > On Mon, Mar 15, 2021 at 8:56 PM Srivatsan S <srivatsan.s at pathpartnertech.com <mailto:srivatsan.s at pathpartnertech.com> > wrote:
> >
> >
> > 	Tim,
> >
> > 	We couldn't do a git pull today to get the latest changes as we had some local changes to get the raspi_kinstall working. Currently,
> > we have moved to a 64-bit kernel image for rpi board and have made some local changes to the script in Functional.raspi_kinstall to get
> it
> > working with a 64-bit kernel image.
> >
> > 	We will do a git pull tomorrow and run our changes on top of it and get back to you.
> >
> > 	Thanks,
> > 	Srivatsan
> >
> >
> > 	On Mon, Mar 15, 2021 at 8:23 PM <Tim.Bird at sony.com <mailto:Tim.Bird at sony.com> > wrote:
> >
> >
> > 		This is good to hear.  Thanks for getting this resolved.
> > 		The errors that Fuego was returning were not very useful
> > 		for determining this problem.  I'd like to think about how
> > 		this condition could be checked for, before or during test
> > 		execution, and reported back with a more useful error message.
> >
> > 		> # Version of this test=1.2-4
> > 		It looks like you are still running an old version of fuego_lab_check.
> > 		Can you please 'git pull' the latest master branch of fuego-core?
> >
> > 		I plan to do more testing this week.
> >
> > 		Thanks.
> > 		 -- Tim
> >
> >
> > 		> -----Original Message-----
> > 		> From: Pavan Arun Deshpande <pavan.ad at pathpartnertech.com <mailto:pavan.ad at pathpartnertech.com> >
> > 		>
> > 		> Hi Tim,
> > 		>
> > 		> The issue regarding Functional.fuego_lab_check has been resolved.
> > 		> The issue was due to the overlapping build numbers as concerned files were already present in the container.
> > 		> Therefore the server was giving an error message saying that " Already have a file cannot proceed".
> > 		> Uploading failed because of the above mentioned issue.
> > 		> I was able to resolve this after removing the previous run files.
> > 		>
> > 		> Functional.reboot also had the same issue and this also has been resolved.
> > 		>
> > 		> This has been verified with multiple tests and iterations.
> > 		>
> > 		> Please find a copy of log messages generated while running Functional.fuego_lab_check
> > 		>
> > 		> ===== doing fuego phase: pre_test =====
> > 		> ===== doing fuego phase: build =====
> > 		> The test is already built
> > 		> Fuego test_build duration=0 seconds
> > 		> ===== doing fuego phase: deploy =====
> > 		> ===== doing fuego phase: snapshot =====
> > 		> Firmware revision: 5.4.83-v8-93349cdff-1+
> > 		> ===== doing fuego phase: run =====
> > 		> -------------------------------------------------
> > 		> TAP version 13
> > 		> 1..1
> > 		> # Version of this test=1.2-4
> > 		> # Location of this test=/fuego-rw/tests/Functional.fuego_lab_check
> > 		> ######### Process list of docker container:
> > 		> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> > 		> root         1  0.0  0.0  19424  4544 pts/0    Ss+  11:25   0:00 /bin/bash
> > 		> jenkins     58  0.0  0.0  18708   184 ?        S    11:25   0:00 /usr/bin/daemon --name=jenkins --inherit --
> > env=JENKINS_HOME=/var/lib/jenkins
> > 		> --output=/var/log/jenkins/jenkins.log --pidfile=/var/run/jenkins/jenkins.pid -- /usr/bin/java -Djava.awt.headless=true -
> > 		> Djenkins.install.runSetupWizard=false -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar
> > 		> /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8090 --prefix=/fuego
> > 		> jenkins     60  2.4 10.8 9946068 1769860 ?     Sl   11:25   1:31 /usr/bin/java -Djava.awt.headless=true -
> > 		> Djenkins.install.runSetupWizard=false -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar
> > 		> /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8090 --prefix=/fuego
> > 		> root       110  0.0  0.0   9588   144 ?        Ss   11:25   0:00 /usr/bin/netserver
> > 		> root       129  0.0  0.0   9440    96 ?        Ss   11:25   0:00 iperf3 -V -s -D -f M
> > 		> jenkins    176  0.6  0.9 8487928 152384 ?      Sl   11:25   0:23 java -jar /fuego-core/slave.jar
> > 		> jenkins    177  0.6  0.9 8421364 150824 ?      Sl   11:25   0:24 java -jar /fuego-core/slave.jar
> > 		> jenkins    336  0.0  0.0  11180  2852 ?        S    11:27   0:00 /bin/sh -xe /tmp/jenkins184800571287563138.sh
> > 		> jenkins    340  0.0  0.0  11256  3104 ?        S    11:27   0:01 /bin/bash /fuego-core/scripts/poll_requests.sh -u -d 12h -w
> > 120
> > 		> jenkins   4769  0.6  0.3  99016 56048 ?        S    12:28   0:00 /usr/bin/python /usr/local/bin/ftc run-request --put-run --
> > allow-upgrade
> > 		> request-2021-03-15_12:27:36.07-pptlab:rpi
> > 		> jenkins   4772  0.3  0.0  12404  4168 ?        S    12:28   0:00 /bin/bash -e /fuego-core/scripts/main.sh
> > 		> jenkins   4855  0.0  0.0  11192  2848 ?        S    12:28   0:00 bash -c { ps -aux; echo $? > /tmp/4772-7340 ; } 2>&1 | tee -
> > a /fuego-
> > 		> rw/logs/Functional.fuego_lab_check/rpi.default.10.10/hostlog.txt
> > 		> jenkins   4856  0.0  0.0  11192   272 ?        S    12:28   0:00 bash -c { ps -aux; echo $? > /tmp/4772-7340 ; } 2>&1 | tee -a
> > /fuego-
> > 		> rw/logs/Functional.fuego_lab_check/rpi.default.10.10/hostlog.txt
> > 		> jenkins   4857  0.0  0.0   5832   656 ?        S    12:28   0:00 tee -a /fuego-
> > rw/logs/Functional.fuego_lab_check/rpi.default.10.10/hostlog.txt
> > 		> jenkins   4858  0.0  0.0  38304  3316 ?        R    12:28   0:00 ps -aux
> > 		> ok 1 - process list of docker container
> > 		> ######### git describe of fuego-core
> > 		> v1.5.9-9-g79f9df5
> > 		> ok 2 - git describe of fuego-core
> > 		> ######### Installed tests - outside of fuego-core
> > 		> total 4
> > 		> drwxr-xr-x 2 root root 4096 Jan 29 04:40 Functional.fuego_lab_check
> > 		> ok 3 - remotely installed tests
> > 		> ######### Console log of the last run
> > 		> # last run=/fuego-rw/logs/Functional.fuego_board_check/rpi.default.91.91
> > 		> # Log:
> > 		> # ===== doing fuego phase: pre_test =====
> > 		> # ===== doing fuego phase: build =====
> > 		> # The test is already built
> > 		> # Fuego test_build duration=0 seconds
> > 		> # ===== doing fuego phase: deploy =====
> > 		> # ===== doing fuego phase: snapshot =====
> > 		> # Firmware revision: 5.4.83-v8-93349cdff-1+
> > 		> # ===== doing fuego phase: run =====
> > 		> # -------------------------------------------------
> > 		> # == Fuego target feature scan results ==
> > 		> # Information:
> > 		> #   account information: uid=0(root) gid=0(root) groups=0(root)
> > 		> #   working directory: /home/pi/test/fuego.Functional.fuego_board_check
> > 		> #   PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> > 		> # -------------------------------------
> > 		> # Scan results:
> > 		> # HAS_PROGRAM_CAT=1
> > 		> # HAS_PROGRAM_DF=1
> > 		> # HAS_PROGRAM_FIND=1
> > 		> # HAS_PROGRAM_FREE=1
> > 		> # HAS_PROGRAM_GREP=1
> > 		> # HAS_PROGRAM_HEAD=1
> > 		> # HAS_PROGRAM_LOGGER=1
> > 		> # HAS_PROGRAM_LOGREAD=0
> > 		> # HAS_PROGRAM_MKDIR=1
> > 		> # HAS_PROGRAM_MOUNT=1
> > 		> # HAS_PROGRAM_PS=1
> > 		> # HAS_PROGRAM_RM=1
> > 		> # HAS_PROGRAM_RMDIR=1
> > 		> # HAS_PROGRAM_SYNC=1
> > 		> # HAS_PROGRAM_TAIL=1
> > 		> # HAS_PROGRAM_TEE=1
> > 		> # HAS_PROGRAM_TOUCH=1
> > 		> # HAS_PROGRAM_TRUE=1
> > 		> # HAS_PROGRAM_UMOUNT=1
> > 		> # HAS_PROGRAM_UNAME=1
> > 		> # HAS_PROGRAM_UPTIME=1
> > 		> # HAS_PROGRAM_XARGS=1
> > 		> # HAS_PROGRAM_BRACKET=1
> > 		> # HAS_MKDIR_P=1
> > 		> # HAS_RM_RF=1
> > 		> # HAS_GREP_F=1
> > 		> # HAS_SBIN_REBOOT=1
> > 		> # HAS_BIN_TRUE=1
> > 		> # HAS_SBIN_ROUTE=1
> > 		> # HAS_VAR_LOG_MESSAGES=0
> > 		> # HAS_VAR_LOG_SYSLOG=1
> > 		> # HAS_PROC_INTERRUPTS=1
> > 		> # HAS_PROC_DROP_CACHES=1
> > 		> # HAS_PROC_OOM_SCORE_ADJ=0
> > 		> # HAS_PROC_CONFIG_GZ=0
> > 		> # HAS_MEMORY=935532 K
> > 		> # HAS_STORAGE=15022716 K
> > 		> # Process 1 is program: /sbin/init
> > 		> # -------------------------------------
> > 		> # Based on this scan,
> > 		> # the recommended DISTRIB value should be:
> > 		> #    nosyslogd.dist
> > 		> # -------------------------------------
> > 		> # RESULT=SUCCESS
> > 		> # All required programs, files, features, and permissions are present
> > 		> # -------------------------------------
> > 		> # 10+0 records in
> > 		> # 10+0 records out
> > 		> # 10000 bytes (10 kB, 9.8 KiB) copied, 0.000395734 s, 25.3 MB/s
> > 		> # Data from host:
> > 		> # TRANSPORT=ssh
> > 		> # Time to put 10K file: 0.68
> > 		> # Time to get 10K file: 0.69
> > 		> # Time to do 10 cmds: 6.74
> > 		> # Average time per cmd: 0.67
> > 		> # -------------------------------------------------
> > 		> # ===== doing fuego phase: post_test =====
> > 		> # ===== doing fuego phase: processing =====
> > 		> # log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal than 1)
> > 		> # chart config not found. Using default values.
> > 		> # Fuego: requested test phases complete!
> > 		> ######### end of log
> > 		> ok 4 - console log of last test run
> > 		> -------------------------------------------------
> > 		> ===== doing fuego phase: post_test =====
> > 		> scp: /home/pi/test/fuego.Functional.fuego_lab_check/Functional.fuego_lab_check.log: No such file or directory
> > 		> INFO: the test did not produce a test log on the target
> > 		> ===== doing fuego phase: processing =====
> > 		> chart config not found. Using default values.
> > 		> Fuego: requested test phases complete!
> > 		> Packaging run 'Functional.fuego_lab_check-default-10-rpi'
> > 		> Run packaged successfully, and is at: /tmp/run-Functional.fuego_lab_check-default-10-on-pptlab:rpi.frp
> > 		> Run package run-Functional.fuego_lab_check-default-10-on-pptlab:rpi.frp was accepted by the server.
> > 		> Run Functional.fuego_lab_check-default-10-rpi put to server
> > 		>
> > 		> Thanks and regards
> > 		>
> > 		> Pavan Arun deshpande
> > 		>
> > 		>
> > 		> On Sat, Mar 13, 2021 at 3:54 AM <Tim.Bird at sony.com <mailto:Tim.Bird at sony.com>  <mailto:Tim.Bird at sony.com
> > <mailto:Tim.Bird at sony.com> > > wrote:
> > 		>
> > 		>
> > 		>       There is a problem with running the test: Functional.fuego_lab_check
> > 		>       in your lab.  I'll send the error in another message.
> > 		>
> > 		>       Here is the error from the testlog:
> > 		>
> > 		>       ls: cannot access '/fuego-rw/tests': No such file or directory
> > 		>       ----
> > 		>
> > 		>       And here is the report from the console log:
> > 		>
> > 		>       ######### Installed tests (outside of fuego-core)
> > 		>       ls: cannot access '/fuego-rw/tests': No such file or directory
> > 		>       In Fuego signal_handler (due to error in run phase)
> > 		>       ##### doing fuego phase: post_test (from signal handler) #####
> > 		>       ...
> > 		>       -----
> > 		>
> > 		>       This output appears to be from an older version of Functional.fuego_lab_check.
> > 		>       One of the things I am testing with this test is the ability to update the version
> > 		>       of a test in your lab.  Since there's a problem with the /fuego-rw/tests directory,
> > 		>       this operation is not working.
> > 		>
> > 		>       Can you please do 3 things:
> > 		>        1) please check your docker container and make sure that it has the directory
> > 		>       /fuego-rw/tests
> > 		>       and that this directory is owned by user jenkins and group jenkins?
> > 		>
> > 		>       2) can you 'git pull' the latest fuego-core from the master repository, to pick
> > 		>       up the latest version of the Functional.fuego_lab_check test?
> > 		>
> > 		>       Please make sure that you can run this test locally (from the /fuego-core/tests
> > 		>       directory).  That is, just run this test from the updated fuego-core repository
> > 		>       in your lab, and confirm that it runs without error.
> > 		>
> > 		>       Once you have done that, I will try some tests to see if I can cause your lab
> > 		>       to run upgraded tests.
> > 		>
> > 		>       Thanks,
> > 		>        -- Tim
> > 		>
> > 		>
> > 		>
> > 		>
> > 		>
> > 		> This message contains confidential information and is intended only for the individual(s) named. If you are not the
> > intended recipient, you
> > 		> are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and
> > attached file/s is strictly
> > 		> prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be
> > guaranteed to be
> > 		> secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
> > contain viruses. The
> > 		> sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a
> > result of e-mail
> > 		> transmission.
> >
> >
> >
> >
> > This message contains confidential information and is intended only for the individual(s) named. If you are not the intended recipient,
> you
> > are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and attached file/s is strictly
> > prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be guaranteed to
> be
> > secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The
> > sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail
> > transmission.
> 
> _______________________________________________
> Fuego mailing list
> Fuego at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego


More information about the Fuego mailing list