[Fuego] Problem with /fuego-rw/tests in your lab
Tim.Bird at sony.com
Tim.Bird at sony.com
Tue Mar 16 17:52:22 UTC 2021
Actually a slight correction below...
> -----Original Message-----
> From: Fuego <fuego-bounces at lists.linuxfoundation.org> On Behalf Of Tim.Bird at sony.com
>
> OK. thanks for this update. This is very useful.
>
> Based on your experience here, I believe I should change the logic for test selection
> for 'ftc run-test'.
That should be: "for 'ftc run-request'".
> Right now, if the test is defined in both /fuego-rw/tests and /fuego-core/tests,
> then Fuego runs the one from /fuego-rw/tests. However, I think that it would be more
> correct to run the one with the latest version (if the test request specifies 'any' or 'latest', for
> the request test version).
>
> Looking at the code, it appears I'll have to do some refactoring to support this behavior.
>
> I try to work on that this week. Thanks again for the report.
> -- Tim
>
> > -----Original Message-----
> > From: Pavan Arun Deshpande <pavan.ad at pathpartnertech.com>
> >
> > Hi Tim,
> >
> > Pulled latest fuego-core sources from the repository
> >
> > fuego_lab_check in fuego-rw/tests directory belonged to an older version and it was picking that for testing as ftc checks for its presence
> in
> > fuego-rw/tests directory.
> >
> > when I removed fuego_lab_check from fuego-rw/tests directory and ran the fuego_lab_check , then it ran the testcase from fuego-
> > core/tests directory which is of the latest version.
> >
> > Please find a copy of log messages generated while running Functional.fuego_lab_check.
> >
> > ===== doing fuego phase: pre_test =====
> > ===== doing fuego phase: build =====
> > The test is already built
> > Fuego test_build duration=0 seconds
> > ===== doing fuego phase: deploy =====
> > ===== doing fuego phase: snapshot =====
> > Firmware revision: 5.4.83-v8-93349cdff-2+
> > ===== doing fuego phase: run =====
> > -------------------------------------------------
> > TAP version 13
> > 1..1
> > # Version of this test=1.4-2
> > # Version of this test (hardcoded)=1.4-2
> > # Location of this test=/fuego-core/tests/Functional.fuego_lab_check
> > ######### Process list of docker container:
> > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > root 1 0.0 0.0 19456 3836 pts/0 Ss+ Mar15 0:00 /bin/bash
> > jenkins 58 0.0 0.0 18708 0 ? S Mar15 0:00 /usr/bin/daemon --name=jenkins --inherit --
> env=JENKINS_HOME=/var/lib/jenkins
> > --output=/var/log/jenkins/jenkins.log --pidfile=/var/run/jenkins/jenkins.pid -- /usr/bin/java -Djava.awt.headless=true -
> > Djenkins.install.runSetupWizard=false -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar
> > /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8090 --prefix=/fuego
> > jenkins 60 0.7 8.7 9950392 1422876 ? Sl Mar15 10:01 /usr/bin/java -Djava.awt.headless=true -
> > Djenkins.install.runSetupWizard=false -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar
> > /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8090 --prefix=/fuego
> > root 110 0.0 0.0 9588 0 ? Ss Mar15 0:00 /usr/bin/netserver
> > root 129 0.0 0.0 9440 8 ? Ss Mar15 0:00 iperf3 -V -s -D -f M
> > jenkins 176 0.5 1.0 8487928 166808 ? Sl Mar15 7:09 java -jar /fuego-core/slave.jar
> > jenkins 177 0.5 1.1 8425932 190836 ? Sl Mar15 7:13 java -jar /fuego-core/slave.jar
> > jenkins 13870 0.9 0.3 99016 55856 ? S 10:50 0:00 /usr/bin/python /usr/local/bin/ftc run-request --put-run --allow-upgrade
> > request-2021-03-16_10:48:31.97-pptlab:rpi
> > jenkins 13873 0.4 0.0 12416 4248 ? S 10:50 0:00 /bin/bash -e /fuego-core/scripts/main.sh
> > jenkins 13962 0.0 0.0 11192 3008 ? S 10:50 0:00 bash -c { ps -aux; echo $? > /tmp/13873-4538 ; } 2>&1 | tee -a /fuego-
> > rw/logs/Functional.fuego_lab_check/rpi.default.11.11/hostlog.txt
> > jenkins 13963 0.0 0.0 11192 272 ? S 10:50 0:00 bash -c { ps -aux; echo $? > /tmp/13873-4538 ; } 2>&1 | tee -a /fuego-
> > rw/logs/Functional.fuego_lab_check/rpi.default.11.11/hostlog.txt
> > jenkins 13964 0.0 0.0 5832 708 ? S 10:50 0:00 tee -a /fuego-
> rw/logs/Functional.fuego_lab_check/rpi.default.11.11/hostlog.txt
> > jenkins 13965 0.0 0.0 38304 3368 ? R 10:50 0:00 ps -aux
> > jenkins 18240 0.0 0.0 11180 2960 ? S 02:05 0:00 /bin/sh -xe /tmp/jenkins4864556993423806461.sh
> > jenkins 18243 0.0 0.0 11256 3104 ? S 02:05 0:12 /bin/bash /fuego-core/scripts/poll_requests.sh -u -d 12h -w 120
> > ok 1 - process list of docker container
> > ######### git describe of fuego-core
> > v1.5.9-16-gcb0dd1f
> > ok 2 - git describe of fuego-core
> > ######### Installed tests - outside of fuego-core
> > ==== ls -lad /fuego-rw
> > drwxr-xr-x 8 root root 4096 Mar 16 06:42 /fuego-rw
> > ==== ls -lad /fuego-rw/tests
> > drwxr-xr-x 2 root root 4096 Mar 16 06:58 /fuego-rw/tests
> > ==== ls -laR /fuego-rw/tests
> > /fuego-rw/tests:
> > total 8
> > drwxr-xr-x 2 root root 4096 Mar 16 06:58 .
> > drwxr-xr-x 8 root root 4096 Mar 16 06:42 ..
> > ==== ls -lad /fuego-core/tests
> > drwxr-xr-x 197 root root 12288 Jan 12 10:17 /fuego-core/tests
> > ok 3 - remotely installed tests
> > ######### Console log of the last run
> > # last run=/fuego-rw/logs/Functional.hello_world/MINNOW.default.10.10
> > # Log:
> > # ===== doing fuego phase: pre_test =====
> > # ===== doing fuego phase: build =====
> > # The test is already built
> > # Fuego test_build duration=0 seconds
> > # ===== doing fuego phase: deploy =====
> > # ===== doing fuego phase: snapshot =====
> > # Firmware revision: 5.4.81-intel-pk-standard
> > # ===== doing fuego phase: run =====
> > # -------------------------------------------------
> > # Hello World!
> > #
> > # This is a test program, to demonstrate test success and failure
> > # Test result: SUCCESS
> > # -------------------------------------------------
> > # ===== doing fuego phase: post_test =====
> > # ===== doing fuego phase: processing =====
> > # log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal than 1)
> > # chart config not found. Using default values.
> > # Fuego: requested test phases complete!
> > ######### end of log
> > ok 4 - console log of last test run
> > -------------------------------------------------
> > ===== doing fuego phase: post_test =====
> > scp: /home/pi/test/fuego.Functional.fuego_lab_check/Functional.fuego_lab_check.log: No such file or directory
> > INFO: the test did not produce a test log on the target
> > ===== doing fuego phase: processing =====
> > chart config not found. Using default values.
> > Fuego: requested test phases complete!
> > Packaging run 'Functional.fuego_lab_check-default-11-rpi'
> > Run packaged successfully, and is at: /tmp/run-Functional.fuego_lab_check-default-11-on-pptlab:rpi.frp
> > Run package run-Functional.fuego_lab_check-default-11-on-pptlab:rpi.frp was accepted by the server.
> > Run Functional.fuego_lab_check-default-11-rpi put to server
> >
> > Thanks and regards
> >
> > Pavan Arun Deshpande
> >
> >
> >
> > On Mon, Mar 15, 2021 at 8:56 PM Srivatsan S <srivatsan.s at pathpartnertech.com <mailto:srivatsan.s at pathpartnertech.com> > wrote:
> >
> >
> > Tim,
> >
> > We couldn't do a git pull today to get the latest changes as we had some local changes to get the raspi_kinstall working. Currently,
> > we have moved to a 64-bit kernel image for rpi board and have made some local changes to the script in Functional.raspi_kinstall to get
> it
> > working with a 64-bit kernel image.
> >
> > We will do a git pull tomorrow and run our changes on top of it and get back to you.
> >
> > Thanks,
> > Srivatsan
> >
> >
> > On Mon, Mar 15, 2021 at 8:23 PM <Tim.Bird at sony.com <mailto:Tim.Bird at sony.com> > wrote:
> >
> >
> > This is good to hear. Thanks for getting this resolved.
> > The errors that Fuego was returning were not very useful
> > for determining this problem. I'd like to think about how
> > this condition could be checked for, before or during test
> > execution, and reported back with a more useful error message.
> >
> > > # Version of this test=1.2-4
> > It looks like you are still running an old version of fuego_lab_check.
> > Can you please 'git pull' the latest master branch of fuego-core?
> >
> > I plan to do more testing this week.
> >
> > Thanks.
> > -- Tim
> >
> >
> > > -----Original Message-----
> > > From: Pavan Arun Deshpande <pavan.ad at pathpartnertech.com <mailto:pavan.ad at pathpartnertech.com> >
> > >
> > > Hi Tim,
> > >
> > > The issue regarding Functional.fuego_lab_check has been resolved.
> > > The issue was due to the overlapping build numbers as concerned files were already present in the container.
> > > Therefore the server was giving an error message saying that " Already have a file cannot proceed".
> > > Uploading failed because of the above mentioned issue.
> > > I was able to resolve this after removing the previous run files.
> > >
> > > Functional.reboot also had the same issue and this also has been resolved.
> > >
> > > This has been verified with multiple tests and iterations.
> > >
> > > Please find a copy of log messages generated while running Functional.fuego_lab_check
> > >
> > > ===== doing fuego phase: pre_test =====
> > > ===== doing fuego phase: build =====
> > > The test is already built
> > > Fuego test_build duration=0 seconds
> > > ===== doing fuego phase: deploy =====
> > > ===== doing fuego phase: snapshot =====
> > > Firmware revision: 5.4.83-v8-93349cdff-1+
> > > ===== doing fuego phase: run =====
> > > -------------------------------------------------
> > > TAP version 13
> > > 1..1
> > > # Version of this test=1.2-4
> > > # Location of this test=/fuego-rw/tests/Functional.fuego_lab_check
> > > ######### Process list of docker container:
> > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> > > root 1 0.0 0.0 19424 4544 pts/0 Ss+ 11:25 0:00 /bin/bash
> > > jenkins 58 0.0 0.0 18708 184 ? S 11:25 0:00 /usr/bin/daemon --name=jenkins --inherit --
> > env=JENKINS_HOME=/var/lib/jenkins
> > > --output=/var/log/jenkins/jenkins.log --pidfile=/var/run/jenkins/jenkins.pid -- /usr/bin/java -Djava.awt.headless=true -
> > > Djenkins.install.runSetupWizard=false -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar
> > > /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8090 --prefix=/fuego
> > > jenkins 60 2.4 10.8 9946068 1769860 ? Sl 11:25 1:31 /usr/bin/java -Djava.awt.headless=true -
> > > Djenkins.install.runSetupWizard=false -Dhudson.model.DirectoryBrowserSupport.allowSymlinkEscape=true -jar
> > > /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8090 --prefix=/fuego
> > > root 110 0.0 0.0 9588 144 ? Ss 11:25 0:00 /usr/bin/netserver
> > > root 129 0.0 0.0 9440 96 ? Ss 11:25 0:00 iperf3 -V -s -D -f M
> > > jenkins 176 0.6 0.9 8487928 152384 ? Sl 11:25 0:23 java -jar /fuego-core/slave.jar
> > > jenkins 177 0.6 0.9 8421364 150824 ? Sl 11:25 0:24 java -jar /fuego-core/slave.jar
> > > jenkins 336 0.0 0.0 11180 2852 ? S 11:27 0:00 /bin/sh -xe /tmp/jenkins184800571287563138.sh
> > > jenkins 340 0.0 0.0 11256 3104 ? S 11:27 0:01 /bin/bash /fuego-core/scripts/poll_requests.sh -u -d 12h -w
> > 120
> > > jenkins 4769 0.6 0.3 99016 56048 ? S 12:28 0:00 /usr/bin/python /usr/local/bin/ftc run-request --put-run --
> > allow-upgrade
> > > request-2021-03-15_12:27:36.07-pptlab:rpi
> > > jenkins 4772 0.3 0.0 12404 4168 ? S 12:28 0:00 /bin/bash -e /fuego-core/scripts/main.sh
> > > jenkins 4855 0.0 0.0 11192 2848 ? S 12:28 0:00 bash -c { ps -aux; echo $? > /tmp/4772-7340 ; } 2>&1 | tee -
> > a /fuego-
> > > rw/logs/Functional.fuego_lab_check/rpi.default.10.10/hostlog.txt
> > > jenkins 4856 0.0 0.0 11192 272 ? S 12:28 0:00 bash -c { ps -aux; echo $? > /tmp/4772-7340 ; } 2>&1 | tee -a
> > /fuego-
> > > rw/logs/Functional.fuego_lab_check/rpi.default.10.10/hostlog.txt
> > > jenkins 4857 0.0 0.0 5832 656 ? S 12:28 0:00 tee -a /fuego-
> > rw/logs/Functional.fuego_lab_check/rpi.default.10.10/hostlog.txt
> > > jenkins 4858 0.0 0.0 38304 3316 ? R 12:28 0:00 ps -aux
> > > ok 1 - process list of docker container
> > > ######### git describe of fuego-core
> > > v1.5.9-9-g79f9df5
> > > ok 2 - git describe of fuego-core
> > > ######### Installed tests - outside of fuego-core
> > > total 4
> > > drwxr-xr-x 2 root root 4096 Jan 29 04:40 Functional.fuego_lab_check
> > > ok 3 - remotely installed tests
> > > ######### Console log of the last run
> > > # last run=/fuego-rw/logs/Functional.fuego_board_check/rpi.default.91.91
> > > # Log:
> > > # ===== doing fuego phase: pre_test =====
> > > # ===== doing fuego phase: build =====
> > > # The test is already built
> > > # Fuego test_build duration=0 seconds
> > > # ===== doing fuego phase: deploy =====
> > > # ===== doing fuego phase: snapshot =====
> > > # Firmware revision: 5.4.83-v8-93349cdff-1+
> > > # ===== doing fuego phase: run =====
> > > # -------------------------------------------------
> > > # == Fuego target feature scan results ==
> > > # Information:
> > > # account information: uid=0(root) gid=0(root) groups=0(root)
> > > # working directory: /home/pi/test/fuego.Functional.fuego_board_check
> > > # PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> > > # -------------------------------------
> > > # Scan results:
> > > # HAS_PROGRAM_CAT=1
> > > # HAS_PROGRAM_DF=1
> > > # HAS_PROGRAM_FIND=1
> > > # HAS_PROGRAM_FREE=1
> > > # HAS_PROGRAM_GREP=1
> > > # HAS_PROGRAM_HEAD=1
> > > # HAS_PROGRAM_LOGGER=1
> > > # HAS_PROGRAM_LOGREAD=0
> > > # HAS_PROGRAM_MKDIR=1
> > > # HAS_PROGRAM_MOUNT=1
> > > # HAS_PROGRAM_PS=1
> > > # HAS_PROGRAM_RM=1
> > > # HAS_PROGRAM_RMDIR=1
> > > # HAS_PROGRAM_SYNC=1
> > > # HAS_PROGRAM_TAIL=1
> > > # HAS_PROGRAM_TEE=1
> > > # HAS_PROGRAM_TOUCH=1
> > > # HAS_PROGRAM_TRUE=1
> > > # HAS_PROGRAM_UMOUNT=1
> > > # HAS_PROGRAM_UNAME=1
> > > # HAS_PROGRAM_UPTIME=1
> > > # HAS_PROGRAM_XARGS=1
> > > # HAS_PROGRAM_BRACKET=1
> > > # HAS_MKDIR_P=1
> > > # HAS_RM_RF=1
> > > # HAS_GREP_F=1
> > > # HAS_SBIN_REBOOT=1
> > > # HAS_BIN_TRUE=1
> > > # HAS_SBIN_ROUTE=1
> > > # HAS_VAR_LOG_MESSAGES=0
> > > # HAS_VAR_LOG_SYSLOG=1
> > > # HAS_PROC_INTERRUPTS=1
> > > # HAS_PROC_DROP_CACHES=1
> > > # HAS_PROC_OOM_SCORE_ADJ=0
> > > # HAS_PROC_CONFIG_GZ=0
> > > # HAS_MEMORY=935532 K
> > > # HAS_STORAGE=15022716 K
> > > # Process 1 is program: /sbin/init
> > > # -------------------------------------
> > > # Based on this scan,
> > > # the recommended DISTRIB value should be:
> > > # nosyslogd.dist
> > > # -------------------------------------
> > > # RESULT=SUCCESS
> > > # All required programs, files, features, and permissions are present
> > > # -------------------------------------
> > > # 10+0 records in
> > > # 10+0 records out
> > > # 10000 bytes (10 kB, 9.8 KiB) copied, 0.000395734 s, 25.3 MB/s
> > > # Data from host:
> > > # TRANSPORT=ssh
> > > # Time to put 10K file: 0.68
> > > # Time to get 10K file: 0.69
> > > # Time to do 10 cmds: 6.74
> > > # Average time per cmd: 0.67
> > > # -------------------------------------------------
> > > # ===== doing fuego phase: post_test =====
> > > # ===== doing fuego phase: processing =====
> > > # log_compare: pattern 'SUCCESS' found 1 times (expected greater or equal than 1)
> > > # chart config not found. Using default values.
> > > # Fuego: requested test phases complete!
> > > ######### end of log
> > > ok 4 - console log of last test run
> > > -------------------------------------------------
> > > ===== doing fuego phase: post_test =====
> > > scp: /home/pi/test/fuego.Functional.fuego_lab_check/Functional.fuego_lab_check.log: No such file or directory
> > > INFO: the test did not produce a test log on the target
> > > ===== doing fuego phase: processing =====
> > > chart config not found. Using default values.
> > > Fuego: requested test phases complete!
> > > Packaging run 'Functional.fuego_lab_check-default-10-rpi'
> > > Run packaged successfully, and is at: /tmp/run-Functional.fuego_lab_check-default-10-on-pptlab:rpi.frp
> > > Run package run-Functional.fuego_lab_check-default-10-on-pptlab:rpi.frp was accepted by the server.
> > > Run Functional.fuego_lab_check-default-10-rpi put to server
> > >
> > > Thanks and regards
> > >
> > > Pavan Arun deshpande
> > >
> > >
> > > On Sat, Mar 13, 2021 at 3:54 AM <Tim.Bird at sony.com <mailto:Tim.Bird at sony.com> <mailto:Tim.Bird at sony.com
> > <mailto:Tim.Bird at sony.com> > > wrote:
> > >
> > >
> > > There is a problem with running the test: Functional.fuego_lab_check
> > > in your lab. I'll send the error in another message.
> > >
> > > Here is the error from the testlog:
> > >
> > > ls: cannot access '/fuego-rw/tests': No such file or directory
> > > ----
> > >
> > > And here is the report from the console log:
> > >
> > > ######### Installed tests (outside of fuego-core)
> > > ls: cannot access '/fuego-rw/tests': No such file or directory
> > > In Fuego signal_handler (due to error in run phase)
> > > ##### doing fuego phase: post_test (from signal handler) #####
> > > ...
> > > -----
> > >
> > > This output appears to be from an older version of Functional.fuego_lab_check.
> > > One of the things I am testing with this test is the ability to update the version
> > > of a test in your lab. Since there's a problem with the /fuego-rw/tests directory,
> > > this operation is not working.
> > >
> > > Can you please do 3 things:
> > > 1) please check your docker container and make sure that it has the directory
> > > /fuego-rw/tests
> > > and that this directory is owned by user jenkins and group jenkins?
> > >
> > > 2) can you 'git pull' the latest fuego-core from the master repository, to pick
> > > up the latest version of the Functional.fuego_lab_check test?
> > >
> > > Please make sure that you can run this test locally (from the /fuego-core/tests
> > > directory). That is, just run this test from the updated fuego-core repository
> > > in your lab, and confirm that it runs without error.
> > >
> > > Once you have done that, I will try some tests to see if I can cause your lab
> > > to run upgraded tests.
> > >
> > > Thanks,
> > > -- Tim
> > >
> > >
> > >
> > >
> > >
> > > This message contains confidential information and is intended only for the individual(s) named. If you are not the
> > intended recipient, you
> > > are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and
> > attached file/s is strictly
> > > prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be
> > guaranteed to be
> > > secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
> > contain viruses. The
> > > sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a
> > result of e-mail
> > > transmission.
> >
> >
> >
> >
> > This message contains confidential information and is intended only for the individual(s) named. If you are not the intended recipient,
> you
> > are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this mail and attached file/s is strictly
> > prohibited. Please notify the sender immediately and delete this e-mail from your system. E-mail transmission cannot be guaranteed to
> be
> > secured or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The
> > sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail
> > transmission.
>
> _______________________________________________
> Fuego mailing list
> Fuego at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego
More information about the Fuego
mailing list