[Fuego] Integration of Fuego and Linaro test-definitons

Chase Qi chase.qi at linaro.org
Thu Feb 14 08:27:01 UTC 2019


Hi Daniel,

Thanks for the comments.

On Thu, Feb 14, 2019 at 9:53 AM <daniel.sangorrin at toshiba.co.jp> wrote:
>
> Hi Chase,
>
> Thanks for your advice and comments. I have created a new thread for discussing the integration of Fuego and Linaro test-definitions.
>
> > From: Chase Qi <chase.qi at linaro.org>
> > BTW, I see you also started working on running fuego tests with LAVA.
> > I did some investigation before Chinese New Year holiday. Here are my
> > findings:
> >
> > * fuego is very much docker and jenkins depended, it is not possible,
> > at least no easy way to run without them.
> > * it is possible to run fuego tests from command line.
>
> I agree that there is no easy way.
>
> Fuego depends on Jenkins for serializing/scheduling tests to boards/nodes. But it can also run without Jenkins when used from the command line, as you mention. To be sure I would have to check if there is some loose end that affects ftc when removing Jenkins from the Dockerfile.
>
> Having Fuego run on docker makes sure that anyone can get the same environment quickly and it protects the host system from Fuego bugs. Having said that, I would like to prepare a script to install fuego on the host system in the future.

Please post on the ML or just let me know when you have it. I *want* it.

>
> > * as you pointed, parsing fuego's test result file in LAVA is easy to do.
>
> The only problem is that I would need to run the Fuego parser on the target board.
> For that, I would need to modularize the parser into a library (e.g. import fuego-parser), and the board would need to install the python modules required by fuego-parser. This is on my TODO list since I proposed it during the last Fuego jamboree. I will try to do it as soon as i can.
>
> What alternatives do I have?
> - send the results to LAVA through a REST API instead of having it monitor the serial cable? probably not possible.
> - create a simplified parser on the test (e.g. using our log_compare function). Not ideal, but possible.
>
> In the end, this stems from the fact that Fuego assumes parsing is done in the host (to use python), while Linaro uses grep/awk/sed directly on the target. There is a trade-off there.
>
> > * existing way to run fuego tests in LAVA are hacks. The problem is
> > they don't scale, 'scale' means remote and distributed CI setup.
>
> Yes, it is a hack.
> I think Fuego is not supposed to run with LAVA, because the goals are very different.
> But parts of Fuego can run with LAVA. This is what I think we can collaborate on.

Yes, +1. When running with LAVA, IMHO, only the backend and real tests
are needed.

>
> > * I am tring to hanld both fuego host controller and DUT with LAVA.
> > The first part is hard part. Still tring to find a way. About the host
> > controller part, I started with LAVA-lxc protocol, but hit some
> > jenkins and docker related issues. I feel build, publish and pull a
> > fuego docker image is the way to go now.
>
> I think this approach might be too hard.

LAVA v2 introduced lxc-protocol. With the protocol, single node test
job can deploy and boot a lxc container to control DUT. Here is an
example: https://lkft.validation.linaro.org/scheduler/job/605270 . The
example job use lxc contianer to deploy imgs to DUT. If DUT was
configed with static IP, the IP is known to lxc container with LAVA
helper lava-target-ip, then ssh connection between lxc and DUT is
possible. Based on these features, I thought we can run fuego tests
with LAVA just like how we run it now. As mentioned above, there is no
and will be no support for docker-protocol in LAVA, and migrating
fuego installation to lxc also is problemic. Please do let me know
once you have a script for fuego installation. I am having problem to
do that, hit jenkins missing, docker missing, permission issues, etc.
Once I am alble to install fuego within lxc, I can propare a job
example. It would be one test definition for all fuego tests. This is
how we do it before. `automated/linux/workload-automation3
` is a good example.

Alternatively, I can lunch docker device and DUT with multinode job,
but that is complex. And fuego docker container eats a lot of
memory(blame jenkins?). The exsting docker devices in our lib only
have 1G memory configured.

>
> This is my current work-in-progress approach:
> https://github.com/sangorrin/test-definitions/tree/master/automated/linux/fuego
>
> - Manual usage (run locally)
>         $ git clone https://github.com/sangorrin/test-definitions
>         $ cd test-definitions
>         $ . ./automated/bin/setenv.sh
>         $ cd automated/linux/fuego/
>         $ ./fuego.sh -d Functional.hello_world
>         $  tree output/
>                 output/
>                 ├── build <- equivalent to fuego buildzone
>                 │   ├── hello
>                 │   ├── hello.c
>                 │   ├── Makefile
>                 │   └── README.md
>                 ├── fuego.Functional.hello_world <- equivalent to board test folder
>                 │   └── hello
>                 └── logs <- equivalent to logdir
>                         └── testlog.txt
> - test-runner usage (run on remote board)
>         $ cd test-definitions
>         $ test-runner -g root at 192.168.1.45 -d ./automated/linux/fuego/fuego.yaml -s -o ../output
>         $ ls ../output
>                 result.csv
>                 result.json
>
> I have yet to add the LAVA messages and prepare result.txt but it will be working soon.

You don't have to. It looks like a done job to me. send-to-lava.sh
will take care of it. When running in LAVA, the helper uses
lava-test-case for result collecting, and when running without LAVA,
the helper prints result lines in a fixed format for result parsing
within test-runner. (When I writing this, I noticed your next reply,
maybe I am looking at the latest code already, I will give it a spin
with LAVA and come back to you)

So basically, we are running in two different directions. From my
point of view, you are porting fuego tests to Linaro test-definitions
natively. Although I am not yet sure how the integration between these
two projects goes, we are happy to see this happening :)

>
> By the way, I couldn't reuse some parts of Fuego that usually run on the host because they depend on bash.
> Currently Functional.hello_world is working on sh but I will find similar issues as I add more test definitions.
> Is sh a hard requirement for you guys? or would you be fine with tests requiring bash.
>

lava-test-shell requires POSIX shell. We normally use /bin/sh which
links to dash on Debian based distros, and we also have some test
definitions like ltp and android trandfed using bash. bash has some
extensions are not POSIX compatiable. IMHO, using bash without these
extensions is totally fine. We are using shellcheck in sanity check to
dedect potential POSIX issues.

Thanks,
Chase


> Thanks,
> Daniel
>
> > We probably should start a new thread for this topic to share progress?
> >
> > Thanks,
> > Chase
> >
> > [1] https://github.com/Linaro/test-definitions/blob/master/automated/lib/sh-test-lib#L250
> >
> >
> > > Thanks,
> > > Daniel
> > >
> > > > -----Original Message-----
> > > > From: Tim.Bird at sony.com <Tim.Bird at sony.com>
> > > > Sent: Thursday, February 14, 2019 6:51 AM
> > > > To: sangorrin daniel(サンゴリン ダニエル ○SWC□OST) <daniel.sangorrin at toshiba.co.jp>;
> > > > fuego at lists.linuxfoundation.org
> > > > Subject: RE: [Fuego] [PATCH] tests: add support for Linaro test-definitons
> > > >
> > > > Comments inline below.
> > > >
> > > > > -----Original Message-----
> > > > > From: Daniel Sangorrin
> > > > >
> > > > > This adds initial support for reusing Linaro test-definitions.
> > > > > It is still a proof of concept and only tested with
> > > > > smoke tests. I have written a few FIXTHIS to indicate what
> > > > > is left.
> > > > >
> > > > > To try it follow these steps:
> > > > >
> > > > > - prepare SSH_KEY for your board
> > > > >     Eg: Inside fuego's docker container do
> > > > >     > su jenkins
> > > > >     > cp path/to/bbb_id_rsa ~/.ssh/
> > > > >     > vi ~/.ssh/config
> > > > >     >  Host 192.167.1.99 <- replace with your boards ip address ($IPADDR)
> > > > >     >    IdentityFile ~/.ssh/bbb_id_rsa
> > > > > - ftc add-job -b bbb -t Functional.linaro
> > > > > - execute the job from jenkins
> > > > > - expected results
> > > > >     - table with each test case and the results (PASS/FAIL/SKIP)
> > > > >     - run.json
> > > > >     - csv
> > > > >
> > > > > Signed-off-by: Daniel Sangorrin <daniel.sangorrin at toshiba.co.jp>
> > > > > ---
> > > > >  tests/Functional.linaro/chart_config.json |  3 ++
> > > > >  tests/Functional.linaro/fuego_test.sh     | 59
> > > > > +++++++++++++++++++++++++++++++
> > > > >  tests/Functional.linaro/parser.py         | 25 +++++++++++++
> > > > >  tests/Functional.linaro/spec.json         | 16 +++++++++
> > > > >  tests/Functional.linaro/test.yaml         | 27 ++++++++++++++
> > > > >  5 files changed, 130 insertions(+)
> > > > >  create mode 100644 tests/Functional.linaro/chart_config.json
> > > > >  create mode 100755 tests/Functional.linaro/fuego_test.sh
> > > > >  create mode 100755 tests/Functional.linaro/parser.py
> > > > >  create mode 100644 tests/Functional.linaro/spec.json
> > > > >  create mode 100644 tests/Functional.linaro/test.yaml
> > > > >
> > > > > diff --git a/tests/Functional.linaro/chart_config.json
> > > > > b/tests/Functional.linaro/chart_config.json
> > > > > new file mode 100644
> > > > > index 0000000..b8c8fb6
> > > > > --- /dev/null
> > > > > +++ b/tests/Functional.linaro/chart_config.json
> > > > > @@ -0,0 +1,3 @@
> > > > > +{
> > > > > +    "chart_type": "testcase_table"
> > > > > +}
> > > > > diff --git a/tests/Functional.linaro/fuego_test.sh
> > > > > b/tests/Functional.linaro/fuego_test.sh
> > > > > new file mode 100755
> > > > > index 0000000..17b56a9
> > > > > --- /dev/null
> > > > > +++ b/tests/Functional.linaro/fuego_test.sh
> > > > > @@ -0,0 +1,59 @@
> > > > > +gitrepo="https://github.com/Linaro/test-definitions.git"
> > > > > +
> > > > > +# Root permissions required for
> > > > > +# - installing dependencies on the target (debian/centos) when -s is not
> > > > > specified
> > > > > +# - executing some of the tests
> > > > > +# FIXTHIS: don't force root permissions for tests that do not require them
> > > > > +NEED_ROOT=1
> > > > > +
> > > > > +function test_pre_check {
> > > > > +    # linaro parser dependencies
> > > > > +    # FIXTHIS: use dependencies specified in the test definition yaml
> > > > > +    assert_has_program sed
> > > > > +    assert_has_program awk
> > > > > +    assert_has_program grep
> > > > > +    assert_has_program egrep
> > > > > +    assert_has_program tee
> > > > I missed this earlier, but Fuego requires 'grep' and 'tee', so there's
> > > > no need to check for them here.
> > > > I already made a patch to remove those lines.
> > > >
> > > > > +
> > > > > +    # test-runner requires a password-less connection
> > > > > +    # Eg: Inside fuego's docker container do
> > > > > +    # su jenkins
> > > > > +    # cp path/to/bbb_id_rsa ~/.ssh/
> > > > > +    # vi ~/.ssh/config
> > > > > +    #  Host 192.167.1.99 <- replace with your boards ip address ($IPADDR)
> > > > > +    #    IdentityFile ~/.ssh/bbb_id_rsa
> > > > > +    assert_define SSH_KEY "Please setup SSH_KEY on your board file (fuego-
> > > > > ro/boards/$NODE_NAME.board)"
> > > > > +}
> > > > > +
> > > > > +function test_build {
> > > > > +    source ./automated/bin/setenv.sh
> > > > > +    pip install -r $REPO_PATH/automated/utils/requirements.txt --user
> > > >
> > > > OK.  I gave this a spin, and here's an error I got:
> > > >
> > > > ===== doing fuego phase: build =====
> > > > Clone repository https://github.com/Linaro/test-definitions.git.
> > > > Cloning into 'fuego_git_repo'...
> > > > Checkout branch/tag/commit id master.
> > > > Already on 'master'
> > > > Your branch is up-to-date with 'origin/master'.
> > > > REPO_PATH: /fuego-rw/buildzone/bbb.default.Functional.linaro-debian-armhf
> > > > BIN_PATH:
> > > >
> > /fuego-rw/buildzone/bbb.default.Functional.linaro-debian-armhf/automated/bin:/usr/local/bin:/usr/local/bin:
> > > > /usr/bin:/bin:/usr/local/games:/usr/games
> > > > Downloading/unpacking pexpect (from -r
> > > > /fuego-rw/buildzone/bbb.default.Functional.linaro-debian-armhf/automated/utils/requirements.txt (line 1))
> > > > Requirement already satisfied (use --upgrade to upgrade): pyyaml in /usr/lib/python2.7/dist-packages (from -r
> > > > /fuego-rw/buildzone/bbb.default.Functional.linaro-debian-armhf/automated/utils/requirements.txt (line 2))
> > > > Requirement already satisfied (use --upgrade to upgrade): requests in /usr/lib/python2.7/dist-packages (from
> > -r
> > > > /fuego-rw/buildzone/bbb.default.Functional.linaro-debian-armhf/automated/utils/requirements.txt (line 3))
> > > > Downloading/unpacking ptyprocess>=0.5 (from pexpect->-r
> > > > /fuego-rw/buildzone/bbb.default.Functional.linaro-debian-armhf/automated/utils/requirements.txt (line 1))
> > > >   Downloading ptyprocess-0.6.0-py2.py3-none-any.whl
> > > > Installing collected packages: pexpect, ptyprocess
> > > > Compiling /tmp/pip-build-65KFAp/pexpect/pexpect/_async.py ...
> > > >   File "/tmp/pip-build-65KFAp/pexpect/pexpect/_async.py", line 19
> > > >     transport, pw = yield from asyncio.get_event_loop()\
> > > >                              ^
> > > > SyntaxError: invalid syntax
> > > >
> > > > Successfully installed pexpect ptyprocess
> > > > Cleaning up...
> > > > Fuego test_build duration=1.56257462502 seconds
> > > >
> > > > Weirdly, I only see this on the first build.  I think the pip install alters the docker
> > > > container, so that even if I specify 'rebuild' for this job, I don't see the "compilation"
> > > > error after the first build of the job.
> > > >
> > > > > +}
> > > > > +
> > > > > +function test_run {
> > > > > +    source $WORKSPACE/$JOB_BUILD_DIR/automated/bin/setenv.sh
> > > > > +
> > > > > +    yaml_file=${FUNCTIONAL_LINARO_YAML:-
> > > > > "automated/linux/smoke/smoke.yaml"}
> > > > > +    if [ ! -e "${REPO_PATH}/$yaml_file" ]; then
> > > > > +            abort_job "$yaml_file not found"
> > > > > +    fi
> > > > > +
> > > > > +    if startswith "$yaml_file" "plans"; then
> > > > > +            echo "using test plan: $yaml_file"
> > > > > +            test_or_plan_flag="-p"
> > > > > +    else
> > > > > +            echo "using test definition: $yaml_file"
> > > > > +            test_or_plan_flag="-d"
> > > > > +    fi
> > > > > +
> > > > > +    if [ -n "$FUNCTIONAL_LINARO_PARAMS" ]; then
> > > > > +        PARAMS="-r $FUNCTIONAL_LINARO_PARAMS"
> > > > > +    else
> > > > > +        PARAMS=""
> > > > > +    fi
> > > > > +
> > > > > +    # FIXTHIS: don't use -s for targets with debian/centos
> > > > > +    test-runner -o ${LOGDIR} $test_or_plan_flag ${REPO_PATH}/$yaml_file
> > > > > $PARAMS -g $LOGIN@$IPADDR -s -e
> > > > > +}
> > > > > +
> > > > > +# FIXTHIS: the log directory is populated with a copy of the whole
> > > > > repository, clean unnecessary files
> > > > > diff --git a/tests/Functional.linaro/parser.py
> > > > > b/tests/Functional.linaro/parser.py
> > > > > new file mode 100755
> > > > > index 0000000..48b502b
> > > > > --- /dev/null
> > > > > +++ b/tests/Functional.linaro/parser.py
> > > > > @@ -0,0 +1,25 @@
> > > > > +#!/usr/bin/python
> > > > > +
> > > > > +import os, sys, collections
> > > > > +import common as plib
> > > > > +import json
> > > > > +
> > > > > +# allocate variable to store the results
> > > > > +measurements = {}
> > > > > +measurements = collections.OrderedDict()
> > > > > +
> > > > > +# read results from linaro result.json format
> > > > > +with open(plib.LOGDIR + "/result.json") as f:
> > > > > +    data = json.load(f)[0]
> > > > > +
> > > > > +for test_case in data['metrics']:
> > > > > +    test_case_id = test_case['test_case_id']
> > > > > +    result = test_case['result']
> > > > > +    # FIXTHIS: add measurements when available
> > > > > +    # measurement = test_case['measurement']
> > > > > +    # units = test_case['units']
> > > > > +    measurements['default.' + test_case_id] = result.upper()
> > > > > +
> > > > > +# FIXTHIS: think about how to get each test's log from stdout.log
> > > > > +
> > > > > +sys.exit(plib.process(measurements))
> > > > > diff --git a/tests/Functional.linaro/spec.json
> > > > > b/tests/Functional.linaro/spec.json
> > > > > new file mode 100644
> > > > > index 0000000..561e2ab
> > > > > --- /dev/null
> > > > > +++ b/tests/Functional.linaro/spec.json
> > > > > @@ -0,0 +1,16 @@
> > > > > +{
> > > > > +    "testName": "Functional.linaro",
> > > > > +    "specs": {
> > > > > +        "default": {
> > > > > +            "yaml": "automated/linux/smoke/smoke.yaml",
> > > > > +            "extra_success_links": {"csv": "result.csv"},
> > > > > +            "extra_fail_links": {"csv": "results.csv"}
> > > > > +        },
> > > > > +        "smoke": {
> > > > > +            "yaml": "automated/linux/smoke/smoke.yaml",
> > > > > +            "params": "TESTS='pwd'",
> > > > > +            "extra_success_links": {"csv": "result.csv"},
> > > > > +            "extra_fail_links": {"csv": "results.csv"}
> > > > > +        }
> > > > > +    }
> > > > > +}
> > > > > diff --git a/tests/Functional.linaro/test.yaml
> > > > > b/tests/Functional.linaro/test.yaml
> > > > > new file mode 100644
> > > > > index 0000000..a2efee8
> > > > > --- /dev/null
> > > > > +++ b/tests/Functional.linaro/test.yaml
> > > > > @@ -0,0 +1,27 @@
> > > > > +fuego_package_version: 1
> > > > > +name: Functional.linaro
> > > > > +description: |
> > > > > +    Linaro test-definitions
> > > > > +license: GPL-2.0
> > > > > +author: Milosz Wasilewski, Chase Qi
> > > > > +maintainer: Daniel Sangorrin <daniel.sangorrin at toshiba.co.jp>
> > > > > +version: latest git commits
> > > > > +fuego_release: 1
> > > > > +type: Functional
> > > > > +tags: ['kernel', 'linaro']
> > > > > +git_src: https://github.com/Linaro/test-definitions
> > > > > +params:
> > > > > +    - YAML:
> > > > > +        description: test definiton or plan.
> > > > > +        example: "automated/linux/smoke/smoke.yaml"
> > > > > +        optional: no
> > > > > +    - PARAMS:
> > > > > +        description: List of params for the test PARAM1=VALUE1
> > > > > [PARAM2=VALUE2]
> > > > > +        example: "TESTS='pwd'"
> > > > > +        optional: yes
> > > > > +data_files:
> > > > > +    - chart_config.json
> > > > > +    - fuego_test.sh
> > > > > +    - parser.py
> > > > > +    - spec.json
> > > > > +    - test.yaml
> > > > > --
> > > > > 2.7.4
> > > >
> > > > And here's output from one of my initial runs.  I haven't debugged it yet.
> > > > The issue may be something weird in my board file or configuration.
> > > >
> > > > ===== doing fuego phase: run =====
> > > > -------------------------------------------------
> > > > REPO_PATH: /fuego-rw/buildzone/bbb.default.Functional.linaro-debian-armhf
> > > > BIN_PATH:
> > > >
> > /fuego-rw/buildzone/bbb.default.Functional.linaro-debian-armhf/automated/bin:/usr/local/bin:/usr/local/bin:
> > > > /usr/bin:/bin:/usr/local/games:/usr/games
> > > > using test definition: automated/linux/smoke/smoke.yaml
> > > > Warning: Permanently added '10.0.1.74' (ECDSA) to the list of known hosts.
> > > > 2019-02-13 21:46:14,364 - RUNNER: INFO: Tests to run:
> > > > Warning: Permanently added '10.0.1.74' (ECDSA) to the list of known hosts.
> > > > 2019-02-13 21:46:14,814 - RUNNER.TestSetup: INFO: Test repo copied to:
> > > > /fuego-rw/logs/Functional.linaro/bbb.default.6.6/smoke_2efe23f3-2a7b-4655-b785-d25c11b84ea8
> > > > 2019-02-13 21:46:14,826 - RUNNER.TestRun: INFO: Archiving test files
> > > > 2019-02-13 21:46:14,845 - RUNNER.TestRun: INFO: Creating test path
> > > > Warning: Permanently added '10.0.1.74' (ECDSA) to the list of known hosts.
> > > > 2019-02-13 21:46:15,133 - RUNNER.TestRun: INFO: Copying test archive to target host
> > > > Warning: Permanently added '10.0.1.74' (ECDSA) to the list of known hosts.
> > > > 2019-02-13 21:46:16,260 - RUNNER.TestRun: INFO: Unarchiving test files on target
> > > > Warning: Permanently added '10.0.1.74' (ECDSA) to the list of known hosts.
> > > > 2019-02-13 21:46:16,674 - RUNNER.TestRun: INFO: Removing test file archive from target
> > > > Warning: Permanently added '10.0.1.74' (ECDSA) to the list of known hosts.
> > > > 2019-02-13 21:46:16,978 - RUNNER.TestRun: INFO: Executing
> > > > /root/output/smoke_2efe23f3-2a7b-4655-b785-d25c11b84ea8/run.sh remotely on root at 10.0.1.74
> > > > {'path':
> > > > '/fuego-rw/buildzone/bbb.default.Functional.linaro-debian-armhf/automated/linux/smoke/smoke.yaml',
> > > > 'uuid': '2efe23f3-2a7b-4655-b785-d25c11b84ea8', 'timeout': None, 'skip_install': True}
> > > > Warning: Permanently added '10.0.1.74' (ECDSA) to the list of known hosts.
> > > > + export TESTRUN_ID=smoke-tests-basic
> > > > + cd /root/output/smoke_2efe23f3-2a7b-4655-b785-d25c11b84ea8
> > > > + cat uuid
> > > > + UUID=2efe23f3-2a7b-4655-b785-d25c11b84ea8
> > > > + echo <STARTRUN smoke-tests-basic 2efe23f3-2a7b-4655-b785-d25c11b84ea8>
> > > > <STARTRUN smoke-tests-basic 2efe23f3-2a7b-4655-b785-d25c11b84ea8>
> > > > + export PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
> > > > + cd ./automated/linux/smoke/
> > > > + ./smoke.sh -s True -t pwd, lsb_release -a, uname -a, ip a, lscpu, vmstat, lsblk
> > > > INFO: install_deps skipped
> > > >
> > > > INFO: Running pwd test...
> > > > /root/output/smoke_2efe23f3-2a7b-4655-b785-d25c11b84ea8/automated/linux/smoke
> > > > pwd pass
> > > >
> > > > INFO: Running lsb_release test...
> > > > ./smoke.sh: 1: eval: lsb_release: not found
> > > > lsb_release fail
> > > >
> > > > INFO: Running uname test...
> > > > Linux beaglebone 4.4.88-ti-r125 #1 SMP Thu Sep 21 19:23:24 UTC 2017 armv7l GNU/Linux
> > > > uname pass
> > > >
> > > > INFO: Running ip test...
> > > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
> > > >     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> > > >     inet 127.0.0.1/8 scope host lo
> > > >        valid_lft forever preferred_lft forever
> > > >     inet6 ::1/128 scope host
> > > >        valid_lft forever preferred_lft forever
> > > > 2: eth0: <BROADCAST,MULTICAST,DYNAMIC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default
> > > > qlen 1000
> > > >     link/ether 90:59:af:54:cd:e6 brd ff:ff:ff:ff:ff:ff
> > > >     inet 10.0.1.74/24 brd 10.0.1.255 scope global eth0
> > > >        valid_lft forever preferred_lft forever
> > > >     inet6 fe80::9259:afff:fe54:cde6/64 scope link
> > > >        valid_lft forever preferred_lft forever
> > > > 3: can0: <NOARP,ECHO> mtu 16 qdisc noop state DOWN group default qlen 10
> > > >     link/can
> > > > 4: can1: <NOARP,ECHO> mtu 16 qdisc noop state DOWN group default qlen 10
> > > >     link/can
> > > > 5: usb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
> > > >     link/ether 90:59:af:54:cd:e8 brd ff:ff:ff:ff:ff:ff
> > > >     inet 192.168.7.2/30 brd 192.168.7.3 scope global usb0
> > > >        valid_lft forever preferred_lft forever
> > > >     inet6 fe80::9259:afff:fe54:cde8/64 scope link
> > > >        valid_lft forever preferred_lft forever
> > > > 6: usb1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
> > > >     link/ether 90:59:af:54:cd:eb brd ff:ff:ff:ff:ff:ff
> > > >     inet 192.168.6.2/30 brd 192.168.6.3 scope global usb1
> > > >        valid_lft forever preferred_lft forever
> > > >     inet6 fe80::9259:afff:fe54:cdeb/64 scope link
> > > >        valid_lft forever preferred_lft forever
> > > > ip pass
> > > >
> > > > INFO: Running lscpu test...
> > > > Architecture:          armv7l
> > > > Byte Order:            Little Endian
> > > > CPU(s):                1
> > > > On-line CPU(s) list:   0
> > > > Thread(s) per core:    1
> > > > Core(s) per socket:    1
> > > > Socket(s):             1
> > > > Model:                 2
> > > > Model name:            ARMv7 Processor rev 2 (v7l)
> > > > CPU max MHz:           1000.0000
> > > > CPU min MHz:           300.0000
> > > > BogoMIPS:              995.32
> > > > Flags:                 half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32
> > > > lscpu pass
> > > >
> > > > INFO: Running vmstat test...
> > > > procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
> > > >  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
> > > >  0  0      0 381532   2780  58072    0    0     1     3   38    2  0  0 99  0  0
> > > > vmstat pass
> > > >
> > > > INFO: Running lsblk test...
> > > > NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
> > > > mmcblk0      179:0    0 14.9G  0 disk
> > > > |-mmcblk0p1  179:1    0    6G  0 part /
> > > > `-mmcblk0p2  179:2    0  8.9G  0 part /data
> > > > mmcblk1      179:8    0  1.8G  0 disk
> > > > `-mmcblk1p1  179:9    0  1.8G  0 part /mnt/on-board-mmc
> > > > mmcblk1boot0 179:16   0    1M  1 disk
> > > > mmcblk1boot1 179:24   0    1M  1 disk
> > > > lsblk pass
> > > > + ../../utils/send-to-lava.sh ./output/result.txt
> > > > <TEST_CASE_ID=pwd RESULT=pass>
> > > > <TEST_CASE_ID=lsb_release RESULT=fail>
> > > > <TEST_CASE_ID=uname RESULT=pass>
> > > > <TEST_CASE_ID=ip RESULT=pass>
> > > > <TEST_CASE_ID=lscpu RESULT=pass>
> > > > <TEST_CASE_ID2019-02-13 21:46:18,376 - RUNNER.TestRun: INFO:
> > > > smoke_2efe23f3-2a7b-4655-b785-d25c11b84ea8 test finished.
> > > >
> > > > 2019-02-13 21:46:18,397 - RUNNER.ResultParser: INFO: Result files saved to:
> > > > /fuego-rw/logs/Functional.linaro/bbb.default.6.6/smoke_2efe23f3-2a7b-4655-b785-d25c11b84ea8
> > > > =vmstat RESULT=pass>
> > > > <TEST_CASE_ID=lsblk RESULT=pass>
> > > > + echo <ENDRUN smoke-tests-basic 2efe23f3-2a7b-4655-b785-d25c11b84ea8>
> > > > <ENDRUN smoke-tests-basic 2efe23f3-2a7b-4655-b785-d25c11b84ea8>
> > > > --- Printing result.csv ---
> > > > name,test_case_id,result,measurement,units,test_params
> > > > smoke-tests-basic,pwd,pass,,,"TESTS=pwd, lsb_release -a, uname -a, ip a, lscpu, vmstat,
> > > > lsblk;SKIP_INSTALL=False"
> > > > smoke-tests-basic,lsb_release,fail,,,"TESTS=pwd, lsb_release -a, uname -a, ip a, lscpu, vmstat,
> > > > lsblk;SKIP_INSTALL=False"
> > > > smoke-tests-basic,uname,pass,,,"TESTS=pwd, lsb_release -a, uname -a, ip a, lscpu, vmstat,
> > > > lsblk;SKIP_INSTALL=False"
> > > > smoke-tests-basic,ip,pass,,,"TESTS=pwd, lsb_release -a, uname -a, ip a, lscpu, vmstat,
> > > > lsblk;SKIP_INSTALL=False"
> > > > smoke-tests-basic,lscpu,pass,,,"TESTS=pwd, lsb_release -a, uname -a, ip a, lscpu, vmstat,
> > > > lsblk;SKIP_INSTALL=False"
> > > > smoke-tests-basic,vmstat,pass,,,"TESTS=pwd, lsb_release -a, uname -a, ip a, lscpu, vmstat,
> > > > lsblk;SKIP_INSTALL=False"
> > > > smoke-tests-basic,lsblk,pass,,,"TESTS=pwd, lsb_release -a, uname -a, ip a, lscpu, vmstat,
> > > > lsblk;SKIP_INSTALL=False"
> > > >
> > > > -------------------------------------------------
> > > > ===== doing fuego phase: post_test =====
> > > > scp: /home/fuego/fuego.Functional.linaro/Functional.linaro.log: No such file or directory
> > > > INFO: the test did not produce a test log on the target
> > > > ===== doing fuego phase: processing =====
> > > > ### WARNING: Program returned exit code ''
> > > > ### WARNING: Log evaluation may be invalid
> > > > ### Unrecognized results format
> > > > ### Unrecognized results format
> > > > ### Unrecognized results format
> > > > ### Unrecognized results format
> > > > ### Unrecognized results format
> > > > ### Unrecognized results format
> > > > ### Unrecognized results format
> > > > ERROR: results did not satisfy the threshold
> > > > Fuego: requested test phases complete!
> > > > Build step 'Execute shell' marked build as failure
> > > >
> > > > ----------
> > > >
> > > > It looks like I'm close.  I'll keep playing with it, but if you see something
> > > > that I should fix, please let me know.
> > > >
> > > > Note that I *do* get a results table in Jenkins.  lsb_release fails, but the
> > > > other tests (ip, lsblk, lscpu, pwd, uname, and vmstat) all pass.   But
> > > > testlog.txt has 'INFO: the test did not produce a test log on the target'.
> > > >
> > > > Just FYI.  Thanks for the code.
> > > >  -- Tim
> > >
> > > _______________________________________________
> > > Fuego mailing list
> > > Fuego at lists.linuxfoundation.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/fuego


More information about the Fuego mailing list