[Fuego] [PATCH] LTP_one_test: create a Fuego test for a single LTP test
Daniel Sangorrin
daniel.sangorrin at toshiba.co.jp
Tue Apr 10 02:37:42 UTC 2018
Hi Tim,
> -----Original Message-----
> From: Tim.Bird at sony.com [mailto:Tim.Bird at sony.com]
> Sent: Saturday, April 7, 2018 4:11 AM
> To: liuwl.fnst at cn.fujitsu.com; daniel.sangorrin at toshiba.co.jp;
> fuego at lists.linuxfoundation.org
> Subject: [PATCH] LTP_one_test: create a Fuego test for a single LTP test
>
> Hey Daniel and Liu,
>
> Here is a test that runs an individual LTP test. It's now in my master
> branch. The reason for pointing this out is to get your feedback
> on the approach of writing a test that uses a sub-set of LTP.
Let me add some background for everyone to understand.
- On Functional.LTP, it is possible to run sub-sets of LTP by using a skip list. In other words,
you first select the test sets (groups of test cases) that you want to run, and then you
apply a skip list to mask the execution of some of the test cases. Let's call this a blacklisting
approach. The main advantage is that you only need to know the name of the test cases
that you want to skip.
- On Functional.LTP_one_test, it is possible to specify what test case to execute directly. For that
reason there is no need to create any skip list. The main advantage is that it allows the user
to quickly run any of the tests inside LTP.
Let me jump into one technical detail though. I see that you are calling the test cases directly
from /testcases/bin/. I don't think that works in general because some of the test cases
require a bit of setup before they are executed. Fortunately, this is already taken care by
runltp through the "-s" option.
$ ./runltp -q -f math -s abs01
[Note] remove -q to see the test output as well
> Note the use of 'ftc run-test' with the '-p b' parameter, to build
> LTP if needed. This avoids having to share build script details
> between this test and LTP.
Maybe it's because the recent update broke everything, but "ftc run-test" and "ftc build-jobs" didn't work on my machine.
Other than that, I like the ability to use "ftc run-test" within the tests. In this case for the building phase, but I want to use
it for creating background loads as well.
Do you think there could be some conflict with the nesting of tests such as variable
overwriting (Fuego using all these global shell variables is a bit scary) or deadlocks?
> I'm thinking this approach might be useful to split out
> the OpenPosix and realtime subsets of LTP.
Sounds good.
> Also, it might
> work for handling setup and cleanup for subtest groups or
> individual tests, like the network-related tests or smack.
> For example, instead of doing smack-specific machine setup
> in the LTP test, I think we should make something like
> Functional.LTP-smack, use the techniques demonstrated here
> for building it. But then have pre_test, deply, run, and processing
> phases be customized for that test's needs. in Functional.LTP-smack's
> fuego_test.sh.
Yeah, let's split corner cases. Otherwise things will get too complicated to handle.
Thanks,
Daniel
>
> Let me know what you think.
> -- Tim
>
> Patch follows:
> ----------------------------
> LTP is a ginormous collection of tests. When something fails,
> it is very handy to be able to run a single test in isolation.
> This test allows you to do that. To run an individual test,
> just add a new spec to the spec file (or, when ftc supports
> setting a test variable on the command line, use that to
> specify the name of the individual test program to run).
>
> This test also demonstrates how one test can use ftc and phases
> to execute just the build of another test. And how one test
> can re-use the build directory of another test.
>
> Signed-off-by: Tim Bird <tim.bird at sony.com>
> ---
> engine/tests/Functional.LTP_one_test/fuego_test.sh | 40
> +++++++++++++++++
> engine/tests/Functional.LTP_one_test/parser.py | 51
> ++++++++++++++++++++++
> engine/tests/Functional.LTP_one_test/spec.json | 12 +++++
> engine/tests/Functional.LTP_one_test/test.yaml | 31 +++++++++++++
> 4 files changed, 134 insertions(+)
> create mode 100755 engine/tests/Functional.LTP_one_test/fuego_test.sh
> create mode 100755 engine/tests/Functional.LTP_one_test/parser.py
> create mode 100644 engine/tests/Functional.LTP_one_test/spec.json
> create mode 100644 engine/tests/Functional.LTP_one_test/test.yaml
>
> diff --git a/engine/tests/Functional.LTP_one_test/fuego_test.sh
> b/engine/tests/Functional.LTP_one_test/fuego_test.sh
> new file mode 100755
> index 0000000..2ebcfc5
> --- /dev/null
> +++ b/engine/tests/Functional.LTP_one_test/fuego_test.sh
> @@ -0,0 +1,40 @@
> +one_test=$FUNCTIONAL_LTP_ONE_TEST_TEST
> +
> +# Don't allow jobs to share build directories
> +# the "test_successfully_built" flag for one spec
> +# is not accurate for another spec, which has a different
> +# individual program to deploy.
> +FUNCTIONAL_LTP_ONE_TEST_PER_JOB_BUILD="true"
> +
> +function test_pre_check {
> + assert_define FUNCTIONAL_LTP_ONE_TEST_TEST
> +}
> +
> +function test_build {
> + # check for LTP build directory
> + LTP_BUILD_DIR=$(echo $JOB_BUILD_DIR | sed s/LTP_one_test/LTP/ | sed
> s/$TESTSPEC/default/)
> +
> + # if not already built, build LTP
> + if [ ! -e ${WORKSPACE}/${LTP_BUILD_DIR}/fuego_test_succesfully_built ] ;
> then
> + echo "Building parent LTP test..."
> + ftc run-test -b $NODE_NAME -t Functional.LTP -p b
> + # NOTE: vars used in ftc run-test should not leak into this environment
> + # that is, none of our test vars should have changed.
> + fi
> +
> + cp ${WORKSPACE}/${LTP_BUILD_DIR}/target_bin/testcases/bin/$one_test .
> +}
> +
> +function test_deploy {
> + put $one_test $BOARD_TESTDIR/fuego.$TESTDIR/
> +}
> +
> +function test_run {
> + # FIXTHIS - deal with arguments better
> + report "cd $BOARD_TESTDIR/fuego.$TESTDIR; \
> + ./$one_test $FUNCTIONAL_LTP_ONE_TEST_ARGS"
> +}
> +
> +function test_processing {
> + return
> +}
> diff --git a/engine/tests/Functional.LTP_one_test/parser.py
> b/engine/tests/Functional.LTP_one_test/parser.py
> new file mode 100755
> index 0000000..fe2336f
> --- /dev/null
> +++ b/engine/tests/Functional.LTP_one_test/parser.py
> @@ -0,0 +1,51 @@
> +#!/usr/bin/python
> +# See common.py for description of command-line arguments
> +
> +import os
> +import sys
> +import collections
> +import re
> +
> +sys.path.insert(0, os.environ['FUEGO_CORE'] + '/engine/scripts/parser')
> +import common as plib
> +
> +results = collections.OrderedDict()
> +
> +# LTP results come in largely 2 formats:
> +# sched_getattr01 1 TPASS : attributes were read back correctly
> +# request_key01.c:49: PASS: request_key() succeed
> +
> +log_lines = open(plib.TEST_LOG, "r").readlines()
> +re_pattern = r"(\S+)\s+(\S)+\s+(TPASS|TCONF)\s+:(.*)"
> +
> +testcase_count = 1
> +for line in log_lines:
> + #print "line=", line
> + if ".c:" in line:
> + parts = line.split(":")
> + filename = parts[0]
> + line_no = parts[1]
> + testcase_num = str(testcase_count)
> + testcase_count += 1
> + result = parts[2].strip()
> + message = parts[3].strip()
> + else:
> + m = re.match(re_pattern, line)
> + #print "m=", m
> + if m:
> + test_name = m.group(1)
> + testcase_num = m.group(2)
> + result = m.group(3)[1:] # strip off the leading 'T'
> + message = m.group(4).strip()
> +
> + # translate to Fuego results
> + if result == 'PASS':
> + results[testcase_num] = 'PASS'
> + elif result == 'FAIL':
> + results[testcase_num] = 'FAIL'
> + elif result == 'CONF':
> + results[testcase_num] = 'SKIP'
> + else:
> + results[testcase_num] = 'ERR'
> +
> +sys.exit(plib.process(results))
> diff --git a/engine/tests/Functional.LTP_one_test/spec.json
> b/engine/tests/Functional.LTP_one_test/spec.json
> new file mode 100644
> index 0000000..9f0b325
> --- /dev/null
> +++ b/engine/tests/Functional.LTP_one_test/spec.json
> @@ -0,0 +1,12 @@
> +{
> + "testName": "Functional.LTP_one_test",
> + "specs": {
> + "default": {
> + "TEST":"brk01"
> + },
> + "fanotify07": {
> + "TEST":"fanotify07"
> + }
> + }
> +}
> +
> diff --git a/engine/tests/Functional.LTP_one_test/test.yaml
> b/engine/tests/Functional.LTP_one_test/test.yaml
> new file mode 100644
> index 0000000..765c98d
> --- /dev/null
> +++ b/engine/tests/Functional.LTP_one_test/test.yaml
> @@ -0,0 +1,31 @@
> +fuego_package_version: 1
> +name: Functional.LTP_one_test
> +description: |
> + This is a convenience test. LTP is a large collection of tests, which
> + takes a lot of infrastructure to set up and execute. Sometimes it is
> + handy to run a single LTP test program by itself. This Fuego test
> + exists to allow you to easily do that.
> +
> + To run an individual test, add a new spec for it in the spec.json file.
> + Specify the individual test name to run in the TEST variable. By
> + convention the name of the spec should match the name of the test
> + to run.
> +license: multiple (same as LTP)
> +author: Tim Bird
> +maintainer: Tim Bird <tim.bird at sony.com>
> +version: 1.0
> +fuego_release: 1
> +type: Functional
> +tags: ['general']
> +params:
> + - TEST:
> + description: Name of the individual LTP test program to run.
> + example: fanotify07
> + optional: no
> + - ARGS:
> + description: Command line arguments for the LTP test program.
> + example: -s foo
> + optional: yes
> +data_files:
> + - spec.json
> + - parser.py
> --
> 1.9.1
>
More information about the Fuego
mailing list