[Fuego] [PATCH] LTP_one_test: create a Fuego test for a single LTP test
Liu, Wenlong
liuwl.fnst at cn.fujitsu.com
Mon Apr 16 05:36:50 UTC 2018
> -----Original Message-----
> From: Daniel Sangorrin [mailto:daniel.sangorrin at toshiba.co.jp]
> Sent: Tuesday, April 10, 2018 10:38 AM
> To: Tim.Bird at sony.com; Liu, Wenlong/刘 文龙 <liuwl.fnst at cn.fujitsu.com>;
> fuego at lists.linuxfoundation.org
> Subject: RE: [PATCH] LTP_one_test: create a Fuego test for a single LTP
> test
>
> Hi Tim,
>
> > -----Original Message-----
> > From: Tim.Bird at sony.com [mailto:Tim.Bird at sony.com]
> > Sent: Saturday, April 7, 2018 4:11 AM
> > To: liuwl.fnst at cn.fujitsu.com; daniel.sangorrin at toshiba.co.jp;
> > fuego at lists.linuxfoundation.org
> > Subject: [PATCH] LTP_one_test: create a Fuego test for a single LTP
> > test
> >
> > Hey Daniel and Liu,
> >
> > Here is a test that runs an individual LTP test. It's now in my
> > master branch. The reason for pointing this out is to get your
> > feedback on the approach of writing a test that uses a sub-set of LTP.
>
> Let me add some background for everyone to understand.
>
> - On Functional.LTP, it is possible to run sub-sets of LTP by using a skip
> list. In other words, you first select the test sets (groups of test cases)
> that you want to run, and then you apply a skip list to mask the execution
> of some of the test cases. Let's call this a blacklisting approach. The
> main advantage is that you only need to know the name of the test cases
> that you want to skip.
> - On Functional.LTP_one_test, it is possible to specify what test case to
> execute directly. For that reason there is no need to create any skip list.
> The main advantage is that it allows the user to quickly run any of the
> tests inside LTP.
>
> Let me jump into one technical detail though. I see that you are calling
> the test cases directly from /testcases/bin/. I don't think that works in
> general because some of the test cases require a bit of setup before they
> are executed. Fortunately, this is already taken care by runltp through
> the "-s" option.
> $ ./runltp -q -f math -s abs01
> [Note] remove -q to see the test output as well
Yes, Daniel, I have the same thought with you.
> > Note the use of 'ftc run-test' with the '-p b' parameter, to build
> > LTP if needed. This avoids having to share build script details
> > between this test and LTP.
>
> Maybe it's because the recent update broke everything, but "ftc run-test"
> and "ftc build-jobs" didn't work on my machine.
> Other than that, I like the ability to use "ftc run-test" within the tests.
> In this case for the building phase, but I want to use it for creating
> background loads as well.
>
> Do you think there could be some conflict with the nesting of tests such
> as variable overwriting (Fuego using all these global shell variables is
> a bit scary) or deadlocks?
>
> > I'm thinking this approach might be useful to split out the OpenPosix
> > and realtime subsets of LTP.
> Sounds good.
>
> > Also, it might
> > work for handling setup and cleanup for subtest groups or
> > individual tests, like the network-related tests or smack.
> > For example, instead of doing smack-specific machine setup
> > in the LTP test, I think we should make something like
> > Functional.LTP-smack, use the techniques demonstrated here
> > for building it. But then have pre_test, deply, run, and processing
> > phases be customized for that test's needs. in Functional.LTP-smack's
> > fuego_test.sh.
>
> Yeah, let's split corner cases. Otherwise things will get too complicated
> to handle.
>
> Thanks,
> Daniel
>
> >
> > Let me know what you think.
> > -- Tim
> >
> > Patch follows:
> > ----------------------------
> > LTP is a ginormous collection of tests. When something fails,
> > it is very handy to be able to run a single test in isolation.
> > This test allows you to do that. To run an individual test,
> > just add a new spec to the spec file (or, when ftc supports
> > setting a test variable on the command line, use that to
> > specify the name of the individual test program to run).
> >
> > This test also demonstrates how one test can use ftc and phases
> > to execute just the build of another test. And how one test
> > can re-use the build directory of another test.
> >
> > Signed-off-by: Tim Bird <tim.bird at sony.com>
> > ---
> > engine/tests/Functional.LTP_one_test/fuego_test.sh | 40
> > +++++++++++++++++
> > engine/tests/Functional.LTP_one_test/parser.py | 51
> > ++++++++++++++++++++++
> > engine/tests/Functional.LTP_one_test/spec.json | 12 +++++
> > engine/tests/Functional.LTP_one_test/test.yaml | 31
> +++++++++++++
> > 4 files changed, 134 insertions(+)
> > create mode 100755 engine/tests/Functional.LTP_one_test/fuego_test.sh
> > create mode 100755 engine/tests/Functional.LTP_one_test/parser.py
> > create mode 100644 engine/tests/Functional.LTP_one_test/spec.json
> > create mode 100644 engine/tests/Functional.LTP_one_test/test.yaml
> >
> > diff --git a/engine/tests/Functional.LTP_one_test/fuego_test.sh
> > b/engine/tests/Functional.LTP_one_test/fuego_test.sh
> > new file mode 100755
> > index 0000000..2ebcfc5
> > --- /dev/null
> > +++ b/engine/tests/Functional.LTP_one_test/fuego_test.sh
> > @@ -0,0 +1,40 @@
> > +one_test=$FUNCTIONAL_LTP_ONE_TEST_TEST
> > +
> > +# Don't allow jobs to share build directories
> > +# the "test_successfully_built" flag for one spec
> > +# is not accurate for another spec, which has a different
> > +# individual program to deploy.
> > +FUNCTIONAL_LTP_ONE_TEST_PER_JOB_BUILD="true"
> > +
> > +function test_pre_check {
> > + assert_define FUNCTIONAL_LTP_ONE_TEST_TEST
> > +}
> > +
> > +function test_build {
> > + # check for LTP build directory
> > + LTP_BUILD_DIR=$(echo $JOB_BUILD_DIR | sed s/LTP_one_test/LTP/ | sed
> > s/$TESTSPEC/default/)
> > +
> > + # if not already built, build LTP
> > + if [ ! -e
> ${WORKSPACE}/${LTP_BUILD_DIR}/fuego_test_succesfully_built ] ;
> > then
> > + echo "Building parent LTP test..."
> > + ftc run-test -b $NODE_NAME -t Functional.LTP -p b
> > + # NOTE: vars used in ftc run-test should not leak into this
> environment
> > + # that is, none of our test vars should have changed.
> > + fi
> > +
> > + cp
> ${WORKSPACE}/${LTP_BUILD_DIR}/target_bin/testcases/bin/$one_test .
> > +}
> > +
> > +function test_deploy {
> > + put $one_test $BOARD_TESTDIR/fuego.$TESTDIR/
> > +}
> > +
> > +function test_run {
> > + # FIXTHIS - deal with arguments better
> > + report "cd $BOARD_TESTDIR/fuego.$TESTDIR; \
> > + ./$one_test $FUNCTIONAL_LTP_ONE_TEST_ARGS"
> > +}
> > +
> > +function test_processing {
> > + return
> > +}
> > diff --git a/engine/tests/Functional.LTP_one_test/parser.py
> > b/engine/tests/Functional.LTP_one_test/parser.py
> > new file mode 100755
> > index 0000000..fe2336f
> > --- /dev/null
> > +++ b/engine/tests/Functional.LTP_one_test/parser.py
> > @@ -0,0 +1,51 @@
> > +#!/usr/bin/python
> > +# See common.py for description of command-line arguments
> > +
> > +import os
> > +import sys
> > +import collections
> > +import re
> > +
> > +sys.path.insert(0, os.environ['FUEGO_CORE'] +
> '/engine/scripts/parser')
> > +import common as plib
> > +
> > +results = collections.OrderedDict()
> > +
> > +# LTP results come in largely 2 formats:
> > +# sched_getattr01 1 TPASS : attributes were read back correctly
> > +# request_key01.c:49: PASS: request_key() succeed
> > +
> > +log_lines = open(plib.TEST_LOG, "r").readlines()
> > +re_pattern = r"(\S+)\s+(\S)+\s+(TPASS|TCONF)\s+:(.*)"
> > +
> > +testcase_count = 1
> > +for line in log_lines:
> > + #print "line=", line
> > + if ".c:" in line:
> > + parts = line.split(":")
> > + filename = parts[0]
> > + line_no = parts[1]
> > + testcase_num = str(testcase_count)
> > + testcase_count += 1
> > + result = parts[2].strip()
> > + message = parts[3].strip()
> > + else:
> > + m = re.match(re_pattern, line)
> > + #print "m=", m
> > + if m:
> > + test_name = m.group(1)
> > + testcase_num = m.group(2)
> > + result = m.group(3)[1:] # strip off the leading 'T'
> > + message = m.group(4).strip()
> > +
> > + # translate to Fuego results
> > + if result == 'PASS':
> > + results[testcase_num] = 'PASS'
> > + elif result == 'FAIL':
> > + results[testcase_num] = 'FAIL'
> > + elif result == 'CONF':
> > + results[testcase_num] = 'SKIP'
> > + else:
> > + results[testcase_num] = 'ERR'
> > +
> > +sys.exit(plib.process(results))
> > diff --git a/engine/tests/Functional.LTP_one_test/spec.json
> > b/engine/tests/Functional.LTP_one_test/spec.json
> > new file mode 100644
> > index 0000000..9f0b325
> > --- /dev/null
> > +++ b/engine/tests/Functional.LTP_one_test/spec.json
> > @@ -0,0 +1,12 @@
> > +{
> > + "testName": "Functional.LTP_one_test",
> > + "specs": {
> > + "default": {
> > + "TEST":"brk01"
> > + },
> > + "fanotify07": {
> > + "TEST":"fanotify07"
> > + }
> > + }
> > +}
> > +
> > diff --git a/engine/tests/Functional.LTP_one_test/test.yaml
> > b/engine/tests/Functional.LTP_one_test/test.yaml
> > new file mode 100644
> > index 0000000..765c98d
> > --- /dev/null
> > +++ b/engine/tests/Functional.LTP_one_test/test.yaml
> > @@ -0,0 +1,31 @@
> > +fuego_package_version: 1
> > +name: Functional.LTP_one_test
> > +description: |
> > + This is a convenience test. LTP is a large collection of tests,
> which
> > + takes a lot of infrastructure to set up and execute. Sometimes it
> is
> > + handy to run a single LTP test program by itself. This Fuego test
> > + exists to allow you to easily do that.
> > +
> > + To run an individual test, add a new spec for it in the spec.json
> file.
> > + Specify the individual test name to run in the TEST variable. By
> > + convention the name of the spec should match the name of the test
> > + to run.
> > +license: multiple (same as LTP)
> > +author: Tim Bird
> > +maintainer: Tim Bird <tim.bird at sony.com>
> > +version: 1.0
> > +fuego_release: 1
> > +type: Functional
> > +tags: ['general']
> > +params:
> > + - TEST:
> > + description: Name of the individual LTP test program to run.
> > + example: fanotify07
> > + optional: no
> > + - ARGS:
> > + description: Command line arguments for the LTP test program.
> > + example: -s foo
> > + optional: yes
> > +data_files:
> > + - spec.json
> > + - parser.py
> > --
> > 1.9.1
> >
>
>
>
>
More information about the Fuego
mailing list