[Fuego] [PATCH] LTP_one_test: create a Fuego test for a single LTP test

Liu, Wenlong liuwl.fnst at cn.fujitsu.com
Mon Apr 16 05:36:50 UTC 2018


> -----Original Message-----
> From: Daniel Sangorrin [mailto:daniel.sangorrin at toshiba.co.jp]
> Sent: Tuesday, April 10, 2018 10:38 AM
> To: Tim.Bird at sony.com; Liu, Wenlong/刘 文龙 <liuwl.fnst at cn.fujitsu.com>;
> fuego at lists.linuxfoundation.org
> Subject: RE: [PATCH] LTP_one_test: create a Fuego test for a single LTP
> test
> 
> Hi Tim,
> 
> > -----Original Message-----
> > From: Tim.Bird at sony.com [mailto:Tim.Bird at sony.com]
> > Sent: Saturday, April 7, 2018 4:11 AM
> > To: liuwl.fnst at cn.fujitsu.com; daniel.sangorrin at toshiba.co.jp;
> > fuego at lists.linuxfoundation.org
> > Subject: [PATCH] LTP_one_test: create a Fuego test for a single LTP
> > test
> >
> > Hey Daniel and Liu,
> >
> > Here is a test that runs an individual LTP test.  It's now in my
> > master branch.  The reason for pointing this out is to get your
> > feedback on the approach of writing a test that uses a sub-set of LTP.
> 
> Let me add some background for everyone to understand.
> 
> - On Functional.LTP, it is possible to run sub-sets of LTP by using a skip
> list. In other words, you first select the test sets (groups of test cases)
> that you want to run, and then you apply a skip list to mask the execution
> of some of the test cases. Let's call this a blacklisting approach. The
> main advantage is that you only need to know the name of the test cases
> that you want to skip.
> - On Functional.LTP_one_test, it is possible to specify what test case to
> execute directly. For that reason there is no need to create any skip list.
> The main advantage is that it allows the user to quickly run any of the
> tests inside LTP.
> 
> Let me jump into one technical detail though. I see that you are calling
> the test cases directly from /testcases/bin/. I don't think that works in
> general because some of the test cases require a bit of setup before they
> are executed. Fortunately, this is already taken care by runltp through
> the "-s" option.
>     $ ./runltp -q -f math -s abs01
>     [Note] remove -q to see the test output as well

Yes, Daniel, I have the same thought with you.

> > Note the use of 'ftc run-test' with the '-p  b' parameter, to build
> > LTP if needed.  This avoids having to share build script details
> > between this test and LTP.
> 
> Maybe it's because the recent update broke everything, but "ftc run-test"
> and "ftc build-jobs" didn't work on my machine.
> Other than that, I like the ability to use "ftc run-test" within the tests.
> In this case for the building phase, but I want to use it for creating
> background loads as well.
> 
> Do you think there could be some conflict with the nesting of tests such
> as variable overwriting (Fuego using all these global shell variables is
> a bit scary) or deadlocks?
> 
> > I'm thinking this approach might be useful to split out the OpenPosix
> > and realtime subsets of LTP.
> Sounds good.
> 
> > Also, it might
> > work for handling setup and cleanup for subtest groups or
> > individual tests, like the network-related tests or smack.
> > For example, instead of doing smack-specific machine setup
> > in the LTP test, I think we should make something like
> > Functional.LTP-smack, use the techniques demonstrated here
> > for building it.  But then have  pre_test, deply, run, and processing
> > phases be customized for that test's needs. in Functional.LTP-smack's
> > fuego_test.sh.
> 
> Yeah, let's split corner cases. Otherwise things will get too complicated
> to handle.
> 
> Thanks,
> Daniel
> 
> >
> > Let me know what you think.
> >    -- Tim
> >
> > Patch follows:
> > ----------------------------
> > LTP is a ginormous collection of tests.  When something fails,
> > it is very handy to be able to run a single test in isolation.
> > This test allows you to do that.  To run an individual test,
> > just add a new spec to the spec file (or, when ftc supports
> > setting a test variable on the command line, use that to
> > specify the name of the individual test program to run).
> >
> > This test also demonstrates how one test can use ftc and phases
> > to execute just the build of another test.  And how one test
> > can re-use the build directory of another test.
> >
> > Signed-off-by: Tim Bird <tim.bird at sony.com>
> > ---
> >  engine/tests/Functional.LTP_one_test/fuego_test.sh | 40
> > +++++++++++++++++
> >  engine/tests/Functional.LTP_one_test/parser.py     | 51
> > ++++++++++++++++++++++
> >  engine/tests/Functional.LTP_one_test/spec.json     | 12 +++++
> >  engine/tests/Functional.LTP_one_test/test.yaml     | 31
> +++++++++++++
> >  4 files changed, 134 insertions(+)
> >  create mode 100755 engine/tests/Functional.LTP_one_test/fuego_test.sh
> >  create mode 100755 engine/tests/Functional.LTP_one_test/parser.py
> >  create mode 100644 engine/tests/Functional.LTP_one_test/spec.json
> >  create mode 100644 engine/tests/Functional.LTP_one_test/test.yaml
> >
> > diff --git a/engine/tests/Functional.LTP_one_test/fuego_test.sh
> > b/engine/tests/Functional.LTP_one_test/fuego_test.sh
> > new file mode 100755
> > index 0000000..2ebcfc5
> > --- /dev/null
> > +++ b/engine/tests/Functional.LTP_one_test/fuego_test.sh
> > @@ -0,0 +1,40 @@
> > +one_test=$FUNCTIONAL_LTP_ONE_TEST_TEST
> > +
> > +# Don't allow jobs to share build directories
> > +# the "test_successfully_built" flag for one spec
> > +# is not accurate for another spec, which has a different
> > +# individual program to deploy.
> > +FUNCTIONAL_LTP_ONE_TEST_PER_JOB_BUILD="true"
> > +
> > +function test_pre_check {
> > +    assert_define FUNCTIONAL_LTP_ONE_TEST_TEST
> > +}
> > +
> > +function test_build {
> > +    # check for LTP build directory
> > +    LTP_BUILD_DIR=$(echo $JOB_BUILD_DIR | sed s/LTP_one_test/LTP/ | sed
> > s/$TESTSPEC/default/)
> > +
> > +    # if not already built, build LTP
> > +    if [ ! -e
> ${WORKSPACE}/${LTP_BUILD_DIR}/fuego_test_succesfully_built ] ;
> > then
> > +        echo "Building parent LTP test..."
> > +	    ftc run-test -b $NODE_NAME -t Functional.LTP -p b
> > +        # NOTE: vars used in ftc run-test should not leak into this
> environment
> > +        # that is, none of our test vars should have changed.
> > +    fi
> > +
> > +    cp
> ${WORKSPACE}/${LTP_BUILD_DIR}/target_bin/testcases/bin/$one_test .
> > +}
> > +
> > +function test_deploy {
> > +	put $one_test $BOARD_TESTDIR/fuego.$TESTDIR/
> > +}
> > +
> > +function test_run {
> > +    # FIXTHIS - deal with arguments better
> > +    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; \
> > +        ./$one_test $FUNCTIONAL_LTP_ONE_TEST_ARGS"
> > +}
> > +
> > +function test_processing {
> > +    return
> > +}
> > diff --git a/engine/tests/Functional.LTP_one_test/parser.py
> > b/engine/tests/Functional.LTP_one_test/parser.py
> > new file mode 100755
> > index 0000000..fe2336f
> > --- /dev/null
> > +++ b/engine/tests/Functional.LTP_one_test/parser.py
> > @@ -0,0 +1,51 @@
> > +#!/usr/bin/python
> > +# See common.py for description of command-line arguments
> > +
> > +import os
> > +import sys
> > +import collections
> > +import re
> > +
> > +sys.path.insert(0, os.environ['FUEGO_CORE'] +
> '/engine/scripts/parser')
> > +import common as plib
> > +
> > +results = collections.OrderedDict()
> > +
> > +# LTP results come in largely 2 formats:
> > +# sched_getattr01    1  TPASS  :  attributes were read back correctly
> > +# request_key01.c:49: PASS: request_key() succeed
> > +
> > +log_lines = open(plib.TEST_LOG, "r").readlines()
> > +re_pattern = r"(\S+)\s+(\S)+\s+(TPASS|TCONF)\s+:(.*)"
> > +
> > +testcase_count = 1
> > +for line in log_lines:
> > +    #print "line=", line
> > +    if ".c:" in line:
> > +        parts = line.split(":")
> > +        filename = parts[0]
> > +        line_no = parts[1]
> > +        testcase_num = str(testcase_count)
> > +        testcase_count += 1
> > +        result = parts[2].strip()
> > +        message = parts[3].strip()
> > +    else:
> > +        m = re.match(re_pattern, line)
> > +        #print "m=", m
> > +        if m:
> > +            test_name = m.group(1)
> > +            testcase_num = m.group(2)
> > +            result = m.group(3)[1:] # strip off the leading 'T'
> > +            message = m.group(4).strip()
> > +
> > +    # translate to Fuego results
> > +    if result == 'PASS':
> > +        results[testcase_num] = 'PASS'
> > +    elif result == 'FAIL':
> > +        results[testcase_num] = 'FAIL'
> > +    elif result == 'CONF':
> > +        results[testcase_num] = 'SKIP'
> > +    else:
> > +        results[testcase_num] = 'ERR'
> > +
> > +sys.exit(plib.process(results))
> > diff --git a/engine/tests/Functional.LTP_one_test/spec.json
> > b/engine/tests/Functional.LTP_one_test/spec.json
> > new file mode 100644
> > index 0000000..9f0b325
> > --- /dev/null
> > +++ b/engine/tests/Functional.LTP_one_test/spec.json
> > @@ -0,0 +1,12 @@
> > +{
> > +    "testName": "Functional.LTP_one_test",
> > +    "specs": {
> > +        "default": {
> > +            "TEST":"brk01"
> > +        },
> > +        "fanotify07": {
> > +            "TEST":"fanotify07"
> > +        }
> > +    }
> > +}
> > +
> > diff --git a/engine/tests/Functional.LTP_one_test/test.yaml
> > b/engine/tests/Functional.LTP_one_test/test.yaml
> > new file mode 100644
> > index 0000000..765c98d
> > --- /dev/null
> > +++ b/engine/tests/Functional.LTP_one_test/test.yaml
> > @@ -0,0 +1,31 @@
> > +fuego_package_version: 1
> > +name: Functional.LTP_one_test
> > +description: |
> > +    This is a convenience test.  LTP is a large collection of tests,
> which
> > +    takes a lot of infrastructure to set up and execute.  Sometimes it
> is
> > +    handy to run a single LTP test program by itself.  This Fuego test
> > +    exists to allow you to easily do that.
> > +
> > +    To run an individual test, add a new spec for it in the spec.json
> file.
> > +    Specify the individual test name to run in the TEST variable.  By
> > +    convention the name of the spec should match the name of the test
> > +    to run.
> > +license: multiple (same as LTP)
> > +author: Tim Bird
> > +maintainer: Tim Bird <tim.bird at sony.com>
> > +version: 1.0
> > +fuego_release: 1
> > +type: Functional
> > +tags: ['general']
> > +params:
> > +    - TEST:
> > +        description: Name of the individual LTP test program to run.
> > +        example: fanotify07
> > +        optional: no
> > +    - ARGS:
> > +        description: Command line arguments for the LTP test program.
> > +        example: -s foo
> > +        optional: yes
> > +data_files:
> > +    - spec.json
> > +    - parser.py
> > --
> > 1.9.1
> >
> 
> 
> 
> 





More information about the Fuego mailing list