[Fuego] [PATCH] LTP_one_test: Support use of runltp and ltp-pan

Tim.Bird at sony.com Tim.Bird at sony.com
Wed Apr 18 03:32:26 UTC 2018


> -----Original Message-----
> From: Liu, Wenlong
> 
> > -----Original Message-----
> > From: Tim Bird
> > OK - Here's an update to LTP_one_test that offers more flexibility when
> > running a single test.
> > ----
> > Add support to the test to run the individual test program on the board
> > either directly (supported previously) or using runltp and ltp-pan.  The
> > benefit of running directly is that it uses a simpler invocation, and you
> > get less 'noise' in the output.
> > The advantage of running using runltp and ltp-pan is that you get more
> setup
> > and test management.
> >
> > In order to run using runltp, the user must specify the 'scenario'
> > file (in the runtest directory) which has an entry for the specified test.
> >
> > Add some additional specs to demonstrate how to do this.
> >
> > Signed-off-by: Tim Bird <tim.bird at sony.com>
> > ---
> >  engine/tests/Functional.LTP_one_test/fuego_test.sh | 54
> > +++++++++++++++++++---
> >  engine/tests/Functional.LTP_one_test/parser.py     | 25 ++++++----
> >  engine/tests/Functional.LTP_one_test/spec.json     | 11 ++++-
> >  engine/tests/Functional.LTP_one_test/test.yaml     | 12 +++++
> >  4 files changed, 84 insertions(+), 18 deletions(-)
> >
> > diff --git a/engine/tests/Functional.LTP_one_test/fuego_test.sh
> > b/engine/tests/Functional.LTP_one_test/fuego_test.sh
> > index 1d99d9c..deb413e 100755
> > --- a/engine/tests/Functional.LTP_one_test/fuego_test.sh
> > +++ b/engine/tests/Functional.LTP_one_test/fuego_test.sh
> > @@ -6,33 +6,73 @@ one_test=$FUNCTIONAL_LTP_ONE_TEST_TEST
> >  # individual program to deploy.
> >  FUNCTIONAL_LTP_ONE_TEST_PER_JOB_BUILD="true"
> >
> > +# This test can either run the test directly, or run it via # runltp
> > +and ltp-pan.  To use runltp, set the value of "scenario"
> > +# in the spec to the correct scenario name.  (e.g. 'syscalls') # The
> > +scenario is the name of the the file in the runtest directory # that
> > +has an entry (possibly with command line arguments) for this test.
> > +
> >  function test_pre_check {
> >      assert_define FUNCTIONAL_LTP_ONE_TEST_TEST  }
> >
> >  function test_build {
> >      # check for LTP build directory
> > -    LTP_BUILD_DIR=$(echo $JOB_BUILD_DIR | sed s/LTP_one_test/LTP/ |
> sed
> > s/$TESTSPEC/default/)
> > +    LTP_BUILD_DIR="${WORKSPACE}/$(echo $JOB_BUILD_DIR | sed
> > s/LTP_one_test/LTP/ | sed s/$TESTSPEC/default/)"
> > +    echo "LTP_BUILD_DIR=${LTP_BUILD_DIR}"
> >
> >      # if not already built, build LTP
> > -    if [ ! -e
> > ${WORKSPACE}/${LTP_BUILD_DIR}/fuego_test_successfully_built ] ; then
> > +    if [ ! -e ${LTP_BUILD_DIR}/fuego_test_successfully_built ] ; then
> >          echo "Building parent LTP test..."
> >          ftc run-test -b $NODE_NAME -t Functional.LTP -p b
> >          # NOTE: vars used in ftc run-test should not leak into this
> > environment
> >          # that is, none of our test vars should have changed.
> >      fi
> >
> > -    cp
> > ${WORKSPACE}/${LTP_BUILD_DIR}/target_bin/testcases/bin/$one_test .
> > +    cp ${LTP_BUILD_DIR}/target_bin/testcases/bin/$one_test .
> >  }
> >
> >  function test_deploy {
> > -    put $one_test $BOARD_TESTDIR/fuego.$TESTDIR/
> > +    # set LTP_BUILD_DIR (possibly again), in case test_build was skipped
> > +    LTP_BUILD_DIR="${WORKSPACE}/$(echo $JOB_BUILD_DIR | sed
> > s/LTP_one_test/LTP/ | sed s/$TESTSPEC/default/)"
> > +
> > +    local bdir="$BOARD_TESTDIR/fuego.$TESTDIR"
> > +    local scenario=$FUNCTIONAL_LTP_ONE_TEST_SCENARIO
> > +    if [ -z "$scenario" ] ; then
> 
> Users can specify different options(arguments) for specified cases, so, I think
> you prefer to use this test for debugging some special cases, right?
> If not, if it'll be better that we can find the "scenario" by grep(ing) the
> "one_test" in runtest folder, and then we also run it with "runltp"?

Yes.  This is primarily to be able to run tests that are failing in LTP, in isolation
from the rest of the test.  This gives faster turn-around time when trying to 
reproduce bugs.  So this is really for debugging special cases (as you state).

This is not intended to be the primary method of running a test, and due
to the issues you point out it is not robust.

It would be possible for some tests to automatically find the scenario
file by grepping in the runtest directory. However, some tests show up in
multiple scenario files, and it's not clear which one the user would prefer
to run (for example rwtest01 shows up in 7 different scenario files).

I could possibly do something, where if the user specifies "*" for the scenario
file, then I grep for the test, and use the first one found.  What do you think
of that?

> > +        put $one_test $bdir/
> > +        # if the user hasn't specified any arguments,
> > +        # warn user if there are default args
> > +        if [ -z "$FUNCTIONAL_LTP_ONE_TEST_ARGS" ] ; then
> > +            args=$(grep $one_test ${LTP_BUILD_DIR}/runtest/* | cut -d"
> > " -f3- | head -n 1)
> > +            if [ -n "$args" ] ; then
> > +                echo "Warning: No arguments specified, but at least one
> > scenario group for"
> > +                echo "test '$one_test' has args of '${args}'"
> > +                echo "Maybe add \"ARGS\":\"${args}\" in spec file?"
> > +            fi
> > +        fi
> > +    else
> > +        # copy helper files, runltp, ltp-pan, scenario file and the
> > +        # test program to the board
> > +        cmd "mkdir -p $bdir/bin $bdir/runtest $bdir/testcases/bin"
> > +        put ${LTP_BUILD_DIR}/target_bin/IDcheck.sh $bdir/
> > +        put ${LTP_BUILD_DIR}/target_bin/ver_linux $bdir/
> > +        put ${LTP_BUILD_DIR}/target_bin/Version $bdir/
> > +        put ${LTP_BUILD_DIR}/target_bin/runltp $bdir/
> > +        put ${LTP_BUILD_DIR}/target_bin/bin/ltp-pan $bdir/bin/
> > +        put ${LTP_BUILD_DIR}/target_bin/runtest/$scenario
> > $bdir/runtest/
> > +        put $one_test $bdir/testcases/bin/
> 
> I think this test can only test those binary based cases currently, e.g. cases in
> scenario "syscalls".
> For those cases using scripts(example with scenario "commands" as below)
> root at porter:/opt/ltp/runtest# head -n10 commands
> #DESCRIPTION:General Linux commands
> ......
> logrotate export TCdat=$LTPROOT/testcases/bin; logrotate_tests.sh
> mail export TCdat=$LTPROOT/testcases/bin; mail_tests.sh
> ......
> 
> I have not tested this yet. But I think testcase like "logrotate" in scenario
> "command" cannot works well.
> I am not sure, but I think those cases using scripts have some extra
> dependencies.

Yes.  They do.  I'm not currently copying any extra scripts, or checking for other
programs invoked in the command line in a scenario file.  Also, I'm not
copying anything in the target_bin/testcases/data directory.

So there are many tests that this won't work for.

Another approach is to install everything (use LTP's deploy), and then run a single
test with "runltp -f $scenario -s $one_test", to solve the dependency problem.
I'd like to use this for a while as is, and see if I run into those problems.
If I do, I'll start looking at methods of supporting those other LTP tests.

> 
> And one more thing, sometimes we cannot just copy files *named* "TEST" in
> spec.json to the target.
Are you saying that, for purposes of ltp-pan, the argument to -s is supposed
to match the tag and not the test program name?  Yes, that's true.  For mosts
tests they are the same, but for some they are not.

> 
> Welcome to point out if there is something wrong or missing.

I think your analysis is correct.  Thank you for looking at it and giving some feedback.

Another method of dealing with this issue (trying to run a single sub-test of LTP), would
be to use a full LTP install, and create a scenario file on the fly, with just the
one LTP test we want to run.  Then use that scenario file with something like:
'runltp -f fuego-onetest'

...

 -- Tim



More information about the Fuego mailing list