[Fuego] [PATCH] LTP_one_test: Support use of runltp and ltp-pan

Liu, Wenlong liuwl.fnst at cn.fujitsu.com
Wed Apr 18 02:02:00 UTC 2018


> -----Original Message-----
> From: fuego-bounces at lists.linuxfoundation.org
> [mailto:fuego-bounces at lists.linuxfoundation.org] On Behalf Of
> Tim.Bird at sony.com
> Sent: Wednesday, April 18, 2018 9:09 AM
> To: fuego at lists.linuxfoundation.org
> Subject: [Fuego] [PATCH] LTP_one_test: Support use of runltp and ltp-pan
> 
> OK - Here's an update to LTP_one_test that offers more flexibility when
> running a single test.
> ----
> Add support to the test to run the individual test program on the board
> either directly (supported previously) or using runltp and ltp-pan.  The
> benefit of running directly is that it uses a simpler invocation, and you
> get less 'noise' in the output.
> The advantage of running using runltp and ltp-pan is that you get more setup
> and test management.
> 
> In order to run using runltp, the user must specify the 'scenario'
> file (in the runtest directory) which has an entry for the specified test.
> 
> Add some additional specs to demonstrate how to do this.
> 
> Signed-off-by: Tim Bird <tim.bird at sony.com>
> ---
>  engine/tests/Functional.LTP_one_test/fuego_test.sh | 54
> +++++++++++++++++++---
>  engine/tests/Functional.LTP_one_test/parser.py     | 25 ++++++----
>  engine/tests/Functional.LTP_one_test/spec.json     | 11 ++++-
>  engine/tests/Functional.LTP_one_test/test.yaml     | 12 +++++
>  4 files changed, 84 insertions(+), 18 deletions(-)
> 
> diff --git a/engine/tests/Functional.LTP_one_test/fuego_test.sh
> b/engine/tests/Functional.LTP_one_test/fuego_test.sh
> index 1d99d9c..deb413e 100755
> --- a/engine/tests/Functional.LTP_one_test/fuego_test.sh
> +++ b/engine/tests/Functional.LTP_one_test/fuego_test.sh
> @@ -6,33 +6,73 @@ one_test=$FUNCTIONAL_LTP_ONE_TEST_TEST
>  # individual program to deploy.
>  FUNCTIONAL_LTP_ONE_TEST_PER_JOB_BUILD="true"
> 
> +# This test can either run the test directly, or run it via # runltp
> +and ltp-pan.  To use runltp, set the value of "scenario"
> +# in the spec to the correct scenario name.  (e.g. 'syscalls') # The
> +scenario is the name of the the file in the runtest directory # that
> +has an entry (possibly with command line arguments) for this test.
> +
>  function test_pre_check {
>      assert_define FUNCTIONAL_LTP_ONE_TEST_TEST  }
> 
>  function test_build {
>      # check for LTP build directory
> -    LTP_BUILD_DIR=$(echo $JOB_BUILD_DIR | sed s/LTP_one_test/LTP/ | sed
> s/$TESTSPEC/default/)
> +    LTP_BUILD_DIR="${WORKSPACE}/$(echo $JOB_BUILD_DIR | sed
> s/LTP_one_test/LTP/ | sed s/$TESTSPEC/default/)"
> +    echo "LTP_BUILD_DIR=${LTP_BUILD_DIR}"
> 
>      # if not already built, build LTP
> -    if [ ! -e
> ${WORKSPACE}/${LTP_BUILD_DIR}/fuego_test_successfully_built ] ; then
> +    if [ ! -e ${LTP_BUILD_DIR}/fuego_test_successfully_built ] ; then
>          echo "Building parent LTP test..."
>          ftc run-test -b $NODE_NAME -t Functional.LTP -p b
>          # NOTE: vars used in ftc run-test should not leak into this
> environment
>          # that is, none of our test vars should have changed.
>      fi
> 
> -    cp
> ${WORKSPACE}/${LTP_BUILD_DIR}/target_bin/testcases/bin/$one_test .
> +    cp ${LTP_BUILD_DIR}/target_bin/testcases/bin/$one_test .
>  }
> 
>  function test_deploy {
> -    put $one_test $BOARD_TESTDIR/fuego.$TESTDIR/
> +    # set LTP_BUILD_DIR (possibly again), in case test_build was skipped
> +    LTP_BUILD_DIR="${WORKSPACE}/$(echo $JOB_BUILD_DIR | sed
> s/LTP_one_test/LTP/ | sed s/$TESTSPEC/default/)"
> +
> +    local bdir="$BOARD_TESTDIR/fuego.$TESTDIR"
> +    local scenario=$FUNCTIONAL_LTP_ONE_TEST_SCENARIO
> +    if [ -z "$scenario" ] ; then

Users can specify different options(arguments) for specified cases, so, I think you prefer to use this test for debugging some special cases, right?
If not, if it'll be better that we can find the "scenario" by grep(ing) the "one_test" in runtest folder, and then we also run it with "runltp"?

> +        put $one_test $bdir/
> +        # if the user hasn't specified any arguments,
> +        # warn user if there are default args
> +        if [ -z "$FUNCTIONAL_LTP_ONE_TEST_ARGS" ] ; then
> +            args=$(grep $one_test ${LTP_BUILD_DIR}/runtest/* | cut -d"
> " -f3- | head -n 1)
> +            if [ -n "$args" ] ; then
> +                echo "Warning: No arguments specified, but at least one
> scenario group for"
> +                echo "test '$one_test' has args of '${args}'"
> +                echo "Maybe add \"ARGS\":\"${args}\" in spec file?"
> +            fi
> +        fi
> +    else
> +        # copy helper files, runltp, ltp-pan, scenario file and the
> +        # test program to the board
> +        cmd "mkdir -p $bdir/bin $bdir/runtest $bdir/testcases/bin"
> +        put ${LTP_BUILD_DIR}/target_bin/IDcheck.sh $bdir/
> +        put ${LTP_BUILD_DIR}/target_bin/ver_linux $bdir/
> +        put ${LTP_BUILD_DIR}/target_bin/Version $bdir/
> +        put ${LTP_BUILD_DIR}/target_bin/runltp $bdir/
> +        put ${LTP_BUILD_DIR}/target_bin/bin/ltp-pan $bdir/bin/
> +        put ${LTP_BUILD_DIR}/target_bin/runtest/$scenario
> $bdir/runtest/
> +        put $one_test $bdir/testcases/bin/

I think this test can only test those binary based cases currently, e.g. cases in scenario "syscalls".
For those cases using scripts(example with scenario "commands" as below)
root at porter:/opt/ltp/runtest# head -n10 commands
#DESCRIPTION:General Linux commands
......
logrotate export TCdat=$LTPROOT/testcases/bin; logrotate_tests.sh
mail export TCdat=$LTPROOT/testcases/bin; mail_tests.sh
......

I have not tested this yet. But I think testcase like "logrotate" in scenario "command" cannot works well.
I am not sure, but I think those cases using scripts have some extra dependencies. 

And one more thing, sometimes we cannot just copy files *named* "TEST" in spec.json to the target.

Welcome to point out if there is something wrong or missing.
Thanks.

Best regards
Liu

> +    fi
>  }
> 
>  function test_run {
> -    # FIXTHIS - deal with arguments better
> -    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; \
> -        ./$one_test $FUNCTIONAL_LTP_ONE_TEST_ARGS"
> +    local bdir="$BOARD_TESTDIR/fuego.$TESTDIR"
> +    local scenario=$FUNCTIONAL_LTP_ONE_TEST_SCENARIO
> +
> +    if [ -z "$scenario" ] ; then
> +        report "cd $bdir; ./$one_test $FUNCTIONAL_LTP_ONE_TEST_ARGS"
> +    else
> +        report "cd $bdir; ./runltp -f $scenario -s $one_test"
> +    fi
>  }
> 
>  function test_processing {
> diff --git a/engine/tests/Functional.LTP_one_test/parser.py
> b/engine/tests/Functional.LTP_one_test/parser.py
> index fe2336f..855efe1 100755
> --- a/engine/tests/Functional.LTP_one_test/parser.py
> +++ b/engine/tests/Functional.LTP_one_test/parser.py
> @@ -20,15 +20,19 @@ re_pattern = r"(\S+)\s+(\S)+\s+(TPASS|TCONF)\s+:(.*)"
> 
>  testcase_count = 1
>  for line in log_lines:
> +    result = None
>      #print "line=", line
>      if ".c:" in line:
>          parts = line.split(":")
>          filename = parts[0]
>          line_no = parts[1]
> -        testcase_num = str(testcase_count)
> -        testcase_count += 1
>          result = parts[2].strip()
>          message = parts[3].strip()
> +        if result == "INFO":
> +            result = None
> +        else:
> +            testcase_num = str(testcase_count)
> +            testcase_count += 1
>      else:
>          m = re.match(re_pattern, line)
>          #print "m=", m
> @@ -39,13 +43,14 @@ for line in log_lines:
>              message = m.group(4).strip()
> 
>      # translate to Fuego results
> -    if result == 'PASS':
> -        results[testcase_num] = 'PASS'
> -    elif result == 'FAIL':
> -        results[testcase_num] = 'FAIL'
> -    elif result == 'CONF':
> -        results[testcase_num] = 'SKIP'
> -    else:
> -        results[testcase_num] = 'ERR'
> +    if result:
> +        if result == 'PASS':
> +            results[testcase_num] = 'PASS'
> +        elif result == 'FAIL':
> +            results[testcase_num] = 'FAIL'
> +        elif result == 'CONF':
> +            results[testcase_num] = 'SKIP'
> +        else:
> +            results[testcase_num] = 'ERR'
> 
>  sys.exit(plib.process(results))
> diff --git a/engine/tests/Functional.LTP_one_test/spec.json
> b/engine/tests/Functional.LTP_one_test/spec.json
> index 9f0b325..cd78581 100644
> --- a/engine/tests/Functional.LTP_one_test/spec.json
> +++ b/engine/tests/Functional.LTP_one_test/spec.json
> @@ -4,9 +4,18 @@
>          "default": {
>              "TEST":"brk01"
>          },
> +        "fanotify06": {
> +            "TEST":"fanotify06"
> +        },
>          "fanotify07": {
>              "TEST":"fanotify07"
> +        },
> +        "mlock03": {
> +            "TEST":"mlock03"
> +        },
> +        "syscalls-mlock03": {
> +            "TEST":"mlock03",
> +            "scenario":"syscalls"
>          }
>      }
>  }
> -
> diff --git a/engine/tests/Functional.LTP_one_test/test.yaml
> b/engine/tests/Functional.LTP_one_test/test.yaml
> index 765c98d..2ff0b19 100644
> --- a/engine/tests/Functional.LTP_one_test/test.yaml
> +++ b/engine/tests/Functional.LTP_one_test/test.yaml
> @@ -26,6 +26,18 @@ params:
>          description: Command line arguments for the LTP test program.
>          example: -s foo
>          optional: yes
> +    - scenario:
> +        description: |
> +            Name of the LTP scenario file that has an entry for this test.
> +            The scenario files are located in LTP's 'runtest' directory.
> +            If this is specified, then Functional.LTP_one_test will
> +            invoke the test using runltp and ltp-pan, which should do
> +            more setup, and better match the environment when the test
> +            was run in the context of the full Functional.LTP test.
> +
> +            If this is not specified, then the test is run directly.
> +        example: syscalls
> +        optional: yes
>  data_files:
>      - spec.json
>      - parser.py
> --
> 2.1.4
> 
> _______________________________________________
> Fuego mailing list
> Fuego at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego
> 





More information about the Fuego mailing list