[Fuego] [PATCH] LTP_one_test: create a Fuego test for a single LTP test

[Tim.Bird at sony.com]

Hey Daniel and Liu,

Here is a test that runs an individual LTP test.  It's now in my master
branch.  The reason for pointing this out is to get your feedback
on the approach of writing a test that uses a sub-set of LTP.
Note the use of 'ftc run-test' with the '-p  b' parameter, to build
LTP if needed.  This avoids having to share build script details
between this test and LTP.

I'm thinking this approach might be useful to split out
the OpenPosix and realtime subsets of LTP.  Also, it might
work for handling setup and cleanup for subtest groups or
individual tests, like the network-related tests or smack.

For example, instead of doing smack-specific machine setup
in the LTP test, I think we should make something like
Functional.LTP-smack, use the techniques demonstrated here
for building it.  But then have  pre_test, deply, run, and processing
phases be customized for that test's needs. in Functional.LTP-smack's
fuego_test.sh.

Let me know what you think.
   -- Tim

Patch follows:
----------------------------
LTP is a ginormous collection of tests.  When something fails,
it is very handy to be able to run a single test in isolation.
This test allows you to do that.  To run an individual test,
just add a new spec to the spec file (or, when ftc supports
setting a test variable on the command line, use that to
specify the name of the individual test program to run).

This test also demonstrates how one test can use ftc and phases
to execute just the build of another test.  And how one test
can re-use the build directory of another test.

Signed-off-by: Tim Bird <tim.bird at sony.com>
---
 engine/tests/Functional.LTP_one_test/fuego_test.sh | 40 +++++++++++++++++
 engine/tests/Functional.LTP_one_test/parser.py     | 51 ++++++++++++++++++++++
 engine/tests/Functional.LTP_one_test/spec.json     | 12 +++++
 engine/tests/Functional.LTP_one_test/test.yaml     | 31 +++++++++++++
 4 files changed, 134 insertions(+)
 create mode 100755 engine/tests/Functional.LTP_one_test/fuego_test.sh
 create mode 100755 engine/tests/Functional.LTP_one_test/parser.py
 create mode 100644 engine/tests/Functional.LTP_one_test/spec.json
 create mode 100644 engine/tests/Functional.LTP_one_test/test.yaml

diff --git a/engine/tests/Functional.LTP_one_test/fuego_test.sh b/engine/tests/Functional.LTP_one_test/fuego_test.sh
new file mode 100755
index 0000000..2ebcfc5
--- /dev/null
+++ b/engine/tests/Functional.LTP_one_test/fuego_test.sh
@@ -0,0 +1,40 @@
+one_test=$FUNCTIONAL_LTP_ONE_TEST_TEST
+
+# Don't allow jobs to share build directories
+# the "test_successfully_built" flag for one spec
+# is not accurate for another spec, which has a different
+# individual program to deploy.
+FUNCTIONAL_LTP_ONE_TEST_PER_JOB_BUILD="true"
+
+function test_pre_check {
+    assert_define FUNCTIONAL_LTP_ONE_TEST_TEST
+}
+
+function test_build {
+    # check for LTP build directory
+    LTP_BUILD_DIR=$(echo $JOB_BUILD_DIR | sed s/LTP_one_test/LTP/ | sed s/$TESTSPEC/default/)
+
+    # if not already built, build LTP
+    if [ ! -e ${WORKSPACE}/${LTP_BUILD_DIR}/fuego_test_succesfully_built ] ; then
+        echo "Building parent LTP test..."
+	    ftc run-test -b $NODE_NAME -t Functional.LTP -p b
+        # NOTE: vars used in ftc run-test should not leak into this environment
+        # that is, none of our test vars should have changed.
+    fi
+
+    cp ${WORKSPACE}/${LTP_BUILD_DIR}/target_bin/testcases/bin/$one_test .
+}
+
+function test_deploy {
+	put $one_test $BOARD_TESTDIR/fuego.$TESTDIR/
+}
+
+function test_run {
+    # FIXTHIS - deal with arguments better
+    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; \
+        ./$one_test $FUNCTIONAL_LTP_ONE_TEST_ARGS"
+}
+
+function test_processing {
+    return
+}
diff --git a/engine/tests/Functional.LTP_one_test/parser.py b/engine/tests/Functional.LTP_one_test/parser.py
new file mode 100755
index 0000000..fe2336f
--- /dev/null
+++ b/engine/tests/Functional.LTP_one_test/parser.py
@@ -0,0 +1,51 @@
+#!/usr/bin/python
+# See common.py for description of command-line arguments
+
+import os
+import sys
+import collections
+import re
+
+sys.path.insert(0, os.environ['FUEGO_CORE'] + '/engine/scripts/parser')
+import common as plib
+
+results = collections.OrderedDict()
+
+# LTP results come in largely 2 formats:
+# sched_getattr01    1  TPASS  :  attributes were read back correctly
+# request_key01.c:49: PASS: request_key() succeed
+
+log_lines = open(plib.TEST_LOG, "r").readlines()
+re_pattern = r"(\S+)\s+(\S)+\s+(TPASS|TCONF)\s+:(.*)"
+
+testcase_count = 1
+for line in log_lines:
+    #print "line=", line
+    if ".c:" in line:
+        parts = line.split(":")
+        filename = parts[0]
+        line_no = parts[1]
+        testcase_num = str(testcase_count)
+        testcase_count += 1
+        result = parts[2].strip()
+        message = parts[3].strip()
+    else:
+        m = re.match(re_pattern, line)
+        #print "m=", m
+        if m:
+            test_name = m.group(1)
+            testcase_num = m.group(2)
+            result = m.group(3)[1:] # strip off the leading 'T'
+            message = m.group(4).strip()
+
+    # translate to Fuego results
+    if result == 'PASS':
+        results[testcase_num] = 'PASS'
+    elif result == 'FAIL':
+        results[testcase_num] = 'FAIL'
+    elif result == 'CONF':
+        results[testcase_num] = 'SKIP'
+    else:
+        results[testcase_num] = 'ERR'
+
+sys.exit(plib.process(results))
diff --git a/engine/tests/Functional.LTP_one_test/spec.json b/engine/tests/Functional.LTP_one_test/spec.json
new file mode 100644
index 0000000..9f0b325
--- /dev/null
+++ b/engine/tests/Functional.LTP_one_test/spec.json
@@ -0,0 +1,12 @@
+{
+    "testName": "Functional.LTP_one_test",
+    "specs": {
+        "default": {
+            "TEST":"brk01"
+        },
+        "fanotify07": {
+            "TEST":"fanotify07"
+        }
+    }
+}
+
diff --git a/engine/tests/Functional.LTP_one_test/test.yaml b/engine/tests/Functional.LTP_one_test/test.yaml
new file mode 100644
index 0000000..765c98d
--- /dev/null
+++ b/engine/tests/Functional.LTP_one_test/test.yaml
@@ -0,0 +1,31 @@
+fuego_package_version: 1
+name: Functional.LTP_one_test
+description: |
+    This is a convenience test.  LTP is a large collection of tests, which
+    takes a lot of infrastructure to set up and execute.  Sometimes it is
+    handy to run a single LTP test program by itself.  This Fuego test
+    exists to allow you to easily do that.
+
+    To run an individual test, add a new spec for it in the spec.json file.
+    Specify the individual test name to run in the TEST variable.  By
+    convention the name of the spec should match the name of the test
+    to run.
+license: multiple (same as LTP)
+author: Tim Bird
+maintainer: Tim Bird <tim.bird at sony.com>
+version: 1.0
+fuego_release: 1
+type: Functional
+tags: ['general']
+params:
+    - TEST:
+        description: Name of the individual LTP test program to run.
+        example: fanotify07
+        optional: no
+    - ARGS:
+        description: Command line arguments for the LTP test program.
+        example: -s foo
+        optional: yes
+data_files:
+    - spec.json
+    - parser.py
-- 
1.9.1