[Fuego] [PATCH] tests: add support for Linaro test-definitons
Daniel Sangorrin
daniel.sangorrin at toshiba.co.jp
Tue Feb 12 08:50:26 UTC 2019
This adds initial support for reusing Linaro test-definitions.
It is still a proof of concept and only tested with
smoke tests. I have written a few FIXTHIS to indicate what
is left.
To try it follow these steps:
- prepare SSH_KEY for your board
Eg: Inside fuego's docker container do
> su jenkins
> cp path/to/bbb_id_rsa ~/.ssh/
> vi ~/.ssh/config
> Host 192.167.1.99 <- replace with your boards ip address ($IPADDR)
> IdentityFile ~/.ssh/bbb_id_rsa
- ftc add-job -b bbb -t Functional.linaro
- execute the job from jenkins
- expected results
- table with each test case and the results (PASS/FAIL/SKIP)
- run.json
- csv
Signed-off-by: Daniel Sangorrin <daniel.sangorrin at toshiba.co.jp>
---
tests/Functional.linaro/chart_config.json | 3 ++
tests/Functional.linaro/fuego_test.sh | 59 +++++++++++++++++++++++++++++++
tests/Functional.linaro/parser.py | 25 +++++++++++++
tests/Functional.linaro/spec.json | 16 +++++++++
tests/Functional.linaro/test.yaml | 27 ++++++++++++++
5 files changed, 130 insertions(+)
create mode 100644 tests/Functional.linaro/chart_config.json
create mode 100755 tests/Functional.linaro/fuego_test.sh
create mode 100755 tests/Functional.linaro/parser.py
create mode 100644 tests/Functional.linaro/spec.json
create mode 100644 tests/Functional.linaro/test.yaml
diff --git a/tests/Functional.linaro/chart_config.json b/tests/Functional.linaro/chart_config.json
new file mode 100644
index 0000000..b8c8fb6
--- /dev/null
+++ b/tests/Functional.linaro/chart_config.json
@@ -0,0 +1,3 @@
+{
+ "chart_type": "testcase_table"
+}
diff --git a/tests/Functional.linaro/fuego_test.sh b/tests/Functional.linaro/fuego_test.sh
new file mode 100755
index 0000000..17b56a9
--- /dev/null
+++ b/tests/Functional.linaro/fuego_test.sh
@@ -0,0 +1,59 @@
+gitrepo="https://github.com/Linaro/test-definitions.git"
+
+# Root permissions required for
+# - installing dependencies on the target (debian/centos) when -s is not specified
+# - executing some of the tests
+# FIXTHIS: don't force root permissions for tests that do not require them
+NEED_ROOT=1
+
+function test_pre_check {
+ # linaro parser dependencies
+ # FIXTHIS: use dependencies specified in the test definition yaml
+ assert_has_program sed
+ assert_has_program awk
+ assert_has_program grep
+ assert_has_program egrep
+ assert_has_program tee
+
+ # test-runner requires a password-less connection
+ # Eg: Inside fuego's docker container do
+ # su jenkins
+ # cp path/to/bbb_id_rsa ~/.ssh/
+ # vi ~/.ssh/config
+ # Host 192.167.1.99 <- replace with your boards ip address ($IPADDR)
+ # IdentityFile ~/.ssh/bbb_id_rsa
+ assert_define SSH_KEY "Please setup SSH_KEY on your board file (fuego-ro/boards/$NODE_NAME.board)"
+}
+
+function test_build {
+ source ./automated/bin/setenv.sh
+ pip install -r $REPO_PATH/automated/utils/requirements.txt --user
+}
+
+function test_run {
+ source $WORKSPACE/$JOB_BUILD_DIR/automated/bin/setenv.sh
+
+ yaml_file=${FUNCTIONAL_LINARO_YAML:-"automated/linux/smoke/smoke.yaml"}
+ if [ ! -e "${REPO_PATH}/$yaml_file" ]; then
+ abort_job "$yaml_file not found"
+ fi
+
+ if startswith "$yaml_file" "plans"; then
+ echo "using test plan: $yaml_file"
+ test_or_plan_flag="-p"
+ else
+ echo "using test definition: $yaml_file"
+ test_or_plan_flag="-d"
+ fi
+
+ if [ -n "$FUNCTIONAL_LINARO_PARAMS" ]; then
+ PARAMS="-r $FUNCTIONAL_LINARO_PARAMS"
+ else
+ PARAMS=""
+ fi
+
+ # FIXTHIS: don't use -s for targets with debian/centos
+ test-runner -o ${LOGDIR} $test_or_plan_flag ${REPO_PATH}/$yaml_file $PARAMS -g $LOGIN@$IPADDR -s -e
+}
+
+# FIXTHIS: the log directory is populated with a copy of the whole repository, clean unnecessary files
diff --git a/tests/Functional.linaro/parser.py b/tests/Functional.linaro/parser.py
new file mode 100755
index 0000000..48b502b
--- /dev/null
+++ b/tests/Functional.linaro/parser.py
@@ -0,0 +1,25 @@
+#!/usr/bin/python
+
+import os, sys, collections
+import common as plib
+import json
+
+# allocate variable to store the results
+measurements = {}
+measurements = collections.OrderedDict()
+
+# read results from linaro result.json format
+with open(plib.LOGDIR + "/result.json") as f:
+ data = json.load(f)[0]
+
+for test_case in data['metrics']:
+ test_case_id = test_case['test_case_id']
+ result = test_case['result']
+ # FIXTHIS: add measurements when available
+ # measurement = test_case['measurement']
+ # units = test_case['units']
+ measurements['default.' + test_case_id] = result.upper()
+
+# FIXTHIS: think about how to get each test's log from stdout.log
+
+sys.exit(plib.process(measurements))
diff --git a/tests/Functional.linaro/spec.json b/tests/Functional.linaro/spec.json
new file mode 100644
index 0000000..561e2ab
--- /dev/null
+++ b/tests/Functional.linaro/spec.json
@@ -0,0 +1,16 @@
+{
+ "testName": "Functional.linaro",
+ "specs": {
+ "default": {
+ "yaml": "automated/linux/smoke/smoke.yaml",
+ "extra_success_links": {"csv": "result.csv"},
+ "extra_fail_links": {"csv": "results.csv"}
+ },
+ "smoke": {
+ "yaml": "automated/linux/smoke/smoke.yaml",
+ "params": "TESTS='pwd'",
+ "extra_success_links": {"csv": "result.csv"},
+ "extra_fail_links": {"csv": "results.csv"}
+ }
+ }
+}
diff --git a/tests/Functional.linaro/test.yaml b/tests/Functional.linaro/test.yaml
new file mode 100644
index 0000000..a2efee8
--- /dev/null
+++ b/tests/Functional.linaro/test.yaml
@@ -0,0 +1,27 @@
+fuego_package_version: 1
+name: Functional.linaro
+description: |
+ Linaro test-definitions
+license: GPL-2.0
+author: Milosz Wasilewski, Chase Qi
+maintainer: Daniel Sangorrin <daniel.sangorrin at toshiba.co.jp>
+version: latest git commits
+fuego_release: 1
+type: Functional
+tags: ['kernel', 'linaro']
+git_src: https://github.com/Linaro/test-definitions
+params:
+ - YAML:
+ description: test definiton or plan.
+ example: "automated/linux/smoke/smoke.yaml"
+ optional: no
+ - PARAMS:
+ description: List of params for the test PARAM1=VALUE1 [PARAM2=VALUE2]
+ example: "TESTS='pwd'"
+ optional: yes
+data_files:
+ - chart_config.json
+ - fuego_test.sh
+ - parser.py
+ - spec.json
+ - test.yaml
--
2.7.4
More information about the Fuego
mailing list