[Fuego] [PATCH] Add test cases for command zabbix.

[Tim.Bird at sony.com]

> -----Original Message-----
> From: Wang Mingyu

The commit message body should rarely be empty.  Please put a synopsis
of test features here.

Also, a test.yaml file describing what 'zabbix' is would be nice.
I'm not familiar with it.

> 
> Signed-off-by: Wang Mingyu <wangmy at cn.fujitsu.com>
> ---
>  engine/tests/Functional.zabbix/fuego_test.sh       | 18
> ++++++++++++++++++
>  engine/tests/Functional.zabbix/parser.py           | 22
> ++++++++++++++++++++++
>  engine/tests/Functional.zabbix/spec.json           |  7 +++++++
>  .../tests/Functional.zabbix/tests/zabbix_help.sh   | 13 +++++++++++++
>  .../tests/Functional.zabbix/tests/zabbix_print.sh  | 13 +++++++++++++
>  engine/tests/Functional.zabbix/zabbix_test.sh      |  4 ++++
>  6 files changed, 77 insertions(+)
>  create mode 100644 engine/tests/Functional.zabbix/fuego_test.sh
>  create mode 100644 engine/tests/Functional.zabbix/parser.py
>  create mode 100644 engine/tests/Functional.zabbix/spec.json
>  create mode 100644 engine/tests/Functional.zabbix/tests/zabbix_help.sh
>  create mode 100644 engine/tests/Functional.zabbix/tests/zabbix_print.sh
>  create mode 100755 engine/tests/Functional.zabbix/zabbix_test.sh
> 
> diff --git a/engine/tests/Functional.zabbix/fuego_test.sh
> b/engine/tests/Functional.zabbix/fuego_test.sh
> new file mode 100644
> index 0000000..0a142bf
> --- /dev/null
> +++ b/engine/tests/Functional.zabbix/fuego_test.sh
> @@ -0,0 +1,18 @@
> +function test_pre_check {
> +    is_on_target_path zabbix_agentd PROGRAM_ZABBIX_AGENTD
> +    assert_define PROGRAM_ZABBIX_AGENTD "Missing 'zabbix_agentd'
> program on target board"
> +}
> +
> +function test_deploy {
> +    put $TEST_HOME/zabbix_test.sh $BOARD_TESTDIR/fuego.$TESTDIR/
> +    put -r $TEST_HOME/tests $BOARD_TESTDIR/fuego.$TESTDIR/
> +}
> +
> +function test_run {
> +    report "cd $BOARD_TESTDIR/fuego.$TESTDIR;\
> +    ./zabbix_test.sh"
> +}
> +
> +function test_processing {
> +    log_compare "$TESTDIR" "0" "TEST-FAIL" "n"
> +}
> diff --git a/engine/tests/Functional.zabbix/parser.py
> b/engine/tests/Functional.zabbix/parser.py
> new file mode 100644
> index 0000000..d85abd7
> --- /dev/null
> +++ b/engine/tests/Functional.zabbix/parser.py
> @@ -0,0 +1,22 @@
> +#!/usr/bin/python
> +# See common.py for description of command-line arguments
> +
> +import os, sys, collections
> +
> +sys.path.insert(0, os.environ['FUEGO_CORE'] + '/engine/scripts/parser')
> +import common as plib
> +
> +measurements = {}
> +measurements = collections.OrderedDict()
> +
> +regex_string = '^ -> (.*): TEST-(.*)$'
> +matches = plib.parse_log(regex_string)
> +
> +if matches:
> +    for m in matches:
> +        measurements['default.' + m[0]] = 'PASS' if m[1] == 'PASS' else 'FAIL'
> +
> +# split the output for each testcase
> +plib.split_output_per_testcase(regex_string, measurements)
> +
> +sys.exit(plib.process(measurements))
> diff --git a/engine/tests/Functional.zabbix/spec.json
> b/engine/tests/Functional.zabbix/spec.json
> new file mode 100644
> index 0000000..50ab91d
> --- /dev/null
> +++ b/engine/tests/Functional.zabbix/spec.json
> @@ -0,0 +1,7 @@
> +{
> +    "testName": "Functional.v10pkgs",'
This is incorrect.  Should be "Functional.zabbix".

> +    "specs": {
> +        "default": {}
> +    }
> +}
> +
> diff --git a/engine/tests/Functional.zabbix/tests/zabbix_help.sh
> b/engine/tests/Functional.zabbix/tests/zabbix_help.sh
> new file mode 100644
> index 0000000..6d7bbd0
> --- /dev/null
> +++ b/engine/tests/Functional.zabbix/tests/zabbix_help.sh
> @@ -0,0 +1,13 @@
> +#!/bin/sh
> +
> +#  The testscript checks the command zabbix_agentd.
> +#  option: -h
> +
> +test="help"
> +
> +if zabbix_agentd -h | grep usage
> +then
> +    echo " -> $test: TEST-PASS"
> +else
> +    echo " -> $test: TEST-FAIL"
> +fi;
> diff --git a/engine/tests/Functional.zabbix/tests/zabbix_print.sh
> b/engine/tests/Functional.zabbix/tests/zabbix_print.sh
> new file mode 100644
> index 0000000..98760d4
> --- /dev/null
> +++ b/engine/tests/Functional.zabbix/tests/zabbix_print.sh
> @@ -0,0 +1,13 @@
> +#!/bin/sh
> +
> +#  The testscript checks the command zabbix_agentd.
> +#  option: -p
> +
> +test="print"
> +
> +if zabbix_agentd -p | grep localhost
> +then
> +    echo " -> $test: TEST-PASS"
> +else
> +    echo " -> $test: TEST-FAIL"
> +fi;
> diff --git a/engine/tests/Functional.zabbix/zabbix_test.sh
> b/engine/tests/Functional.zabbix/zabbix_test.sh
> new file mode 100755
> index 0000000..dd5ce37
> --- /dev/null
> +++ b/engine/tests/Functional.zabbix/zabbix_test.sh
> @@ -0,0 +1,4 @@
> +#!/bin/sh
> +for i in tests/*.sh; do
> +    sh $i
> +done
> --
> 1.8.3.1

These tests that only check the usage line and some trivial output
are not useful in the long run.  I guess they detect that the command is
present.  But the likelihood that this test will catch a bug is extremely
low.  Commands don't change their usage very often, and this is extremely
unlikely to catch anything of importance for the operation of the tool.

I've been letting these "stub" tests through, on the theory that it's good to have
a placeholder and the infrastructure for a test of a command, even if
I'm unfamiliar with it, because then it will be easier for someone to add
additional testcases for that command in the future.  However, if no
additional testcases are ever written, these tests are not very useful.
They are actually detrimental, because they provide a false sense that
the command has been tested thoroughly, which is not really true.

Please let me know if Fujitsu plans to enhance these simple tests with
additional testcases over time.

If so, please add at least one more testcase now, which tests some actual
functionality of zabbix_agentd.  And, please fix the other issues listed above.

For now, this is NOT applied.

Regards,
 -- Tim