[Fuego] testplans and job inter-dependency

[Tim.Bird at sony.com]

> -----Original Message-----
> From:  Daniel Sangorrin on Friday, November 09, 2018 12:27 AM
> 
> Hello Tim,
> 
> I prepared a couple of patches that fix testplans' job
> execution order, and allow triggering a new job once
> the current one is finished.
> 
> [PATCH 1/2] testplans: execute tests in order
> [PATCH 2/2] add-jobs: add trigger parameter to specify next job

OK - I looked at the patches, but I'm going to put my questions here.

I like the idea of being able to specify that a job can invoke another
job, or be dependent on the previous invocation of another job.

However, what happens if a job is part of more than one test plan?
It looks to me like the 'next_job' is a permanent part of a job definition.
So if Functional.hello_world is created as part of testplan_default,
I can't also use it as part of testplan_smoketest, or launch it standalone.
It seems like launching the job standalone would invoke the entire
sequence of next_jobs.

Also, this approach of using Jenkins is declarative in nature, which means
there it's difficult to do the dependencies conditionally, or to do complex
sequences where more than one job is launched (maybe in parallel)
when another job completes.

I haven't looked in detail at Jenkins pipelines, but my limited understanding
is that they essentially allow you to specify the sequence of operations (ie tests to run) 
procedurally in a high-level language of your choice.

I think I would prefer to see testplans in Fuego converted to this type of
system.

What if we composed test sequences using a Fuego test, with arbitrary
code in the test_run() function of fuegotest.sh

Here's an example:
make a directory:
fuego-core/engine/tests/Testplan.smoketest
with file: fuego_test.sh
that contained:
function test_run {
   ftc run-test -b ${NODE_NAME} -s default -t Benchmark.Dhrystone || true
   ftc run-test -b ${NODE_NAME} -s default -t Benchmark.dbench4 || true
   ftc run-test -b ${NODE_NAME} -s default -t Benchmark.hackbench || true
   ...
   ftc run-test --timeout 15m -b ${NODE_NAME} -s default -t Benchmark.OpenSSL || true
   ...
   ftc run-test -b ${NODE_NAME} -s default -t Functional.hello_world || true
}

Then we modify ftc and the core scripts to support Testplan as a legal kind
of test (along with "Benchmark" and "Functional") - which is just for composing other tests.

I have been thinking of adding an "Action" kind of test, that would be used for other
"lab" operations like deployment of the software under test, or setup of lab equipment
testing, or generating reports, or generating the binary cache of test program packages.

There are issues with using "ftc build-job" instead of "ftc run-test", since I don't think
Jenkins will executed nested tests (due to Jenkins starting only 1 test per executor,
and Fuego defining only one executor per node (or board).

This strategy allows us to run plans like we run tests, with something like:
ftc run-test -b minnowboard -t Testplan.smoketest

(Actually, calling them "Testplan" is not really required, but I don't want to call
them either "Functional" or "Benchmark", and I might use the "Testplan" prefix
to avoid invoking other test phases.)

Note that since the test invocations are procedural, you can do them conditionally
on the outcome of a previous test, or start them in parallel, or decide whether
to stop when a particular one fails.

> I put a mention on the commit logs but I wanted to discuss a couple
> of questions:
> - currently in my patches, the next job is triggered no matter
>   if the current job fails or succeeds. That behavior can be
>   easily changed. I want to know if you prefer to have a
>   new interface to switch the behavior, or just hardwire it to
>   a predefined one (e.g. "always continue" or "stop running jobs
>   if one fails").
I would want the behavior to match exactly what we have now,
for the first iteration of the feature.

>   I can see cases where you want to stop (eg: if the kernel
>   build fails, then the next tests are worthless). On the other
>   hand, i can also see cases where you want to go on (e.g.
>   when the jobs in the testplan do not really depend on each
>   other).
>   If you like the "behavior switch" interface better, then
>   I could add a parameter (parallel=True?) to each test in a testplan
>   to define that behavior and have a default behavior when
>   omitted. Any preferences for the default behavior?
> - the second question is related to the interface used when
>   adding jobs without a testplan. The current interface
>   (--trigger myjob) only allows you to define one job to
>   trigger, but i can add more in the future. The question
>   is: if i do --trigger job1,job2,job3, should all those
>   jobs be triggered in order (one after the other) or not.
>   If we want a "behavior switch" here, the interface could
>   become a bit convoluted. For example:
>    --trigger job1->job2->job3,job4,job5->job6
>   This could be used to define jobs that can run in parallel,
>   and jobs that must run in a specific order.
> 
> Your feedback is very welcomed.

Let me know what you think of my counter-idea to convert testplans
into procedurally defined lists of tests to execute, inside a test structure
like the other tests (away from the declarative syntax we have now).

  -- Tim