[Fuego] testplans and job inter-dependency

Daniel Sangorrin daniel.sangorrin at toshiba.co.jp
Tue Nov 13 07:01:26 UTC 2018

Hi Tim, 

Thanks for your feedback!! See my comments below.

> -----Original Message-----
> From: Tim.Bird at sony.com <Tim.Bird at sony.com>
> > -----Original Message-----
> > From:  Daniel Sangorrin on Friday, November 09, 2018 12:27 AM
> >
> > Hello Tim,
> >
> > I prepared a couple of patches that fix testplans' job
> > execution order, and allow triggering a new job once
> > the current one is finished.
> >
> > [PATCH 1/2] testplans: execute tests in order
> > [PATCH 2/2] add-jobs: add trigger parameter to specify next job
> OK - I looked at the patches, but I'm going to put my questions here.
> I like the idea of being able to specify that a job can invoke another
> job, or be dependent on the previous invocation of another job.
> However, what happens if a job is part of more than one test plan?
> It looks to me like the 'next_job' is a permanent part of a job definition.
> So if Functional.hello_world is created as part of testplan_default,
> I can't also use it as part of testplan_smoketest, or launch it standalone.
> It seems like launching the job standalone would invoke the entire
> sequence of next_jobs.

That's something I hadn't thought about!

One possible, although dirty, way to avoid such "collisions" would be to
add the testplan name to the job names.
For example: mytestplan.bbb.100M.Benchmark.Dhrystone.

Having said that, it wouldn't be as powerful as what you describe below.

> Also, this approach of using Jenkins is declarative in nature, which means
> there it's difficult to do the dependencies conditionally, or to do complex
> sequences where more than one job is launched (maybe in parallel)
> when another job completes.
> I haven't looked in detail at Jenkins pipelines, but my limited understanding
> is that they essentially allow you to specify the sequence of operations (ie tests to
> run)
> procedurally in a high-level language of your choice.

The procedural pipelines that you mention are probably the ones called "scripted style" which can be
written in groovy (maybe in other languages too, i don't know).

I think that Jenkins has now declarative pipelines where you can also do conditional execution (when, post, etc..):

Unfortunately, we can't rely on them if we want to stay independent of Jenkins in the future.

> I think I would prefer to see testplans in Fuego converted to this type of
> system.
> What if we composed test sequences using a Fuego test, with arbitrary
> code in the test_run() function of fuegotest.sh
> Here's an example:
> make a directory:
> fuego-core/engine/tests/Testplan.smoketest
> with file: fuego_test.sh
> that contained:
> function test_run {
>    ftc run-test -b ${NODE_NAME} -s default -t Benchmark.Dhrystone || true
>    ftc run-test -b ${NODE_NAME} -s default -t Benchmark.dbench4 || true
>    ftc run-test -b ${NODE_NAME} -s default -t Benchmark.hackbench || true
>    ...
>    ftc run-test --timeout 15m -b ${NODE_NAME} -s default -t Benchmark.OpenSSL
> || true
>    ...
>    ftc run-test -b ${NODE_NAME} -s default -t Functional.hello_world || true
> }
> Then we modify ftc and the core scripts to support Testplan as a legal kind
> of test (along with "Benchmark" and "Functional") - which is just for composing
> other tests.
> I have been thinking of adding an "Action" kind of test, that would be used for other
> "lab" operations like deployment of the software under test, or setup of lab
> equipment
> testing, or generating reports, or generating the binary cache of test program
> packages.
> There are issues with using "ftc build-job" instead of "ftc run-test", since I don't think
> Jenkins will executed nested tests (due to Jenkins starting only 1 test per executor,
> and Fuego defining only one executor per node (or board).
> This strategy allows us to run plans like we run tests, with something like:
> ftc run-test -b minnowboard -t Testplan.smoketest
> (Actually, calling them "Testplan" is not really required, but I don't want to call
> them either "Functional" or "Benchmark", and I might use the "Testplan" prefix
> to avoid invoking other test phases.)
> Note that since the test invocations are procedural, you can do them conditionally
> on the outcome of a previous test, or start them in parallel, or decide whether
> to stop when a particular one fails.

It sounds like a really powerful and flexible method. I like it because it is easier to understand than a declarative abstraction layer.
Should we still keep the current testplan files as a way to add jobs to jenkins in a declarative style?
By the way, are we going to keep relying on Jenkins for board access serialization? I thought you were going to create a board/resource reservation system.

> > I put a mention on the commit logs but I wanted to discuss a couple
> > of questions:
> > - currently in my patches, the next job is triggered no matter
> >   if the current job fails or succeeds. That behavior can be
> >   easily changed. I want to know if you prefer to have a
> >   new interface to switch the behavior, or just hardwire it to
> >   a predefined one (e.g. "always continue" or "stop running jobs
> >   if one fails").
> I would want the behavior to match exactly what we have now,
> for the first iteration of the feature.
> >   I can see cases where you want to stop (eg: if the kernel
> >   build fails, then the next tests are worthless). On the other
> >   hand, i can also see cases where you want to go on (e.g.
> >   when the jobs in the testplan do not really depend on each
> >   other).
> >   If you like the "behavior switch" interface better, then
> >   I could add a parameter (parallel=True?) to each test in a testplan
> >   to define that behavior and have a default behavior when
> >   omitted. Any preferences for the default behavior?
> > - the second question is related to the interface used when
> >   adding jobs without a testplan. The current interface
> >   (--trigger myjob) only allows you to define one job to
> >   trigger, but i can add more in the future. The question
> >   is: if i do --trigger job1,job2,job3, should all those
> >   jobs be triggered in order (one after the other) or not.
> >   If we want a "behavior switch" here, the interface could
> >   become a bit convoluted. For example:
> >    --trigger job1->job2->job3,job4,job5->job6
> >   This could be used to define jobs that can run in parallel,
> >   and jobs that must run in a specific order.
> >
> > Your feedback is very welcomed.
> Let me know what you think of my counter-idea to convert testplans
> into procedurally defined lists of tests to execute, inside a test structure
> like the other tests (away from the declarative syntax we have now).

The idea is very powerful, I really like it.
My only questions are:
- Should we completely remove the testplan files? or keep them around as an easy way of adding a set of jobs to Jenkins.
- If we do keep them, how about adding the testplan's name to the job name to avoid collisions between jobs from different testplans?


More information about the Fuego mailing list