[Fuego] testplans and job inter-dependency

[Tim.Bird at sony.com]

> -----Original Message-----
> From: Daniel Sangorrin 
> 
> Hi Tim,
> 
> Thanks for your feedback!! See my comments below.
> 
> > -----Original Message-----
> > From: Tim.Bird at sony.com <Tim.Bird at sony.com>
> > > -----Original Message-----
> > > From:  Daniel Sangorrin on Friday, November 09, 2018 12:27 AM
> > >
> > > Hello Tim,
> > >
> > > I prepared a couple of patches that fix testplans' job
> > > execution order, and allow triggering a new job once
> > > the current one is finished.
> > >
> > > [PATCH 1/2] testplans: execute tests in order
> > > [PATCH 2/2] add-jobs: add trigger parameter to specify next job
> >
> > OK - I looked at the patches, but I'm going to put my questions here.
> >
> > I like the idea of being able to specify that a job can invoke another
> > job, or be dependent on the previous invocation of another job.
> >
> > However, what happens if a job is part of more than one test plan?
> > It looks to me like the 'next_job' is a permanent part of a job definition.
> > So if Functional.hello_world is created as part of testplan_default,
> > I can't also use it as part of testplan_smoketest, or launch it standalone.
> > It seems like launching the job standalone would invoke the entire
> > sequence of next_jobs.
> 
> That's something I hadn't thought about!
> 
> One possible, although dirty, way to avoid such "collisions" would be to
> add the testplan name to the job names.
> For example: mytestplan.bbb.100M.Benchmark.Dhrystone.
The names are already long.  I'd like to come up with a different
mechanism for this if possible.
> 
> Having said that, it wouldn't be as powerful as what you describe below.
My way is powerful, but it does have tradeoffs.

> 
> > Also, this approach of using Jenkins is declarative in nature, which means
> > there it's difficult to do the dependencies conditionally, or to do complex
> > sequences where more than one job is launched (maybe in parallel)
> > when another job completes.
> >
> > I haven't looked in detail at Jenkins pipelines, but my limited understanding
> > is that they essentially allow you to specify the sequence of operations (ie
> tests to
> > run)
> > procedurally in a high-level language of your choice.
> 
> The procedural pipelines that you mention are probably the ones called
> "scripted style" which can be
> written in groovy (maybe in other languages too, i don't know).
> https://medium.com/@Lenkovits/jenkins-pipelines-and-their-dirty-secrets-
> 1-9e535cd603f4
> 
> I think that Jenkins has now declarative pipelines where you can also do
> conditional execution (when, post, etc..):
> https://jenkins.io/blog/2018/04/09/whats-in-declarative/
> https://jenkins.io/doc/pipeline/tour/running-multiple-steps/#finishing-up

Ugh.  My head is spinning with all their declarative syntax.  I think these
types of things are easy to read but nearly impossible to write without
constantly referring to a manual.  And sometimes the manual is very
hard to find or non-existent.  Well, maybe the same would be true
of a scripted, or procedural style. Either one requires knowledge that
is very specific to the test framework.

In terms of automated operations on the testplans themselves (aside from
running them), declarative style has the benefit that it's regular enough
for certain operations like automatic scanning.  This can be useful
for things like customizing the testplans.

I stumbled across this today:
https://git.linaro.org/qa/test-definitions.git/tree/plans/test-plan-overlay-example.yaml

I have always thought the best way to customize a testplan is just to copy it and
edit the parts you need to change.  But apparently LAVA allows you to
separate your modifications into a separate file.  I guess which approach would be better
would depend on how often the 'base' testplans were modified.  I don't
think they are modified that often, so I'm not sure it's worth the sacrifice
in writability and expressiveness to go with declarative style.

Note that LAVA's declarative style is much easier to read than Jenkins', IMHO.
(Maybe because they have less features.)
See https://git.linaro.org/qa/test-definitions.git/tree/plans/qcomlt/smoke.yaml
for a relatively simple example.

> Unfortunately, we can't rely on them if we want to stay independent of
> Jenkins in the future.

Indeed, my preference would be to do something outside of Jenkins.

> 
> > I think I would prefer to see testplans in Fuego converted to this type of
> > system.
> >
> > What if we composed test sequences using a Fuego test, with arbitrary
> > code in the test_run() function of fuegotest.sh
> >
> > Here's an example:
> > make a directory:
> > fuego-core/engine/tests/Testplan.smoketest
> > with file: fuego_test.sh
> > that contained:
> > function test_run {
> >    ftc run-test -b ${NODE_NAME} -s default -t Benchmark.Dhrystone || true
> >    ftc run-test -b ${NODE_NAME} -s default -t Benchmark.dbench4 || true
> >    ftc run-test -b ${NODE_NAME} -s default -t Benchmark.hackbench || true
> >    ...
> >    ftc run-test --timeout 15m -b ${NODE_NAME} -s default -t
> Benchmark.OpenSSL
> > || true
> >    ...
> >    ftc run-test -b ${NODE_NAME} -s default -t Functional.hello_world ||
> true
> > }
> >
> > Then we modify ftc and the core scripts to support Testplan as a legal kind
> > of test (along with "Benchmark" and "Functional") - which is just for
> composing
> > other tests.
> >
> > I have been thinking of adding an "Action" kind of test, that would be used
> for other
> > "lab" operations like deployment of the software under test, or setup of
> lab
> > equipment
> > testing, or generating reports, or generating the binary cache of test
> program
> > packages.
> >
> > There are issues with using "ftc build-job" instead of "ftc run-test", since I
> don't think
> > Jenkins will executed nested tests (due to Jenkins starting only 1 test per
> executor,
> > and Fuego defining only one executor per node (or board).
> >
> > This strategy allows us to run plans like we run tests, with something like:
> > ftc run-test -b minnowboard -t Testplan.smoketest
> >
> > (Actually, calling them "Testplan" is not really required, but I don't want to
> call
> > them either "Functional" or "Benchmark", and I might use the "Testplan"
> prefix
> > to avoid invoking other test phases.)
> >
> > Note that since the test invocations are procedural, you can do them
> conditionally
> > on the outcome of a previous test, or start them in parallel, or decide
> whether
> > to stop when a particular one fails.
> 
> It sounds like a really powerful and flexible method. I like it because it is
> easier to understand than a declarative abstraction layer.
> Should we still keep the current testplan files as a way to add jobs to jenkins
> in a declarative style?
Yes.  I try not to get rid of anything that people might be using.
We might deprecate it, but I don't see a need to remove it in the short term.

> By the way, are we going to keep relying on Jenkins for board access
> serialization? I thought you were going to create a board/resource
> reservation system.
Well, I have a way to reserve a board and release a reservation.

See 'ftc reserve-resource' and 'ftc release-resource'

A bit more is needed before it's fully functional for board serialization.
I need to add functionality to wait for a board to be released (essentially
a wait queue for a board).  We're creeping up on a "test scheduler"
feature, but I definitely don't want it to get more complicated before
the 1.4 release.  I don't want to introduce full board serialization
at the ftc layer before 1.4, because I suspect
there will be issues with not releasing the lock on test aborts initiated
by Jenkins.  Basically I want to introduce it when I have time to do some
testing.

> 
> > > I put a mention on the commit logs but I wanted to discuss a couple
> > > of questions:
> > > - currently in my patches, the next job is triggered no matter
> > >   if the current job fails or succeeds. That behavior can be
> > >   easily changed. I want to know if you prefer to have a
> > >   new interface to switch the behavior, or just hardwire it to
> > >   a predefined one (e.g. "always continue" or "stop running jobs
> > >   if one fails").
> > I would want the behavior to match exactly what we have now,
> > for the first iteration of the feature.
> >
> > >   I can see cases where you want to stop (eg: if the kernel
> > >   build fails, then the next tests are worthless). On the other
> > >   hand, i can also see cases where you want to go on (e.g.
> > >   when the jobs in the testplan do not really depend on each
> > >   other).
> > >   If you like the "behavior switch" interface better, then
> > >   I could add a parameter (parallel=True?) to each test in a testplan
> > >   to define that behavior and have a default behavior when
> > >   omitted. Any preferences for the default behavior?
> > > - the second question is related to the interface used when
> > >   adding jobs without a testplan. The current interface
> > >   (--trigger myjob) only allows you to define one job to
> > >   trigger, but i can add more in the future. The question
> > >   is: if i do --trigger job1,job2,job3, should all those
> > >   jobs be triggered in order (one after the other) or not.
> > >   If we want a "behavior switch" here, the interface could
> > >   become a bit convoluted. For example:
> > >    --trigger job1->job2->job3,job4,job5->job6
> > >   This could be used to define jobs that can run in parallel,
> > >   and jobs that must run in a specific order.
> > >
> > > Your feedback is very welcomed.
> >
> > Let me know what you think of my counter-idea to convert testplans
> > into procedurally defined lists of tests to execute, inside a test structure
> > like the other tests (away from the declarative syntax we have now).
> 
> The idea is very powerful, I really like it.
> My only questions are:
> - Should we completely remove the testplan files? or keep them around as
> an easy way of adding a set of jobs to Jenkins.
Keep them.

> - If we do keep them, how about adding the testplan's name to the job name
> to avoid collisions between jobs from different testplans?
I'd rather not.

I don't want to put off this feature (serializing the testplan jobs using a different
mechanism than we've got today).  But I'm not sure how to proceed with the least
impact.  Your changes were pretty small, since they leveraged a Jenkins feature.
To build up an alternative in Fuego core will take more time and thought.

Alternative ideas?  Either fuego core or Jenkins has to launch subsequent jobs.
If it's Jenkins, then we'll (obviously) have to use a Jenkins feature. If it's fuego core,
we'll likely have to implement some new code, while trying to leverage as much
existing functionality as we can.

On a related topic, I want to add a feature to track the batch-id for a run.  That is,
when tests are executed as part of the same batch, I want them to have all to have
the same batch-id in their run.json file.  This is so that we can query the results and
generate reports with this batch-id.  With the "test plan is a fuego test" design
I think this is easy:
function test_run {
    export FUEGO_BATCH_ID=$(get_next_batch_id)
    ...
}
and a couple of changes in ftc to save this in the run.json file.
  -- Tim