[Fuego] testplans and job inter-dependency

Tim.Bird at sony.com Tim.Bird at sony.com
Wed Nov 14 07:22:38 UTC 2018

> -----Original Message-----
> From: Daniel Sangorrin 
> > -----Original Message-----
> > From: Tim.Bird at sony.com <Tim.Bird at sony.com>
> [...]
> > > Having said that, it wouldn't be as powerful as what you describe below.
> > My way is powerful, but it does have tradeoffs.
> One of them is that you are not using jobs, and therefore our current
> visualization tools based on Jenkins will not work.
This is true.

However, if a Jenkins job exists, even if you do 'ftc run-test' outside of Jenkins,
ftc attempts to populate the Jenkins directories with suitable Jenkins
build files.  I haven't figured out how to get Jenkins to re-read the build directories
and update the build status for jobs without manually telling it to reload its
configuration from disk, but this seems to work OK.

I have done the following:
* create a job for a board-test combination
* build the job in Jenkins
* "ftc run-test" that board-test combination from the command line
* tell Jenkins to reload configuration from disk
   * or, inside the container do: service jenkins stop; service jenkins start
And the data for the outside job is in the visualization.

There's a fixthis in ftc to figure out how to have ftc cause the reload.

Also, there are still some thorny issues with build numbers when you do 'ftc run-test'
outside of Jenkins before creating a job for a board/test combination.  ftc has already
used some build numbers in that case, and it confuses Jenkins, which wants to start
the job with build number 1.  Ideally we would create the correct set of Jenkins
build files at the time of job creation, but that's probably difficult.

> > Note that LAVA's declarative style is much easier to read than Jenkins',
> > (Maybe because they have less features.)
> > See https://git.linaro.org/qa/test-
> definitions.git/tree/plans/qcomlt/smoke.yaml
> > for a relatively simple example.
> LAVA uses an elegant yaml abstraction.
> However, I think Fuego wants to focus on usability. Even Fuego beginners
> should be able to easily modify an existing testplan without having to read
> through a lot of documentation.

> > Well, I have a way to reserve a board and release a reservation.
> >
> > See 'ftc reserve-resource' and 'ftc release-resource'
> Cool. Shall we use that on the testplans? (e.g.: when executing 2 testplans on
> the same board at the same time, each testplan must first reserve the board)
Not until after 1.4.

> >
> > A bit more is needed before it's fully functional for board serialization.
> > I need to add functionality to wait for a board to be released (essentially
> > a wait queue for a board).  We're creeping up on a "test scheduler"
> > feature, but I definitely don't want it to get more complicated before
> > the 1.4 release.  I don't want to introduce full board serialization
> > at the ftc layer before 1.4, because I suspect
> > there will be issues with not releasing the lock on test aborts initiated
> > by Jenkins.  Basically I want to introduce it when I have time to do some
> > testing.
> OK, then I guess the testplans functionality should also come after 1.4.

Unfortunately, yes.
> The thing is that we still do not have a better visualization tool than Jenkins.
> For that reason, using my approach has the benefit of reusing our current
> visualization infrastructure.
> #When I have time i will try to port Squad support back into Fuego to solve
> that.
> > On a related topic, I want to add a feature to track the batch-id for a run.
> That is,
> > when tests are executed as part of the same batch, I want them to have all
> to have
> > the same batch-id in their run.json file.  This is so that we can query the
> results and
> > generate reports with this batch-id.  With the "test plan is a fuego test"
> design
> > I think this is easy:
> > function test_run {
> >     export FUEGO_BATCH_ID=$(get_next_batch_id)
> >     ...
> > }
> > and a couple of changes in ftc to save this in the run.json file.
> I will explore the posibility of exporting FUEGO_BATCH_ID between the
> trigger job and the triggered job.
> If that was possible, we could solve the "collisions" problem something like
> this:
> if $FUEGO_BATCH_ID == "testplan_lts"
>    start the next job in the testplan lts
> if ...

Interesting.  I was thinking the batch-id would be something unique, like a timestamp.
But if it included the testplan name, then we could have ftc trigger the next job.

We could experiment with 'ftc build-job' in something like 'Functional.plan_lts/fuego_test.sh,
without changing anything in fuego-core. (We would still have to overcome the issue
of nested use of a Jenkins node).

What's easier to fix - causing reload of Jenkins build data, or working around the concurrency
lock on a Jenkins node?
 -- Tim

More information about the Fuego mailing list