[Fuego] Fuego's version up and other changes

[Bird, Timothy]

> -----Original Message-----
> From: Jan-Simon Möller on Wednesday, February 01, 2017 2:46 AM
>
> Am Mittwoch, 1. Februar 2017, 09:56:03 schrieb Daniel Sangorrin:
>
> > Another approach would be to create a single
> > job and then have fuego-core take care of reading the tests specs of each
> > test from the testplan, and repeat the test for each test spec. Then a
> > modified parser would display and compare the results of the test across
> > the different test specs.
> > [Note] I think that AGL's fuego has a similar parser.

This seems to be getting into the territory of matrix testing, which is
what Avacado seems pretty good at (they use YAML to specify
the set of parameters for varying in the matrix of tests to run.
I'd like to have the dust settle on basic testing (execute a single
test, or a set of tests on a board) before we get into how to 
effectively handle matrix testing.  I think Jenkins has some stuff
for this as well, which I haven't looked into yet.

In any event, I want to avoid too much complexity in Fuego 
that can be done by another tool.
 
> Having all in one job let's you not execute one specific test easily.
> But if we simplify the shell call that jenkins has to do, creating jobs
> with specific names and parameters/env-variables is trivial.

Agreed.

> E.g.:
>     builders:
>       - shell: |
>           #
>           source $FUEGO_RO/conf/boards/{machine}.sh
>           source $FUEGO_CORE/engine/env/${{DISTRO}}.sh         # DISTRO set in
> machine.sh
>           source $FUEGO_CORE/engine/testplans/{testplan}.sh    # taken from jjb
>           #
>           export Reboot=false
>           export Rebuild=true
>           export Target_Cleanup=true
>           export TESTDIR={testname}
>           export TESTNAME=$(echo "{testname}" | sed -e "s#.*\.##")
>           timeout --signal=9 ${{JOB_TIMEOUT}} /bin/bash
> $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{TESTNAME}}.sh
> 
> In this case, ${{JOB_TIMEOUT}} can be defined in the
> $FUEGO_RO/conf/boards/{machine}.sh  env script.

I'd like to cut down the amount of stuff in the Jenkins job configuration,
while still allowing some parameters to be managed with the Jenkins
interface.  Optimally, we could have something as simple as:
shell: fuego run ${{TESTNAME}}
with TESTDIR implied and other parameters like the JOB_TIMEOUT
being auto-calculated from data in the transport, job file, distribution,
test itself, and board file.

...

> 
> We access the board very early even before the build (pre_test).
> The we build, deploy, run, fetch results, analyse. At this point

Jenkins basically has the board busy during the entire sequence.
But if fuego could run the phases independently (like bitbake
or rpm can do for the phases of their operations), then 
we could have the board busy during less time.

Possibly pretest needs to be split into two parts.  One that
tests the link, and the other that gathers data to save as part
of the test results.

For example, if we had something like this:
 $ fuego --board beaglebone --phase build Functional.LTP
 $ fuego --board beaglebone --phase "pretest,deploy,run,gather" Functional.LTP
 $ fuego --board beaglebone --phase post_test Functional.LTP

I don't know if there's a way to have Jenkins only lock the node
during the middle of those operations.  The last can be put in the
Jenkins post_test.  I don't see any post_test operation in Daniel's
new system.  Did I miss something?  Where does cleanup occur?

> 
> > > - Integration of board up/down could happen in a few ways:
> > > -- Either ppl should just generate a wrapper around the (blocking) batch
> > > job and trigger it>
> > > with their pre/post to their needs. Done.
> > > -- Or we allow hooks like the current "TARGET_SETUP_LINK" and amend
> it
> > > with a matching "TARGET_TEARDOWN_LINK">
> > > - If a model like the "TARGET_SETUP_LINK" is used, we add a delay here
> as
> > > this call must block until the board is up.>
> > > In this case our predifined timeouts are bogus as they track not just the
> > > test run, but all processing.
> > Good point. I wasn't using the reboot feature so I hadn't thought about it.
> > I guess we would need to define a timeout for the board to reboot, and
> then
> > the timeout for each test. The timeout for the board could be called
> > BOARD_REBOOT_TIMEOUT for example and be defined in the asdf.board
> file. And
> > the timeout for each test defined in another variable such as
> TEST_TIMEOUT.
> 
> As I said, there are multiple ways to integrate that.
> a) The wrapper-job triggering the batch leaves fuego out of the picture. But
> we need to block the whole chain to make flow-control possible.
> b) TARGET_SETUP_LINK gives us a _per job_ way of brining up the board.
> Not just for the wrapp'ed batch.
> 
> Most users might get away and be happy with a) . While a per-job control as
> in b) might scale better if you think of the case that you could have
> multiple target boards of the same type (multiple executors!) and could run
> jobs 'in parallel' !
> 
> > BOARD_REBOOT_TIMEOUT=1m
> > TEST_TIMEOUT=15m
> > /bin/bash $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{TESTNAME}}.sh
> >
> > Then, internally the timeouts would be used separately.
> > # we could also split timeouts between execution time and post-processing
> > time
> 
> Yep ... we probably should split the build, test and postproc into separate
> scripts.
> or instead of $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{TESTNAME}}.sh in
> jenkins, we could call ftc as you would do on the terminal.

> 
> Also the above should be in separate jobs ... think of it:
> - the build phase doesn't need to block, can run in parallel,
>   e.g. on master node!
> - the test batch needs to block to allow flow-control and
>   run only on the specified executor. No concurrency but multiple jobs
>   of the *same* batch could run in parallel, given we can handle multiple
>   boards of the same type at the same time (which is true with an
> 'orchestrator' like lava).
> - the batch is only  deploy/run/fetch result  but can trigger postprocessing
> - the postprocessing is another job, can run in parallel, independent of the
> board, e.g. on master

I should have read farther...   this is what I describe above, but
you beat me to it.

I agree.
 -- Tim