[Fuego] Parameter passing to tests

Bird, Timothy Tim.Bird at sony.com
Sat Jun 3 03:08:22 UTC 2017

> -----Original Message-----
> From: Daniel Sangorrin on Friday, June 02, 2017 9:49 AM
> > Let's say that I have a device with three network interfaces and I want to
> run the iperf (Benchmark.iperf) test on all three of them on
> > different servers. How would I pass the parameters?

IMHO, given the current Fuego features, I would specify
the network interfaces should be declared in the board file,
and the servers in the spec file.  There's an issue here with scheduling
the servers, that is outside the scope of Fuego.

> Thanks for rising this problem. The short answer is that right now you would
> have create 3 specs (e.g. eth0spec, eth1spec, eth2spec)
> with different interface parameters.

I think the Ethernet interface parameters would best be described in the
board file as they are a description of the hardware on that board.
But the references to the iperf servers should be in the spec (at least in
current Fuego).  Maybe this is what you mean by the above.

So the board file would have:

and the spec would have:
spec: {

Better yet would be to have the test automatically select 
an iperf server.  This might require some new capability
in Fuego to schedule secondary resources to avoid conflicts,
and provide them to tests.  So maybe the test calls something like:
   RESOURCE_ID=$(wait_for_avail_resource IPERF_SERVER)
that returns the next iperf server when it is available ("locking"
it for this test, when it becomes available), and later:
   release_resource RESOURCE_ID
to free it up for other tests.

> > As it seems to me now (and correct me if I'm wrong) the only way to pass
> parameters is either on the board file or on the spec.
> >
> > If there are lots of tests in fuego on the future, placing test parameters on
> the board file doesn't scale. IMO the board file should
> > contain only data related and to the board <-> fuego transport and maybe
> some hardware related staff. Test-specific variables like
> > "LTP_OPEN_POSIX_SUBTEST_COUNT_POS" seem a bit out of place there
> to me.
> Agree.
> > If one does place parameters on the .spec files, then it's harder to
> contribute tests, as everyone will have dirty specs files with its own
> > configuration.
> Yeah, I understand. I have the same problem.
> Ideally we want to share specs that add value (e.g.: parameter sets that test
> different parts), but avoid sharing the
> "dirty" specs where you only changed something like the ip address.
> >It's also a bit rigid when developing: if you want to temporarily change a
> parameter you have to modify, submit a new
> > project to jenkins, test, remove a project from jenkins and undo the
> change.

I'm not sure I understand this.  You can change the value of a spec variable
and re-run a test (in 1.2) without having to remove the Jenkins job and re-create it.

> > Jenkins projects have build parameters (parameterized build) which can be
> modified when building either through the web or
> > through the CLI and are accessible from the scripts as environment
> variables. IMO a .spec file could only be a key-value list of Jenkins
> > parameters. 
> >
> > Doing this way places all the test (user) configuration outside of the fuego
> repository (fuego-core) on the user automated build scripts.
> > It removes the need to have multiple projects of the same test with
> different specifications in jenkins. There is just a test project for
> > each board.
> At the moment, what you can do is to configure your job:
>  Jenkins > Testname > configure
>      export PARAM1="hello param 1"
>      export PARAM2="hello param 2"
> That should achieve the same result as using the parameterized build plugin.
> But it would be more
> convenient to add an option to ftc:
> ftc add-jobs -p PARAM1="..."

I think maybe what Rafael wants is something more like:
ftc run-test -p PARAM1="..."
ftc build-job -p PARAM1="..."

These would both be very easy to implement.

> > So with this mail I just want to get confirmation that things work as I think
> they do and to discuss the implications and viability of such
> > change.
> Yes, you are correct. This is a problem that I wanted to solve.
> Configuring your job with new params is OK for debugging but in the long run
> you may also want to save those values into your board's testplan.
> My proposal is as follows:
>   - testplans should generally be created by the user. We provide some
> testplans for reference only. In other words, a testplan is where you put your
> "dirty" changes.
That's an interesting place to put these kinds of items.  I think that's probably
valuable.  If we're going to allow passing variable parameters at test run time,
then it would be nice to be able to save them somewhere.

>   - The idea is that you could add 3 jobs to your testplan using the same iperf
> spec, and then _OVERWRITE_ those variables that you want to change (e.g.
> IP address)
>   - testplans should be moved to /fuego/fuego-ro/boards.

I'd rather not put testplans in a board directory.

> To achieve this we would need to update the testplans path throughout
> Fuego and then modify the overlay generator so that
> testplan definitions overwrite those in the spec when prolog.sh is generated.

This would not be hard.  I think it could be useful, but I don't
know if it addresses Rafael's problem.
 -- Tim

More information about the Fuego mailing list