[Fuego] Parameter passing to tests
Tim.Bird at sony.com
Sat Jun 3 02:36:12 UTC 2017
OK - I know this is a long thread, and I'm sorry for chiming in late, but
it's been a busy week.
I'm going to answer issues raised in this thread in order, as
otherwise it will get too confusing for me. Hopefully, by the
time I get to the last message on the thread my answers
will make sense and match the context of the questions
or discussion. :-)
> -----Original Message-----
> From: Rafael Gago Castano on Thursday, June 01, 2017 5:51 PM
> Let's say that I have a device with three network interfaces and I want to run
> the iperf (Benchmark.iperf) test on all three of them on different servers.
> How would I pass the parameters?
> As it seems to me now (and correct me if I'm wrong) the only way to pass
> parameters is either on the board file or on the spec.
That's correct. There's an addition to the board file, where 'dynamic
variables' are storable, in the 1.2 release. But those are for a different
purpose than the parameters you're describing here.
Another thing coming soon is a global configuration file. This is intended
to have lab-specific or host-specific information (in the form of variables
passed to every test) , and might solve some of the issues you describe.
I'll get back to this later.
> If there are lots of tests in fuego on the future, placing test parameters on
> the board file doesn't scale. IMO the board file should contain only data
> related and to the board <-> fuego transport and maybe some hardware
> related staff. Test-specific variables like
> "LTP_OPEN_POSIX_SUBTEST_COUNT_POS" seem a bit out of place there to
I agree. I've never liked having that variable in the board file, for a few different
reasons. It's the best place for the moment, but in the future it would be better
to calculate this information from run data for an 'approved' run. Essentially,
this variable (and others like it) are part of an 'expected value' for a test, which
may or may not be directly related to the board (e.g. it could depend on kconfig,
on the spec used, on user parameters passed, etc.)
> If one does place parameters on the .spec files, then it's harder to contribute
> tests, as everyone will have dirty specs files with its own configuration. It's
> also a bit rigid when developing: if you want to temporarily change a
> parameter you have to modify, submit a new project to jenkins, test,
> remove a project from jenkins and undo the change.
Agreed. It is a high priority to share tests and specs, so that's not a good
place for lab-specific expected values, or temporary parameters.
> Jenkins projects have build parameters (parameterized build) which can be
> modified when building either through the web or through the CLI and are
> accessible from the scripts as environment variables. IMO a .spec file could
> only be a key-value list of Jenkins parameters. The Key would be the variable
> name and the Value the default variable value.
> Adding the values can be done on the ftc's "add_job" function when it does
> generate the project XML by adding this:
> <defaultValue>PARAM1 DEFAULT VALUE</defaultValue>
> <defaultValue>PARAM2 DEFAULT VALUE</defaultValue>
> Then PARAM1 and PARAM2 are made environment variables and are visible
> from the fuego test scripts. They can be parameterized for each build in an
> automated way:
> java -jar /opt/jenkins/jenkins-cli.jar -s http://localhost:8080/fuego
> <testname> -p PARAM1="hello param 1" -p PARAM2="hello param 2"
> Doing this way places all the test (user) configuration outside of the fuego
> repository (fuego-core) on the user automated build scripts. It removes the
> need to have multiple projects of the same test with different specifications
> in jenkins. There is just a test project for each board.
> So with this mail I just want to get confirmation that things work as I think
> they do and to discuss the implications and viability of such change.
The ability to pass a run-specific parameter is something we should definitely
think about. I'll discuss it more in subsequent responses.
Thanks for the input.
More information about the Fuego