[Fuego] Parameter passing to tests
Rafael Gago Castano
RGC at hms.se
Thu Jun 1 08:51:06 UTC 2017
Let's say that I have a device with three network interfaces and I want to run the iperf (Benchmark.iperf) test on all three of them on different servers. How would I pass the parameters?
As it seems to me now (and correct me if I'm wrong) the only way to pass parameters is either on the board file or on the spec.
If there are lots of tests in fuego on the future, placing test parameters on the board file doesn't scale. IMO the board file should contain only data related and to the board <-> fuego transport and maybe some hardware related staff. Test-specific variables like "LTP_OPEN_POSIX_SUBTEST_COUNT_POS" seem a bit out of place there to me.
If one does place parameters on the .spec files, then it's harder to contribute tests, as everyone will have dirty specs files with its own configuration. It's also a bit rigid when developing: if you want to temporarily change a parameter you have to modify, submit a new project to jenkins, test, remove a project from jenkins and undo the change.
Jenkins projects have build parameters (parameterized build) which can be modified when building either through the web or through the CLI and are accessible from the scripts as environment variables. IMO a .spec file could only be a key-value list of Jenkins parameters. The Key would be the variable name and the Value the default variable value.
Adding the values can be done on the ftc's "add_job" function when it does generate the project XML by adding this:
<defaultValue>PARAM1 DEFAULT VALUE</defaultValue>
<defaultValue>PARAM2 DEFAULT VALUE</defaultValue>
Then PARAM1 and PARAM2 are made environment variables and are visible from the fuego test scripts. They can be parameterized for each build in an automated way:
java -jar /opt/jenkins/jenkins-cli.jar -s http://localhost:8080/fuego <testname> -p PARAM1="hello param 1" -p PARAM2="hello param 2"
Doing this way places all the test (user) configuration outside of the fuego repository (fuego-core) on the user automated build scripts. It removes the need to have multiple projects of the same test with different specifications in jenkins. There is just a test project for each board.
So with this mail I just want to get confirmation that things work as I think they do and to discuss the implications and viability of such change.
More information about the Fuego