[Fuego] Parameter passing to tests

[Daniel Sangorrin]

Hi Rafael,

> Let's say that I have a device with three network interfaces and I want to run the iperf (Benchmark.iperf) test on all three of them on
> different servers. How would I pass the parameters?

Thanks for rising this problem. The short answer is that right now you would have create 3 specs (e.g. eth0spec, eth1spec, eth2spec) 
with different interface parameters.

> As it seems to me now (and correct me if I'm wrong) the only way to pass parameters is either on the board file or on the spec.
> 
> If there are lots of tests in fuego on the future, placing test parameters on the board file doesn't scale. IMO the board file should
> contain only data related and to the board <-> fuego transport and maybe some hardware related staff. Test-specific variables like
> "LTP_OPEN_POSIX_SUBTEST_COUNT_POS" seem a bit out of place there to me.

Agree.

> If one does place parameters on the .spec files, then it's harder to contribute tests, as everyone will have dirty specs files with its own
> configuration. 

Yeah, I understand. I have the same problem.

Ideally we want to share specs that add value (e.g.: parameter sets that test different parts), but avoid sharing the
"dirty" specs where you only changed something like the ip address.
 
>It's also a bit rigid when developing: if you want to temporarily change a parameter you have to modify, submit a new
> project to jenkins, test, remove a project from jenkins and undo the change.
> 
> Jenkins projects have build parameters (parameterized build) which can be modified when building either through the web or
> through the CLI and are accessible from the scripts as environment variables. IMO a .spec file could only be a key-value list of Jenkins
> parameters. The Key would be the variable name and the Value the default variable value.
> 
> Adding the values can be done on the ftc's "add_job" function when it does generate the project XML by adding this:
> 
>   <properties>
>     <hudson.model.ParametersDefinitionProperty>
>       <parameterDefinitions>
>         <hudson.model.StringParameterDefinition>
>           <name>PARAM1</name>
>           <description></description>
>           <defaultValue>PARAM1 DEFAULT VALUE</defaultValue>
>         </hudson.model.StringParameterDefinition>
>         <hudson.model.StringParameterDefinition>
>           <name>PARAM2</name>
>           <description></description>
>           <defaultValue>PARAM2 DEFAULT VALUE</defaultValue>
>         </hudson.model.StringParameterDefinition>
>       </parameterDefinitions>
>     </hudson.model.ParametersDefinitionProperty>
>   </properties>
> 
> Then PARAM1 and PARAM2 are made environment variables and are visible from the fuego test scripts. They can be parameterized
> for each build in an automated way:
> 
> java -jar /opt/jenkins/jenkins-cli.jar -s http://localhost:8080/fuego <testname> -p PARAM1="hello param 1" -p PARAM2="hello param
> 2"
> 
> Doing this way places all the test (user) configuration outside of the fuego repository (fuego-core) on the user automated build scripts.
> It removes the need to have multiple projects of the same test with different specifications in jenkins. There is just a test project for
> each board.

At the moment, what you can do is to configure your job:
 Jenkins > Testname > configure
     export PARAM1="hello param 1"
     export PARAM2="hello param 2"

That should achieve the same result as using the parameterized build plugin. But it would be more
convenient to add an option to ftc:
ftc add-jobs -p PARAM1="..."

> So with this mail I just want to get confirmation that things work as I think they do and to discuss the implications and viability of such
> change.

Yes, you are correct. This is a problem that I wanted to solve. 

Configuring your job with new params is OK for debugging but in the long run
you may also want to save those values into your board's testplan.

My proposal is as follows:
  - testplans should generally be created by the user. We provide some testplans for reference only. In other words, a testplan is where you put your "dirty" changes.
  - The idea is that you could add 3 jobs to your testplan using the same iperf spec, and then _OVERWRITE_ those variables that you want to change (e.g. IP address)
  - testplans should be moved to /fuego/fuego-ro/boards. 

To achieve this we would need to update the testplans path throughout Fuego and then modify the overlay generator so that
testplan definitions overwrite those in the spec when prolog.sh is generated.

Please let me know what you think about this solution.

Thanks,
Daniel