[Fuego] [PATCH 2/3] cyclictest: modify specs

[Daniel Sangorrin]

> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> Sent: Thursday, November 02, 2017 2:16 AM
> To: Daniel Sangorrin; fuego at lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 2/3] cyclictest: modify specs
> 
> 
> 
> > -----Original Message-----
> > From: fuego-bounces at lists.linuxfoundation.org [mailto:fuego-
> > bounces at lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> > Sent: Tuesday, October 31, 2017 1:32 AM
> > To: fuego at lists.linuxfoundation.org
> > Subject: [Fuego] [PATCH 2/3] cyclictest: modify specs
> >
> > cyclictest params can be very flexible so instead of
> > giving a fixed pattern we use a single variable PARAMS
> > to hold them.
> >
> > TODO: make builds parameterizable so that users can
> > modify the PARAMS from ftc without needing to create
> > a new spec.
> 
> Just a comment on this.  I support this idea, and it's actually
> pretty easy to do.  We just need to be careful that this doesn't negatively
> affect another of Fuego's goal, which is sharing of 1) methods of invoking
> test programs, and 2) ability to compare results from multiple systems.
> 
> Test specs are one of the primary ways that Fuego users can share their
> experience with each other - by creating canonical sets of build variables
> that produce meaningful results for particular test scenarios.
> 
> If use of custom parameters for tests becomes widespread, then it becomes
> less useful to compare results between different Fuego sites.
> 
> I'm not saying we shouldn't add the feature, just be judicious in it's
> use, and possibly provide recommendations for when it would be
> appropriate or not.

Yes, I was thinking that ftc could create the spec.json automatically 
on the log folder and then put the corresponding shell environment 
variable to it (actually this can be done in the general case as well). 
Once it's in the log folder, we can share it.

Another thing we could add is "default" parameters so that users
only need to modify those that are important to them and leave the
rest untouched. Maybe that's easy to do by just specifying the base
"default" parameters with -s as now, and then override the variables.

Thanks,
Daniel