[Fuego] Fuego criteria file documentation

[Daniel Sangorrin]

> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> > -----Original Message-----
> > From: Daniel Sangorrin on Sunday, November 05, 2017 9:03 PM
> > > -----Original Message-----
> > > From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> > >
> > > Hello all (but especially Daniel),
> > >
> > > I've been writing the documentation for new features in the Fuego 1.2
> > release,
> > > and I've put together a page about the new criteria file.
> > >
> > > The page is here:
> > > http://fuegotest.org/wiki/criteria.json
> > >
> > > Can you please take a look and see the information there is correct?
> >
> > Thanks for putting all this important documentation together.
> > I just reviewed it and I think that it reflects accurately the current status of
> > criteria.json.
> >
> > I have some comments about the section "Customizing the criteria.json file
> > for a board":
> >
> > -  I found a small errata: FUEGO_CRITERA_JSON_PATH ->
> > FUEGO_CRITERIA_JSON_PATH
> Fixed.  Thanks.
> 
> >
> > - FUEGO_CRITERIA_JSON_PATH: it seems to me that this functionality is not
> > completely
> > implemented yet. For example, there are no checks on ftc add-job/run-test
> > and the
> > variable is not stored together with other test flags (such as REBUILD or
> > REBOOT).
> > Personally, I would add a "--criteria-json-path" parameter to ftc add-job/run-
> > test.
> > What do you think? I can work on this.
> 
> Sounds good.  I think it would be better as just "--criteria-path".  I don't
> want to encode the file format in the command line argument.
> Using the same rationale, maybe we should remove "JSON_" from
> FUEGO_CRITERIA_JSON_PATH as well (so it's just FUEGO_CRITERIA_PATH).

OK, I will work on this.

> > - I think we should provide a flag "--store-criteria" (by default active) to
> > indicate whether criteria.json
> > should be stored on the log folder for sharing or future reference.
> I would lean towards always storing it, and not having a flag.

OK. I will work on this as well.
Should we store criteria.json files that have not been modified as well?
Probably we could infer it from the "fuego_core_version" variable but it would
be a bit tedious (git checkout etc..)

> > - The same we are doing for criteria.json is probably required for specs.json
> > and chart_config.json.
> I would agree with saving the spec that was used in the test.  Since someone
> might change the spec after running the test, it should get recorded for an
> individual run.  I'm not sure we need the whole file, and specs are short enough
> that it might just be easier to save the variable definitions from the spec, in
> the run.json file.  Let me know what you think.
> 
> With regard to chart_config.json - that's a presentation detail that isn't tied
> to specific run.  It's tied more to a test than to a run.

For chart_config.json I meant that we should also provide a parameter (--chart-config-path?)
for the user to customize what gets displayed. For example, depending on the board, spec or
even the user's taste/goal the test results to display may differ. Suppose I am
working a specific part of the kernel and I want LTP to only display a table with results 
of those testcases that I consider relevant. 

When sharing the results to a centralized server, having the chart_config.json 
could be beneficial if the user wants to specify what results should be displayed
and which not. It would also easy backwards-compatibility if the test names
change.

Thanks,
Daniel