[Fuego] Fuego criteria file documentation

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on Sunday, November 05, 2017 9:03 PM
> > -----Original Message-----
> > From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> >
> > Hello all (but especially Daniel),
> >
> > I've been writing the documentation for new features in the Fuego 1.2
> release,
> > and I've put together a page about the new criteria file.
> >
> > The page is here:
> > http://fuegotest.org/wiki/criteria.json
> >
> > Can you please take a look and see the information there is correct?
> 
> Thanks for putting all this important documentation together.
> I just reviewed it and I think that it reflects accurately the current status of
> criteria.json.
> 
> I have some comments about the section "Customizing the criteria.json file
> for a board":
> 
> -  I found a small errata: FUEGO_CRITERA_JSON_PATH ->
> FUEGO_CRITERIA_JSON_PATH
Fixed.  Thanks.

> 
> - FUEGO_CRITERIA_JSON_PATH: it seems to me that this functionality is not
> completely
> implemented yet. For example, there are no checks on ftc add-job/run-test
> and the
> variable is not stored together with other test flags (such as REBUILD or
> REBOOT).
> Personally, I would add a "--criteria-json-path" parameter to ftc add-job/run-
> test.
> What do you think? I can work on this.

Sounds good.  I think it would be better as just "--criteria-path".  I don't
want to encode the file format in the command line argument.
Using the same rationale, maybe we should remove "JSON_" from
FUEGO_CRITERIA_JSON_PATH as well (so it's just FUEGO_CRITERIA_PATH).

> 
> - I think we should provide a flag "--store-criteria" (by default active) to
> indicate whether criteria.json
> should be stored on the log folder for sharing or future reference.
I would lean towards always storing it, and not having a flag.

> 
> - The same we are doing for criteria.json is probably required for specs.json
> and chart_config.json.
I would agree with saving the spec that was used in the test.  Since someone
might change the spec after running the test, it should get recorded for an
individual run.  I'm not sure we need the whole file, and specs are short enough
that it might just be easier to save the variable definitions from the spec, in
the run.json file.  Let me know what you think.

With regard to chart_config.json - that's a presentation detail that isn't tied
to specific run.  It's tied more to a test than to a run.
 -- Tim