[Fuego] [RFC]AFL-Fuego output test result change from xml to json

[Bird, Timothy]

> -----Original Message-----
> From: Cai, Song on Wednesday, March 22, 2017 7:19 PM
> Hello Jan-Simon, AGL members
> 
> Fuego members consider that it will be more versatile if we output the test
> results in json format,
> 
> Since json is widely used, and many tool (e.g. Kernel CI) is also in json format.
>
> I totally agree with that.
>
I strongly agree with this as well.  Our intermediate format for results should be
json.  By "intermediate", I mean the format used to store results
in the system, as opposed to the "display format".   We should support multiple display
output formats (such as html, xml, pdf, or wiki markup), that can be generated
from the json formatted data.
 
> And we plan to change the current result output format from xml to json,
> which is scheduled to be finished until late May.
> 
> I would appreciate any comments on this activity.

Please note that Fuego already has the capability to put results in json format now.
However, the work is scattered in different functions, and could use some
better organization and coverage.

Here is some information that may be useful in working on this issue:
 * data for a functional test that is executed by Jenkins is in the file build.xml
    * this is converted to a "run.json" file inside the run package if you execute "ftc package-run" for the run
       * so, ftc has the capability to read the Jenkins build.xml file and produce a Fuego run.json file now
 * data for a functional test that is executed by 'ftc run-test' creates the run.json file directly, and it is present in the log directory for that run
 * data for each metric of a benchmark test is placed in a file with the suffix ".res.json"
    * this is currently only generated when Fuego is executing a batch job (this should be extended to all job execution contexts)
    * this file is created by parser.py, when the benchmark metrics are collected from the log
 * additionally, when a batch job is executed, a "logrun" file is created, in JSON format
    * this file is used to collect the data from the multiple tests in the batch job, so they can be formatted into a PDF by the report generator
 * There are also some json files used for generating plots using the 'flot' module
    * there is one file per metric in the log/<testname> directory
    * there is an 'info.json' file in the log/<testname> directory, which has additional data used for the plot

Note that some of these features are currently broken in the 'next' branch, as we work through
the integration of the Toshiba and Sony code bases (the Jenkins integration refactoring).
I hope to have some of the broken issues fixed by the 1.1 release, and all of them resolved 
by the 1.2 release.

Please see the following pages, which have information on the logrun, res.json, and run.json files:
 * http://bird.org/fuego/Report_Generator
 * http://bird.org/fuego/run.json

What I'd like to see happen, is for all test result data to be put into the run.json file.
This includes data that is now put into res.json, and data that is put into the xlsx files for 
tests that have lots of sub-results (e.g. LTP).

Also, we should make sure that the data and the run.json file are created in all
contexts - not just when a test is run from ftc, or when a batch job is executed.

Also, it sounds like it would be handy to have the board data in json format.
We have python code now that parses the board file (both in ovgen.py and
in ftc), and produces python data structures for the board data.  It might be
possible to reuse this parsing code for your work.  In particular, the board data
might be getting more complex in the future, as we add the data dictionary
feature to Fuego to store dynamic variables for boards.  I can trivially add some
code to ftc to output the board data in json format, for processing by other tools.
Let me know if you would like me to do this.

Please let me know what you are planning on doing in this regard, and we can discuss
details of the design, to make sure it fits with the long-term vision for Fuego results
(and report generation) storage and display.

BTW - I saw your slides from the Japan Jamboree presentation.  They look good.
I'll post a link to them on http://bird.org/fuego/Presentations.

Thanks,
 -- Tim

P.S. Note that I've made a slight modification to reports.sh in my 'next' branch, just
today, based on this discussion.  The function set_testres_file in reports.sh used
to only set the GEN_TESTRES_FILE when operating in batch mode.  However,
I've removed this condition.  Tests should now always create a res.json file.
See https://bitbucket.org/tbird20d/fuego-core/commits/6cd8fe8bde38887b12b1a5b1713877a0ec754a2d