[Fuego] Discussion about Fuego unified results format

Bird, Timothy Tim.Bird at sony.com
Thu Apr 13 19:19:22 UTC 2017


Daniel and Cai,

Based on discussion in our call yesterday, I have a few notes I'd like to make.
First - for those not involved in the call, here is some introduction.  We are
working, for the 1.2 release, on unifying the results format produced by Fuego
tests, to make report generation easier, and support (ultimately) multiple
report output formats (e.g. HTML, XML, PDF, Excel).  The intent is to support
all current AGL-JTA reports, previous Fuego reports, and new ones envisioned
for the system (in the 1.2 release).

Note from the call and about this work are at:
http://bird.org/fuego/Unified_Results_Format_Project

Here are a couple of extra ideas about this work that I had after thinking
about our call discussion.

It appears that Song is working on the individual test results,
and the supporting infrastructure for that (outputting what used to be in
the test_results.xml file into a new JSON-formatted file).  It appears that
Daniel is working on aggregate results (results from multiple runs) that will
replace what is now in plot.data (and maybe the former 'logrun' files?).  These
are for use by flot and to generate other reports or views of the data.

I think these two efforts are orthogonal, but let me know if I am wrong.

A test 'run' should always have a standalone file that reports its results, in the JSON
results format, that is contained in the log directory for that run.  In the future will
would like to be able to transmit and store this 'run' information separate from the
host where they were generated.  Indeed ftc currently supports packaging and transmitting
run data, with the 'ftc package-run' and 'ftc put-run' commands. These will have to
change (very slightly) to accommodate the new file(s), but this is very little effort, which I'll
take care of.

In any event, please be prepared to add a 'host' field to the schema for the metadata for
the test results.  I will be adding the 'host' definition to the global configuration for Fuego, so
it will be in the 'run.json' file, and can easily be transferred to the results.json file, if desired.

We need to decide if we will put all the run meta-data into a single results.json file, or leave some
of it in the run.json file.  There are pros and cons to both approaches.  The run.json file is created
at the very end of a test, and it's nice to have it be a separate file from the results.json file, in terms
of division of responsibility for populating the files by the test tools and libraries provided by Fuego.
But duplicating fields is kind of annoying, as is having to manage the data for a run in two separate files.
I think this is a minor point that is easy to change, but we should discuss it before we finalize the
schema for both files for the 1.2 release.

As you know, I'm going on vacation next week. Before Song starts replicating all the new Functional
parser.py programs into all the tests, I'd like to review it.  But I don't want things to slow down while I'm
gone.  I leave it to your discretion to decide how much to proliferate the new programs before I've
had a chance to review them.

That's it for now.  Have fun.
 -- Tim





More information about the Fuego mailing list