[Fuego] Ideas for ftc report generation
Tim.Bird at sony.com
Thu Aug 10 01:35:03 UTC 2017
In order to test that our json files have some utility not previously available, I've started
working on report generation.
I'm starting simple, and will work up from there.
I have 'ftc list-runs' working again (it broke with the run.json changes).
I decided it's useful to have a run.json even if the test aborts, so I added (back)
a function in functions.sh to create one at the start of the test, with the information
available for the test so far.
I'm starting by adding some ftc commands with text output for some simple reports, and
then I'll graduate to HTML output and other formats.
I'd like to be able to control what runs are included in the report (the input), as well
as the format, fields and layout of the output.
- format=text, html, excel, pdf
- fields=what meta-data, testcases or measures to include
- layout = tables, summaries, plots, etc.
I'm thinking of starting small this release, just to get proof of concept, and then
building on that in subsequent releases.
Here are some sample command lines:
ftc gen-report --where test=Functional.LTP,testcases=~mem.* --format=text --fields=status
This would generate a text report for the status of all LTP tests starting with 'mem', in a default layout.
The default layout would have a standard set of metadata, and a list of results in table format,
I don't want to create my own whole language for reports, but I'm thinking it would be useful
to support a template for reports. So we could do something like:
ftc gen-report --where test=Functional.LTP,since=2017-08-03,board=beaglebone --template=my-report1
In this case, 'my-report' would indicate the format, fields and layout for the data for the report.
I'm working toward supporting something like the following for my own needs:
ftc gen-report --where=ltsi-boards-and-tests.txt --template=ltsi-report
In this case, the 'where' parameters come from a file instead of the command line,
and the ltsi-report template specifies (or generates) the output.
Suggestions for a templating language would be welcome. LAVA seems to like jinja, but that seems
dedicated to HTML output. There are lots to choose from (https://wiki.python.org/moin/Templating)
Anyway - I've got to run, but I'll be doing some report prototyping tomorrow.
Feedback is appreciated.
More information about the Fuego