[Fuego] Status update for Fuego

[Bird, Timothy]

> -----Original Message-----
> From: Daniel on Sunday, August 20, 2017 7:42 PM
>
> Thanks for the status update. I just checked gen-report and looks good for
> now.
> 
> > -----Original Message-----
> > From: fuego-bounces at lists.linuxfoundation.org [mailto:fuego-
> bounces at lists.linuxfoundation.org] On Behalf Of Bird, Timothy
> > Sent: Friday, August 18, 2017 9:12 AM
>  ...
> > Also, plotting seems to be broken at the moment.  When Daniel
> > gets back into town maybe he can take a look at this.
> 
> I'm back into town, although I will be quite busy this and next month.
> 
> About plotting, we had 2 types of plotting:
>   - the plot.png generated, mostly for benchmark tests, with matplotlib.
>   - the interactive plots made with the flot javascript library
> 
> My plan is to move the matplotlib plots to the ftc gen-report command.

That was my plan as well.  the plot.png generated by matplotlib would be
good for inclusion in PDF and HTML reports.

> I need to think about how to integrate that with the current interface.
> 
> About the interactive plots, I have disabled them in favor of the HTML
> reports.
> # The plotting code is still there in mod.js but temporarily disabled.
> 
> One of the reasons I left it disabled was that despite for simple tests the
> interactive plots where
> fine, tests with many test_cases and measures required at least as many
> interactive
> plots and that caused a rather high overhead on my browser.
> Do you want the interactive plots back in addition to the HTML tables output?

I gave this some thought, and came to the conclusion that the information
that was originally stored in the metrics.json file for each test was a kind
of "default display" indicator for each benchmark test.  That is, even if a benchmark
had a huge number of measures, only the ones that were specified in the
metrics.json file were presented to the user in the Jenkins interface.

Ultimately, I'd like to extend this concept, and have a json file that indicates
to the Fuego system what to put as the default visualization for a test on the
Jenkins page.  This could be:
 -  an HTML table indicating history of pass or fail
 - a plot of some measures over multiple runs
 - multiple plots of different measures, grouped by board, by platform, by kernel, etc.

This would be a like a simplified version of the report configuration file.
The report configuration file, in my vision, will indicate the tests to include
in the report, some filters to select specific data to show, and some ways
to control how the data is formatted (as a table, as a list, or as a plot, and how
the data is grouped).  Some of these would be implicit in a default configuration
for the Jenkins page, such as the test, and the amount of runs to include.
A Jenkins page for a test should show the history for the last 'n' runs, with
'n' determined by a global Fuego variable.

I think for the 1.2 release I'm fine leaving it as the html table - although for me
the HTML table for LTP is too big to show in my browser window.
 -- Tim