[Fuego] [agl-discussions] [RFC]AGL-Fuego output test result change from xml to json

Tue Apr 4 04:20:34 UTC 2017

Sounds good.  Let me know when you are ready to resume work on this part of Fuego.
I'm currently doing a lot of integration work with Daniel Sangorrin, of Toshiba.
Hopefully that will be settled down by the end of this week, and we'll have some time
to work on the AGL integration and unified output and report generation issues.
 -- Tim

> -----Original Message-----
> From: automotive-discussions-bounces at lists.linuxfoundation.org
> [mailto:automotive-discussions-bounces at lists.linuxfoundation.org] On
> Behalf Of Cai, Song
> Sent: Friday, March 31, 2017 4:03 AM
> To: Bird, Timothy <Tim.Bird at sony.com>; automotive-
> discussions at lists.linuxfoundation.org; fuego at lists.linuxfoundation.org
> Subject: Re: [agl-discussions] [RFC]AGL-Fuego output test result change from
> xml to json
> 
> Hi Tim:
>   Sorry for my late reply.
>   Our team are solving the cases of LTP on 64 bits boards.
> 
>   And we have accepted the AGL community job "integrate AGL-Fuego to
> Fuego" recently, so we need to keep progress on it after we finish LTP cases.
> 
>   We can deal with the community job and the json issue at the same time,
> since it is included in integration.
> 
>   Hope for your understanding.
>   Thanks.
> B.R
> CaiSong
> 
> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> Sent: Friday, March 24, 2017 3:37 AM
> To: Cai, Song/蔡 嵩 <cais.fnst at cn.fujitsu.com>; automotive-
> discussions at lists.linuxfoundation.org; fuego at lists.linuxfoundation.org
> Subject: RE: [RFC]AFL-Fuego output test result change from xml to json
> 
> > -----Original Message-----
> > From: Cai, Song on Wednesday, March 22, 2017 7:19 PM Hello Jan-Simon,
> > AGL members
> >
> > Fuego members consider that it will be more versatile if we output the
> > test results in json format,
> >
> > Since json is widely used, and many tool (e.g. Kernel CI) is also in json
> format.
> >
> > I totally agree with that.
> >
> I strongly agree with this as well.  Our intermediate format for results should
> be json.  By "intermediate", I mean the format used to store results
> in the system, as opposed to the "display format".   We should support
> multiple display
> output formats (such as html, xml, pdf, or wiki markup), that can be
> generated from the json formatted data.
> 
> > And we plan to change the current result output format from xml to
> > json, which is scheduled to be finished until late May.
> >
> > I would appreciate any comments on this activity.
> 
> Please note that Fuego already has the capability to put results in json format
> now.
> However, the work is scattered in different functions, and could use some
> better organization and coverage.
> 
> Here is some information that may be useful in working on this issue:
>  * data for a functional test that is executed by Jenkins is in the file build.xml
>     * this is converted to a "run.json" file inside the run package if you execute
> "ftc package-run" for the run
>        * so, ftc has the capability to read the Jenkins build.xml file and produce a
> Fuego run.json file now
>  * data for a functional test that is executed by 'ftc run-test' creates the
> run.json file directly, and it is present in the log directory for that run
>  * data for each metric of a benchmark test is placed in a file with the suffix
> ".res.json"
>     * this is currently only generated when Fuego is executing a batch job (this
> should be extended to all job execution contexts)
>     * this file is created by parser.py, when the benchmark metrics are
> collected from the log
>  * additionally, when a batch job is executed, a "logrun" file is created, in
> JSON format
>     * this file is used to collect the data from the multiple tests in the batch job,
> so they can be formatted into a PDF by the report generator
>  * There are also some json files used for generating plots using the 'flot'
> module
>     * there is one file per metric in the log/<testname> directory
>     * there is an 'info.json' file in the log/<testname> directory, which has
> additional data used for the plot
> 
> Note that some of these features are currently broken in the 'next' branch,
> as we work through the integration of the Toshiba and Sony code bases (the
> Jenkins integration refactoring).
> I hope to have some of the broken issues fixed by the 1.1 release, and all of
> them resolved by the 1.2 release.
> 
> Please see the following pages, which have information on the logrun,
> res.json, and run.json files:
>  * http://bird.org/fuego/Report_Generator
>  * http://bird.org/fuego/run.json
> 
> What I'd like to see happen, is for all test result data to be put into the
> run.json file.
> This includes data that is now put into res.json, and data that is put into the
> xlsx files for tests that have lots of sub-results (e.g. LTP).
> 
> Also, we should make sure that the data and the run.json file are created in
> all contexts - not just when a test is run from ftc, or when a batch job is
> executed.
> 
> Also, it sounds like it would be handy to have the board data in json format.
> We have python code now that parses the board file (both in ovgen.py and
> in ftc), and produces python data structures for the board data.  It might be
> possible to reuse this parsing code for your work.  In particular, the board
> data might be getting more complex in the future, as we add the data
> dictionary feature to Fuego to store dynamic variables for boards.  I can
> trivially add some code to ftc to output the board data in json format, for
> processing by other tools.
> Let me know if you would like me to do this.
> 
> Please let me know what you are planning on doing in this regard, and we
> can discuss details of the design, to make sure it fits with the long-term vision
> for Fuego results (and report generation) storage and display.
> 
> BTW - I saw your slides from the Japan Jamboree presentation.  They look
> good.
> I'll post a link to them on http://bird.org/fuego/Presentations.
> 
> Thanks,
>  -- Tim
> 
> P.S. Note that I've made a slight modification to reports.sh in my 'next'
> branch, just today, based on this discussion.  The function set_testres_file in
> reports.sh used to only set the GEN_TESTRES_FILE when operating in batch
> mode.  However, I've removed this condition.  Tests should now always
> create a res.json file.
> See https://bitbucket.org/tbird20d/fuego-
> core/commits/6cd8fe8bde38887b12b1a5b1713877a0ec754a2d
> 
> 
> 
> 
> 
> _______________________________________________
> automotive-discussions mailing list
> automotive-discussions at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/automotive-discussions