[Fuego] [agl-discussions] [RFC]AGL-Fuego output test result change from xml to json

Wed Apr 12 10:40:09 UTC 2017

Hi Tim:
  Our job about LTP is going to draw to an end.
  So, can you take some time for us to discuss the issues "xml to json recently", since both of us have done the work by ourselves before.
  Thursday and Friday this week is available, or other time at your convinces. My time zone is UTC +8.
  Considering the network in China, goto meeting maybe be good choice.
  Looking forward to your reply.
  Thanks.

--------------------------------------------------
B.R.
Cai Song
--------------------------------------------------

-----Original Message-----
From: Bird, Timothy [mailto:Tim.Bird at sony.com] 
Sent: Tuesday, April 04, 2017 12:21 PM
To: Cai, Song/蔡 嵩 <cais.fnst at cn.fujitsu.com>; automotive-discussions at lists.linuxfoundation.org; fuego at lists.linuxfoundation.org
Subject: RE: [agl-discussions] [RFC]AGL-Fuego output test result change from xml to json

Sounds good.  Let me know when you are ready to resume work on this part of Fuego.
I'm currently doing a lot of integration work with Daniel Sangorrin, of Toshiba.
Hopefully that will be settled down by the end of this week, and we'll have some time to work on the AGL integration and unified output and report generation issues.
 -- Tim

> -----Original Message-----
> From: automotive-discussions-bounces at lists.linuxfoundation.org
> [mailto:automotive-discussions-bounces at lists.linuxfoundation.org] On 
> Behalf Of Cai, Song
> Sent: Friday, March 31, 2017 4:03 AM
> To: Bird, Timothy <Tim.Bird at sony.com>; automotive- 
> discussions at lists.linuxfoundation.org; fuego at lists.linuxfoundation.org
> Subject: Re: [agl-discussions] [RFC]AGL-Fuego output test result 
> change from xml to json
> 
> Hi Tim:
>   Sorry for my late reply.
>   Our team are solving the cases of LTP on 64 bits boards.
> 
>   And we have accepted the AGL community job "integrate AGL-Fuego to 
> Fuego" recently, so we need to keep progress on it after we finish LTP cases.
> 
>   We can deal with the community job and the json issue at the same 
> time, since it is included in integration.
> 
>   Hope for your understanding.
>   Thanks.
> B.R
> CaiSong
> 
> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> Sent: Friday, March 24, 2017 3:37 AM
> To: Cai, Song/蔡 嵩 <cais.fnst at cn.fujitsu.com>; automotive- 
> discussions at lists.linuxfoundation.org; fuego at lists.linuxfoundation.org
> Subject: RE: [RFC]AFL-Fuego output test result change from xml to json
> 
> > -----Original Message-----
> > From: Cai, Song on Wednesday, March 22, 2017 7:19 PM Hello 
> > Jan-Simon, AGL members
> >
> > Fuego members consider that it will be more versatile if we output 
> > the test results in json format,
> >
> > Since json is widely used, and many tool (e.g. Kernel CI) is also in 
> > json
> format.
> >
> > I totally agree with that.
> >
> I strongly agree with this as well.  Our intermediate format for 
> results should be json.  By "intermediate", I mean the format used to store results
> in the system, as opposed to the "display format".   We should support
> multiple display
> output formats (such as html, xml, pdf, or wiki markup), that can be 
> generated from the json formatted data.
> 
> > And we plan to change the current result output format from xml to 
> > json, which is scheduled to be finished until late May.
> >
> > I would appreciate any comments on this activity.
> 
> Please note that Fuego already has the capability to put results in 
> json format now.
> However, the work is scattered in different functions, and could use 
> some better organization and coverage.
> 
> Here is some information that may be useful in working on this issue:
>  * data for a functional test that is executed by Jenkins is in the file build.xml
>     * this is converted to a "run.json" file inside the run package if 
> you execute "ftc package-run" for the run
>        * so, ftc has the capability to read the Jenkins build.xml file 
> and produce a Fuego run.json file now
>  * data for a functional test that is executed by 'ftc run-test' 
> creates the run.json file directly, and it is present in the log 
> directory for that run
>  * data for each metric of a benchmark test is placed in a file with 
> the suffix ".res.json"
>     * this is currently only generated when Fuego is executing a batch 
> job (this should be extended to all job execution contexts)
>     * this file is created by parser.py, when the benchmark metrics 
> are collected from the log
>  * additionally, when a batch job is executed, a "logrun" file is 
> created, in JSON format
>     * this file is used to collect the data from the multiple tests in 
> the batch job, so they can be formatted into a PDF by the report 
> generator
>  * There are also some json files used for generating plots using the 'flot'
> module
>     * there is one file per metric in the log/<testname> directory
>     * there is an 'info.json' file in the log/<testname> directory, 
> which has additional data used for the plot
> 
> Note that some of these features are currently broken in the 'next' 
> branch, as we work through the integration of the Toshiba and Sony 
> code bases (the Jenkins integration refactoring).
> I hope to have some of the broken issues fixed by the 1.1 release, and 
> all of them resolved by the 1.2 release.
> 
> Please see the following pages, which have information on the logrun, 
> res.json, and run.json files:
>  * http://bird.org/fuego/Report_Generator
>  * http://bird.org/fuego/run.json
> 
> What I'd like to see happen, is for all test result data to be put 
> into the run.json file.
> This includes data that is now put into res.json, and data that is put 
> into the xlsx files for tests that have lots of sub-results (e.g. LTP).
> 
> Also, we should make sure that the data and the run.json file are 
> created in all contexts - not just when a test is run from ftc, or 
> when a batch job is executed.
> 
> Also, it sounds like it would be handy to have the board data in json format.
> We have python code now that parses the board file (both in ovgen.py 
> and in ftc), and produces python data structures for the board data.  
> It might be possible to reuse this parsing code for your work.  In 
> particular, the board data might be getting more complex in the 
> future, as we add the data dictionary feature to Fuego to store 
> dynamic variables for boards.  I can trivially add some code to ftc to 
> output the board data in json format, for processing by other tools.
> Let me know if you would like me to do this.
> 
> Please let me know what you are planning on doing in this regard, and 
> we can discuss details of the design, to make sure it fits with the 
> long-term vision for Fuego results (and report generation) storage and display.
> 
> BTW - I saw your slides from the Japan Jamboree presentation.  They 
> look good.
> I'll post a link to them on http://bird.org/fuego/Presentations.
> 
> Thanks,
>  -- Tim
> 
> P.S. Note that I've made a slight modification to reports.sh in my 'next'
> branch, just today, based on this discussion.  The function 
> set_testres_file in reports.sh used to only set the GEN_TESTRES_FILE 
> when operating in batch mode.  However, I've removed this condition.  
> Tests should now always create a res.json file.
> See https://bitbucket.org/tbird20d/fuego-
> core/commits/6cd8fe8bde38887b12b1a5b1713877a0ec754a2d
> 
> 
> 
> 
> 
> _______________________________________________
> automotive-discussions mailing list
> automotive-discussions at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/automotive-discussi
> ons