[Fuego] [agl-discussions] [RFC]AGL-Fuego output test result change from xml to json

Wed Apr 12 11:23:19 UTC 2017

Hi Tim / Song,

I can host a gotomeeting bridge if needed. Just ping with the timeslot.

Best,
JS
Am Mittwoch, 12. April 2017, 10:40:09 schrieb Cai, Song:
> Hi Tim:
>   Our job about LTP is going to draw to an end.
>   So, can you take some time for us to discuss the issues "xml to json
> recently", since both of us have done the work by ourselves before.
> Thursday and Friday this week is available, or other time at your
> convinces. My time zone is UTC +8. Considering the network in China, goto
> meeting maybe be good choice. Looking forward to your reply.
>   Thanks.
> 
> --------------------------------------------------
> B.R.
> Cai Song
> --------------------------------------------------
> 
> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> Sent: Tuesday, April 04, 2017 12:21 PM
> To: Cai, Song/蔡 嵩 <cais.fnst at cn.fujitsu.com>;
> automotive-discussions at lists.linuxfoundation.org;
> fuego at lists.linuxfoundation.org Subject: RE: [agl-discussions]
> [RFC]AGL-Fuego output test result change from xml to json
> 
> Sounds good.  Let me know when you are ready to resume work on this part of
> Fuego. I'm currently doing a lot of integration work with Daniel Sangorrin,
> of Toshiba. Hopefully that will be settled down by the end of this week,
> and we'll have some time to work on the AGL integration and unified output
> and report generation issues. -- Tim
> 
> > -----Original Message-----
> > From: automotive-discussions-bounces at lists.linuxfoundation.org
> > [mailto:automotive-discussions-bounces at lists.linuxfoundation.org] On
> > Behalf Of Cai, Song
> > Sent: Friday, March 31, 2017 4:03 AM
> > To: Bird, Timothy <Tim.Bird at sony.com>; automotive-
> > discussions at lists.linuxfoundation.org; fuego at lists.linuxfoundation.org
> > Subject: Re: [agl-discussions] [RFC]AGL-Fuego output test result
> > change from xml to json
> > 
> > Hi Tim:
> >   Sorry for my late reply.
> >   Our team are solving the cases of LTP on 64 bits boards.
> >   
> >   And we have accepted the AGL community job "integrate AGL-Fuego to
> > 
> > Fuego" recently, so we need to keep progress on it after we finish LTP
> > cases.> 
> >   We can deal with the community job and the json issue at the same
> > 
> > time, since it is included in integration.
> > 
> >   Hope for your understanding.
> >   Thanks.
> > 
> > B.R
> > CaiSong
> > 
> > -----Original Message-----
> > From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> > Sent: Friday, March 24, 2017 3:37 AM
> > To: Cai, Song/蔡 嵩 <cais.fnst at cn.fujitsu.com>; automotive-
> > discussions at lists.linuxfoundation.org; fuego at lists.linuxfoundation.org
> > Subject: RE: [RFC]AFL-Fuego output test result change from xml to json
> > 
> > > -----Original Message-----
> > > From: Cai, Song on Wednesday, March 22, 2017 7:19 PM Hello
> > > Jan-Simon, AGL members
> > > 
> > > Fuego members consider that it will be more versatile if we output
> > > the test results in json format,
> > > 
> > > Since json is widely used, and many tool (e.g. Kernel CI) is also in
> > > json
> > 
> > format.
> > 
> > > I totally agree with that.
> > 
> > I strongly agree with this as well.  Our intermediate format for
> > results should be json.  By "intermediate", I mean the format used to
> > store results in the system, as opposed to the "display format".   We
> > should support multiple display
> > output formats (such as html, xml, pdf, or wiki markup), that can be
> > generated from the json formatted data.
> > 
> > > And we plan to change the current result output format from xml to
> > > json, which is scheduled to be finished until late May.
> > > 
> > > I would appreciate any comments on this activity.
> > 
> > Please note that Fuego already has the capability to put results in
> > json format now.
> > However, the work is scattered in different functions, and could use
> > some better organization and coverage.
> > 
> > Here is some information that may be useful in working on this issue:
> >  * data for a functional test that is executed by Jenkins is in the file
> >  build.xml>  
> >     * this is converted to a "run.json" file inside the run package if
> > 
> > you execute "ftc package-run" for the run
> > 
> >        * so, ftc has the capability to read the Jenkins build.xml file
> > 
> > and produce a Fuego run.json file now
> > 
> >  * data for a functional test that is executed by 'ftc run-test'
> > 
> > creates the run.json file directly, and it is present in the log
> > directory for that run
> > 
> >  * data for each metric of a benchmark test is placed in a file with
> > 
> > the suffix ".res.json"
> > 
> >     * this is currently only generated when Fuego is executing a batch
> > 
> > job (this should be extended to all job execution contexts)
> > 
> >     * this file is created by parser.py, when the benchmark metrics
> > 
> > are collected from the log
> > 
> >  * additionally, when a batch job is executed, a "logrun" file is
> > 
> > created, in JSON format
> > 
> >     * this file is used to collect the data from the multiple tests in
> > 
> > the batch job, so they can be formatted into a PDF by the report
> > generator
> > 
> >  * There are also some json files used for generating plots using the
> >  'flot'
> > 
> > module
> > 
> >     * there is one file per metric in the log/<testname> directory
> >     * there is an 'info.json' file in the log/<testname> directory,
> > 
> > which has additional data used for the plot
> > 
> > Note that some of these features are currently broken in the 'next'
> > branch, as we work through the integration of the Toshiba and Sony
> > code bases (the Jenkins integration refactoring).
> > I hope to have some of the broken issues fixed by the 1.1 release, and
> > all of them resolved by the 1.2 release.
> > 
> > Please see the following pages, which have information on the logrun,
> > 
> > res.json, and run.json files:
> >  * http://bird.org/fuego/Report_Generator
> >  * http://bird.org/fuego/run.json
> > 
> > What I'd like to see happen, is for all test result data to be put
> > into the run.json file.
> > This includes data that is now put into res.json, and data that is put
> > into the xlsx files for tests that have lots of sub-results (e.g. LTP).
> > 
> > Also, we should make sure that the data and the run.json file are
> > created in all contexts - not just when a test is run from ftc, or
> > when a batch job is executed.
> > 
> > Also, it sounds like it would be handy to have the board data in json
> > format. We have python code now that parses the board file (both in
> > ovgen.py and in ftc), and produces python data structures for the board
> > data. It might be possible to reuse this parsing code for your work.  In
> > particular, the board data might be getting more complex in the
> > future, as we add the data dictionary feature to Fuego to store
> > dynamic variables for boards.  I can trivially add some code to ftc to
> > output the board data in json format, for processing by other tools.
> > Let me know if you would like me to do this.
> > 
> > Please let me know what you are planning on doing in this regard, and
> > we can discuss details of the design, to make sure it fits with the
> > long-term vision for Fuego results (and report generation) storage and
> > display.
> > 
> > BTW - I saw your slides from the Japan Jamboree presentation.  They
> > look good.
> > I'll post a link to them on http://bird.org/fuego/Presentations.
> > 
> > Thanks,
> > 
> >  -- Tim
> > 
> > P.S. Note that I've made a slight modification to reports.sh in my 'next'
> > branch, just today, based on this discussion.  The function
> > set_testres_file in reports.sh used to only set the GEN_TESTRES_FILE
> > when operating in batch mode.  However, I've removed this condition.
> > Tests should now always create a res.json file.
> > See https://bitbucket.org/tbird20d/fuego-> > core/commits/6cd8fe8bde38887b12b1a5b1713877a0ec754a2d
> > 
> > 
> > 
> > 
> > 
> > _______________________________________________
> > automotive-discussions mailing list
> > automotive-discussions at lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/automotive-discussi
> > ons
> 
> _______________________________________________
> automotive-discussions mailing list
> automotive-discussions at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/automotive-discussions