[Fuego] Fuego Digest, Vol 13, Issue 14

dhinakar k dhinakar.k at gmail.com
Fri Aug 18 03:07:36 UTC 2017


Hi Tim,

Thanks a lot for the hard work and update about Fuego Release v1.2.
I am eagerly waiting for it.

Regards,
Dhinakar


On Fri, Aug 18, 2017 at 8:19 AM, <fuego-request at lists.linuxfoundation.org>
wrote:

> Send Fuego mailing list submissions to
>         fuego at lists.linuxfoundation.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.linuxfoundation.org/mailman/listinfo/fuego
> or, via email, send a message with subject or body 'help' to
>         fuego-request at lists.linuxfoundation.org
>
> You can reach the person managing the list at
>         fuego-owner at lists.linuxfoundation.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Fuego digest..."
>
>
> Today's Topics:
>
>    1. Re: Status of LTP parser - populating run.json (Daniel Sangorrin)
>    2. Status update for Fuego (Bird, Timothy)
>    3. Re: Status update for Fuego (Khiem Nguyen)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 10 Aug 2017 10:41:52 +0900
> From: "Daniel Sangorrin" <daniel.sangorrin at toshiba.co.jp>
> To: "'Bird, Timothy'" <Tim.Bird at sony.com>,
>         <fuego at lists.linuxfoundation.org>
> Subject: Re: [Fuego] Status of LTP parser - populating run.json
> Message-ID: <003601d31179$d76baf30$86430d90$@toshiba.co.jp>
> Content-Type: text/plain;       charset="UTF-8"
>
>
>
> > -----Original Message-----
> > From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> > Sent: Thursday, August 10, 2017 10:11 AM
> > To: Daniel Sangorrin; fuego at lists.linuxfoundation.org
> > Subject: RE: Status of LTP parser - populating run.json
> >
> >
> >
> > > -----Original Message-----
> > > From: Daniel Sangorrin on Wednesday, August 09, 2017 5:49 PM
> > > > -----Original Message-----
> > > > From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> > > > I've been doing some testing with LTP and the new parser. I'm only
> seeing
> > > one result show up
> > > > in the run.json file.  This is due to a bug in the process_data
> routine.  When
> > > I made it backwards
> > > > compatible with the other functional tests, I broke it for LTP.
> > > >
> > >
> > > Don't worry, it was broken already by my changes and waiting for an
> > > upgrade.
> > >
> > > > I think it would be better to move to the LTP parser.py program
> calling
> > > plib.process(), rather than
> > > > process_data.  Currently, only old-style functional tests call that
> routine,
> > > with a  single value
> > > > provided by generic-parser.py, and old-style benchmark tests, with
> values
> > > that can be made
> > > > into measurements.
> > >
> > > Yes, definitely.
> >
> > It's fixed in my 'next' branch - at least LTP and other tests work for me
> > (populating run.json with testcase status).
> >
> > ...
> >
> > >
> > > Thanks for the fixes.
> > > Now that the parser seems to work for many tests, it would be good to
> > > upgrade the LTP to support
> > > the new parser natively. The script already uses "test_category" which
> will
> > > become "test_set" and
> > > "test_case" which has a direct equivalent. I hope it will not be too
> much
> > > work.
> >
> > I didn't change the variable names in LTP's parser.py, but I think it
> > does what we want now.  Actually, looking at it just now, I'm fine
> leaving
> > it as "test_category".  I'm not sure exactly what the LTP nomenclature
> > for the thing is  - I've seen the wording "test scenario" used.
> >
> > LTP is a bit weird, they have multiple test defined in runtests which
> can call
> > lots of sub-tests.  I found many different test scenarios (if that's the
> right phrase)
> > that call syscall-related test programs.  So these things can be mixed
> and matched.
> >
> > I thought about adding the sub-test results as "measures" under the test
> cases,
> > but I'm holding off until I make more progress on the report generation.
>
> I think they are fine as test_cases. Probably you are looking for nested
> results,
> I already implemented a prototype and it was too complicated. The best
> solution I found
> was to collapse that hierarchy in the test_set name through a convention.
>
> For example: results[category1.subcategory2.subsubcategory3] = "PASS"
> This can be represented with a test_set called "category1.subcategory2"
> and a test_case called subsubcategory3.
> Just make sure that test_sets do not have 'dots' in their names by
> convention.
>
> >
> > As I'm working through things, there are a few items I might like to
> propose
> > for the schema, but overall things seem to be working well.
> >  -- Tim
>
> Oh I didn't notice the latest commits. Thanks!
>
> By the way, there is no reference.json or criteria.json files yet.
> Do you think we should add them?
>
> - The reference.json is useful to know what tests were skipped either
> because
> the result was CONF/BROK or because they were in the skiplist for example.
> - The criteria.json would probably need to be customized for each board,
> but a default one may be useful as well.
>
> Other than that, I think there is a way for the schema to allow adding new
> arbitrary data to an entry. With that, we can also add test-dependent
> information
> such as the LTP errtype value, or the messages generated on test cases
> that ended up
> in INFO/WARN/CONF/BROK for example. In the current schema, we have
> "attachments"
> but this is not enough.
>
> Thanks,
> Daniel
>
>
>
>
>
>
>
>
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Fri, 18 Aug 2017 00:11:49 +0000
> From: "Bird, Timothy" <Tim.Bird at sony.com>
> To: "fuego at lists.linuxfoundation.org"
>         <fuego at lists.linuxfoundation.org>
> Subject: [Fuego] Status update for Fuego
> Message-ID:
>         <ECADFF3FD767C149AD96A924E7EA6EAF3AD09D81 at USCULXMSG01.am.sony.com>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi everyone,
>
> Over the last few days I did some proof-of-concept work on
> report generation for Fuego using the new unified output
> format.  I also did some additional research on output formats
> produced by different test frameworks or harnesses.
>
> I think I can safely say that our current format (as documented
> here
> https://bitbucket.org/tbird20d/fuego-core/src/
> fac7d69b91008ce0d1c7ccf0dc55dab5b8a1ce64/engine/scripts/
> parser/fuego-schema.json?at=next
> )
>
> contains enough information for us to
> be compatible with kernelci.  I am confident we can convert our data to
>  xunit or TAP format as well, which are already supported for visualization
> by Jenkins plugins.
>
> There are a few little details remaining, but I think we can hash those out
> fairly soon.  I'm trying to figure out what details need to be closed in
> order to begin the 1.2 release.
>
> In terms of the big 4 goals for this release:
>  - unified output format - basically complete, although there may be some
>     minor adjustments in the future
>  - LAVA support - this is getting pushed to a future release
>  - documentation update for 1.2 - I've already started this.  There's
>     a 1.2 version of the Fuego wiki, that I'll swap into place when the
>     release occurs, but it needs a lot more work to document the
>     differences from 1.1 to 1.2.  This can be done during the 1.2-rc
> series,
>     so is not technically blocking the start of the release.
>  - support AGL/JTA reports (HTML output)
>
> There are also fixes to individual tests (such as a major fix for
> LTP and libaio-dev installation).
>
> This support for AGL/JTA reports is something that needs work.
> Can someone provide me with sample HTML reports from
> AGL/JTA for the following:
>  1) Benchmark.dbench
>  2) Functional.hello_world
>  3) Functional.LTP?
>
> I think with those three report samples, I can add the necessary
> support to my report generator (in ftc) to finish this feature.
>
> Also, plotting seems to be broken at the moment.  When Daniel
> gets back into town maybe he can take a look at this.
>
> In separate news, I'll be presenting at Linaro Connect (SFO17) on the
> topic of shared standards between Linux test frameworks, and we'll
> have a few slots in Embedded Linux Conference Europe on testing
> and test framework topics.
>
> After the 1.2 release, I plan to ramp up test production and
> Fuego promotion - possibly planning a mini-summit in Portland
> at ELC in March 2018.
>
> Sorry this release has taken so long, but I think we're closing in
> on the 1.2 release (hopefully the first release candidate will
> be sometime next week.)
>  -- Tim
>
>
>
>
> ------------------------------
>
> Message: 3
> Date: Fri, 18 Aug 2017 02:49:26 +0000
> From: Khiem Nguyen <khiem.nguyen.xt at renesas.com>
> To: "Bird, Timothy" <Tim.Bird at sony.com>
> Cc: Phat Le <phat.le.wh at renesas.com>, Toru Oishi
>         <toru.oishi.zj at renesas.com>,    "fuego at lists.linuxfoundation.org"
>         <fuego at lists.linuxfoundation.org>,      Canh Nguyen
>         <canh.nguyen.vz at rvc.renesas.com>
> Subject: Re: [Fuego] Status update for Fuego
> Message-ID:
>         <PS1PR06MB16741040892FBB063080E9EDA8800 at PS1PR06MB1674.
> apcprd06.prod.outlook.com>
>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Tim,
>
> Thanks for sending the update.
>
> > Over the last few days I did some proof-of-concept work on report
> generation
> > for Fuego using the new unified output format.  I also did some
> additional
> > research on output formats produced by different test frameworks or
> harnesses.
> >
> > I think I can safely say that our current format (as documented here
> > https://bitbucket.org/tbird20d/fuego-
> > core/src/fac7d69b91008ce0d1c7ccf0dc55dab5b8a1ce64/engine/scripts/
> parser/f
> > uego-schema.json?at=next
> > )
> > contains enough information for us to
> > be compatible with kernelci.  I am confident we can convert our data to
> xunit or
> > TAP format as well, which are already supported for visualization by
> Jenkins
> > plugins.
>
> Great.
>
> > There are a few little details remaining, but I think we can hash those
> out fairly
> > soon.  I'm trying to figure out what details need to be closed in order
> to begin the
> > 1.2 release.
> >
> > In terms of the big 4 goals for this release:
> >  - unified output format - basically complete, although there may be some
> >     minor adjustments in the future
> >  - LAVA support - this is getting pushed to a future release
> >  - documentation update for 1.2 - I've already started this.  There's
> >     a 1.2 version of the Fuego wiki, that I'll swap into place when the
> >     release occurs, but it needs a lot more work to document the
> >     differences from 1.1 to 1.2.  This can be done during the 1.2-rc
> series,
> >     so is not technically blocking the start of the release.
> >  - support AGL/JTA reports (HTML output)
>
> Those changes will reduce the gap between Fuego and AGL/JTA, I guess. :)
>
> > In separate news, I'll be presenting at Linaro Connect (SFO17) on the
> topic of
> > shared standards between Linux test frameworks, and we'll have a few
> slots in
> > Embedded Linux Conference Europe on testing and test framework topics.
>
> I personally expect more demonstration with Fuego and board setup.
>
> >
> > After the 1.2 release, I plan to ramp up test production and Fuego
> promotion -
> > possibly planning a mini-summit in Portland at ELC in March 2018.
> >
> > Sorry this release has taken so long, but I think we're closing in on
> the 1.2 release
> > (hopefully the first release candidate will be sometime next week.)
>
> Great.
> We can try it out and give feedback.
>
> Best regards,
> KHIEM Nguyen
>
> ------------------------------
>
> _______________________________________________
> Fuego mailing list
> Fuego at lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego
>
>
> End of Fuego Digest, Vol 13, Issue 14
> *************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/fuego/attachments/20170818/90f42d53/attachment-0001.html>


More information about the Fuego mailing list