[Fuego] LPC Increase Test Coverage in a Linux-based OS

Guillermo Adrian Ponce Castañeda ga.poncec at gmail.com
Tue Nov 8 19:38:13 UTC 2016

Hello Tim and Victor,

I am a co-author of this code and I must confess that it was more or less
my fault that it was made on Perl.

Regarding how many logs the program analyzes, I think it is nowhere near
5000, it is much less, but taking in count that some logs are similar I
think it is possible that some logs that haven't been tested are going to
work, but who knows :).

And about the output file, right now it delivers a comma separated list of
numbers, without headers, this is because this code is part of a  bigger
tool, I think that code is not open source yet, but that doesn't matter I
guess, the thing here is that I think the output could be changed into a
json like you suggested and i can try to translate the code from Perl to
Python, still not sure how long it's gonna take, but I can sure try.

- Guillermo Ponce

On Mon, Nov 7, 2016 at 6:26 PM, Bird, Timothy <Tim.Bird at am.sony.com> wrote:

> Victor,
> Thanks for raising this topic.  I think it's an important one.  I have
> some comments below, inline.
> > -----Original Message-----
> > From: Victor Rodriguez on Saturday, November 05, 2016 10:15 AM
> >
> > This week I presented a case of study for the problem of lack of test
> > log output standardization in the majority of packages that are used
> > to build the current Linux distributions. This was presented as a BOF
> > ( https://www.linuxplumbersconf.org/2016/ocw/proposals/3555)  during
> > the Linux Plumbers Conference.
> >
> > it was a productive  discussion that let us share the problem that we
> > have in the current projects that we use every day to build a
> > distribution ( either in embedded as in a cloud base distribution).
> > The open source projects don't follow a standard output log format to
> > print the passing and failing tests that they run during packaging
> > time ( "make test" or "make check" )
> >
> > The Clear Linux project is using a simple Perl script that helps them
> > to count the number of passing and failing tests (which should be
> > trivial if could have a single standard output among all the projects,
> > but we don’t):
> >
> > https://github.com/clearlinux/autospec/blob/master/autospec/count.pl
> >
> > # perl count.pl <build.log>
> A few remarks about this.  This will be something of a stream of ideas, not
> very well organized.  I'd like to prevent requiring too many different
> language skills in Fuego.  In order to write a test for Fuego, we already
> require
> knowledge of shell script, python (for the benchmark parsers) and json
> formats
> (for the test specs and plans).  I'd be hesitant to adopt something in
> perl, but maybe
> there's a way to leverage the expertise embedded in your script.
> I'm not that fond of the idea of integrating all the parsers into a single
> program.
> I think it's conceptually simpler to have a parser per log file format.
> However,
> I haven't looked in detail at your parser, so I can't really comment on
> it's
> complexity.  I note that 0day has a parser per test (but I haven't checked
> to
> see if they re-use common parsers between tests.)  Possibly some
> combination
> of code-driven and data-driven parsers is best, but I don't have the
> experience
> you guys do with your parser.
> If I understood your presentation, you are currently parsing
> logs for thousands of packages. I thought you said that about half of the
> 20,000 packages in a distro have unit tests, and I thought you said that
> your parser was covering about half of those (so, about 5000 packages
> currently).
> And this is with 26 log formats parsed so far.
> I'm guessing that packages have a "long tail" of formats, with them getting
> weirder and weirder the farther out on the tail of formats you get.
> Please correct my numbers if I'm mistaken.
> > Examples of real packages build logs:
> >
> > https://kojipkgs.fedoraproject.org//packages/
> gcc/6.2.1/2.fc25/data/logs/x8
> > 6_64/build.log
> > https://kojipkgs.fedoraproject.org//packages/
> acl/2.2.52/11.fc24/data/logs/x
> > 86_64/build.log
> >
> > So far that simple (and not well engineered) parser has found 26
> > “standard” outputs ( and counting ) .
> This is actually remarkable, as Fuego is only handing the formats for the
> standalone tests we ship with Fuego.  As I stated in the BOF, we have two
> mechanisms, one for functional tests that uses shell, grep and diff, and
> one for benchmark tests that uses a very small python program that uses
> regexes.   So, currently we only have 50 tests covered, but many of these
> parsers use very simple one-line grep regexes.
> Neither of these Fuego log results parser methods supports tracking
> individual
> subtest results.
> > The script has the fail that it
> > does not recognize the name of the tests in order to detect
> > regressions. Maybe one test was passing in the previous release and in
> > the new one is failing, and then the number of failing tests remains
> > the same.
> This is a concern with the Fuego log parsing as well.
> I would like to modify Fuego's parser to not just parse out counts, but to
> also convert the results to something where individual sub-tests can be
> tracked over time.  Daniel Sangorrin's recent work converting the output
> of LTP into excel format might be one way to do this (although I'm not
> that comfortable with using a proprietary format - I would prefer CSV
> or json, but I think Daniel is going for ease of use first.)
> I need to do some more research, but I'm hoping that there are Jenkins
> plugins (maybe xUnit) that will provide tools to automatically handle
> visualization of test and sub-test results over time.  If so, I might
> try converting the Fuego parsers to product that format.
> > To be honest, before presenting at LPC I was very confident that this
> > script ( or another version of it , much smarter ) could be beginning
> > of the solution to the problem we have. However, during the discussion
> > at LPC I understand that this might be a huge effort (not sure if
> > bigger) in order to solve the nightmare we already have.
> So far, I think you're solving a bit different problem than Fuego is, and
> in one sense are
> much farther along than Fuego.  I'm hoping we can learn from your
> experience with this.
> I do think we share the goal of producing a standard, or at least a
> recommendation,
> for a common test log output format.  This would help the industry going
> forward.
> Even if individual tests don't produce the standard format, it will help
> 3rd parties
> write parsers that conform the test output to the format, as well as
> encourage the
> development of tools that utilize the format for visualization or
> regression checking.
> Do you feel confident enough to propose a format?  I don't at the moment.
> I'd like to survey the industry for 1) existing formats produced by tests
> (which you have good experience
> with, which is already maybe capture well by your perl script), and 2)
> existing tools
> that use common formats as input (e.g. the Jenkins xunit plugin).  From
> this I'd like
> to develop some ideas about the fields that are most commonly used, and a
> good language to
> express those fields. My preference would be JSON - I'm something of an
> XML naysayer, but
> I could be talked into YAML.  Under no circumstances do I want to invent a
> new language for
> this.
> > Tim Bird participates at the BOF and recommends me to send a mail to
> > the Fuego project team in order to look for more inputs and ideas bout
> > this topic.
> >
> > I really believe in the importance of attack this problem before we
> > have a bigger problem
> >
> > All feedback is more than welcome
> Here is how I propose moving forward on this.  I'd like to get a group
> together to study this
> issue.  I wrote down a list of people at LPC who seem to be working on
> test issues.  I'd like to
> do the following:
>  1) perform a survey of the areas I mentioned above
>  2) write up a draft spec
>  3) send it around for comments (to what individual and lists? is an open
> issue)
>  4) discuss it at a future face-to-face meeting (probably at ELC or maybe
> next year's plumbers)
>  5) publish it as a standard endorsed by the Linux Foundation
> Let me know what you think, and if you'd like to be involved.
> Thanks and regards,
>  -- Tim

- Guillermo Ponce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxfoundation.org/pipermail/fuego/attachments/20161108/f1c63cb3/attachment.html>

More information about the Fuego mailing list