[Fuego] LPC Increase Test Coverage in a Linux-based OS

Victor Rodriguez vm.rod25 at gmail.com
Wed Nov 9 20:07:53 UTC 2016


Hi team

Sorry for the delay

On Wed, Nov 9, 2016 at 1:04 PM, Guillermo Adrian Ponce Castañeda
<ga.poncec at gmail.com> wrote:
> Hi Tim and Victor,
>
> Ok, if we would like to talk about the tools that are used to visualize the
> results, I will try to describe the way it works without incurring in
> revealing proprietary information, since that code is not open source yet.
>
> There is an initial script that gets the active packages names for the linux
> distro and gets the build logs for each one.
> That script calls count.pl script and attaches the package name to the
> results of the count.pl script. That string will result like
> '<package>,100,80,20,0,0', if I remind correctly, and each package output
> will be appended to big csv file with headers.
> After we have that csv file we pass it to another script that will create
> some graphs.
>
> So basically it is all CSV and home made tools to analyze them, I think it
> can be automated, but I guess Victor can give us more details on the current
> process on that matter if any.
>
> Thanks and Regards.
> - Guillermo Ponce
>
>
> On Tue, Nov 8, 2016 at 6:21 PM, Bird, Timothy <Tim.Bird at am.sony.com> wrote:
>>
>>
>>
>> > -----Original Message-----
>> > From: Guillermo Adrian Ponce Castañeda on Tuesday, November 08, 2016
>> > 11:38 AM
>> >
>> > I am a co-author of this code and I must confess that it was more or
>> > less my
>> > fault that it was made on Perl.
>>
>> No blame intended. :-)
>>
>> >
>> > Regarding how many logs the program analyzes, I think it is nowhere near
>> > 5000, it is much less, but taking in count that some logs are similar I
>> > think it is
>> > possible that some logs that haven't been tested are going to work, but
>> > who
>> > knows :).
>> >
>> >
>> > And about the output file, right now it delivers a comma separated list
>> > of
>> > numbers, without headers, this is because this code is part of a  bigger
>> > tool, I
>> > think that code is not open source yet, but that doesn't matter I guess,
>> > the
>> > thing here is that I think the output could be changed into a json like
>> > you
>> > suggested and i can try to translate the code from Perl to Python, still
>> > not
>> > sure how long it's gonna take, but I can sure try.
>>
>> Well, don't do any re-writing just yet.  I think we need to consider the
>> output format some more, and decide whether it makes sense to have a
>> single vs. multiple parsers first.
>>
>> An important issue here is scalability of the project, and making it easy
>> to allow (and incentivize) other developers to create and maintain
>> parsers for the log files.  Or, to help encourage people to use a common
>> format either initially, or by conversion from their current log format.
>> The only way to scale this is by having 3rd parties adopt the format, and
>> be willing to maintain compatibility with it over time.
>>

I am more than happy to send patches to upstream once we could get an
standard output

Memo , I will try to count how many packages are using each parser and
get the most used one

>> I think it's important to consider what will motivate people to adopt a
>> common
>> log format.  They either need to 1) write a parser for their current
>> format, or
>> 2) identify an existing parser which is close enough and modify it to
>> support their format, or 3) convert their test output directly to the
>> desired
>> format.  This will be some amount of work whichever route people take.
>>
>> I think what will be of value is having tools that read and process the
>> format,
>> and provide utility to those who use the format for output.  So I want to
>> do a bit
>> of a survey on what tools (visualizers, aggregators, automated processors,
>> notifiers, etc.) might be useful to different developer groups, and make
>> sure
>> the format is something that can be used by existing tools or by
>> envisioned
>> future tools, that would be valuable to community members.
>>

I think this is a pretty good standard if we are looking for compatibility

https://testanything.org/

Jenkins also have plugins for this


>> In more high-level terms, we should trying to create a double-sided
>> network effect,
>> where use  (output) of the format drives tools creation, and tools usage
>> of the format (input) drives format popularity.
>>
>> Can you describe a bit more what tools, if any, you use to view the
>> results,
>> or any other processing systems that the results are used with?  If you
>> are reviewing
>> results manually, are there steps you are doing now by hand that you'd
>> like to
>> do automatically in the future, that a common format would help you with?
>>
>> I'll go first - Fuego is currently just using the standard Jenkins
>> "weather" report
>> and 'list of recent overall pass/failure' for each test. So we don't have
>> anything
>> visualizing the results of sub-tests, or even displaying the counts for
>> each test run, at the moment.
>> Daniel Sangorrin has just recently proposed a facility to put LTP results
>> into spreadsheet format,
>> to allow visualizing test results over time via spreadsheet tools.  I'd
>> like to add better
>> sub-test visualization in the future, but that's lower on our priority
>> list at the moment.
>>
>> Also in the future, we'd like to do test results aggregation, to allow for
>> data mining
>> of results from tests on different hardware platforms and embedded
>> distributions.
>> This will require that the parsed log output be machine-readable, and
>> consistent.

Agree , this will be for multiple patforms


Thanks a lot for follow up this Tim , this is a good topic that if we
coudl solve , we will help a lot to the Linux comunity

Do you consider that we should do some survay in other OS
distributions ? ( debian , fedora , suse ... maybe , even yocto )

Where else do you think we should rise this problem ?

Do you think we should wirte an small article about it in LWN ( or
something similar ) to get atraction ( a call of action )  ?

The idea of a linux testing summit is pretty much amazing , count on
me for help with that :)

Regards

Victor Rodriguez

>>  -- Tim
>>
>> > On Mon, Nov 7, 2016 at 6:26 PM, Bird, Timothy <Tim.Bird at am.sony.com
>> > <mailto:Tim.Bird at am.sony.com> > wrote:
>> >
>> >
>> >       Victor,
>> >
>> >       Thanks for raising this topic.  I think it's an important one.  I
>> > have
>> > some comments below, inline.
>> >
>> >       > -----Original Message-----
>> >       > From: Victor Rodriguez on Saturday, November 05, 2016 10:15 AM
>> >       >
>> >       > This week I presented a case of study for the problem of lack of
>> > test
>> >       > log output standardization in the majority of packages that are
>> > used
>> >       > to build the current Linux distributions. This was presented as
>> > a BOF
>> >       > ( https://www.linuxplumbersconf.org/2016/ocw/proposals/3555
>> > <https://www.linuxplumbersconf.org/2016/ocw/proposals/3555> )  during
>> >       > the Linux Plumbers Conference.
>> >       >
>> >       > it was a productive  discussion that let us share the problem
>> > that
>> > we
>> >       > have in the current projects that we use every day to build a
>> >       > distribution ( either in embedded as in a cloud base
>> > distribution).
>> >       > The open source projects don't follow a standard output log
>> > format
>> > to
>> >       > print the passing and failing tests that they run during
>> > packaging
>> >       > time ( "make test" or "make check" )
>> >       >
>> >       > The Clear Linux project is using a simple Perl script that helps
>> > them
>> >       > to count the number of passing and failing tests (which should
>> > be
>> >       > trivial if could have a single standard output among all the
>> > projects,
>> >       > but we don’t):
>> >       >
>> >       >
>> > https://github.com/clearlinux/autospec/blob/master/autospec/count.pl
>> > <https://github.com/clearlinux/autospec/blob/master/autospec/count.pl>
>> >       >
>> >       > # perl count.pl <http://count.pl>  <build.log>
>> >
>> >       A few remarks about this.  This will be something of a stream of
>> > ideas, not
>> >       very well organized.  I'd like to prevent requiring too many
>> > different
>> >       language skills in Fuego.  In order to write a test for Fuego, we
>> > already require
>> >       knowledge of shell script, python (for the benchmark parsers) and
>> > json formats
>> >       (for the test specs and plans).  I'd be hesitant to adopt
>> > something in
>> > perl, but maybe
>> >       there's a way to leverage the expertise embedded in your script.
>> >
>> >       I'm not that fond of the idea of integrating all the parsers into
>> > a single
>> > program.
>> >       I think it's conceptually simpler to have a parser per log file
>> > format.
>> > However,
>> >       I haven't looked in detail at your parser, so I can't really
>> > comment on
>> > it's
>> >       complexity.  I note that 0day has a parser per test (but I haven't
>> > checked to
>> >       see if they re-use common parsers between tests.)  Possibly some
>> > combination
>> >       of code-driven and data-driven parsers is best, but I don't have
>> > the
>> > experience
>> >       you guys do with your parser.
>> >
>> >       If I understood your presentation, you are currently parsing
>> >       logs for thousands of packages. I thought you said that about half
>> > of
>> > the
>> >       20,000 packages in a distro have unit tests, and I thought you
>> > said
>> > that
>> >       your parser was covering about half of those (so, about 5000
>> > packages currently).
>> >       And this is with 26 log formats parsed so far.
>> >
>> >       I'm guessing that packages have a "long tail" of formats, with
>> > them
>> > getting
>> >       weirder and weirder the farther out on the tail of formats you
>> > get.
>> >
>> >       Please correct my numbers if I'm mistaken.
>> >
>> >       > Examples of real packages build logs:
>> >       >
>> >       >
>> >
>> > https://kojipkgs.fedoraproject.org//packages/gcc/6.2.1/2.fc25/data/logs/x8
>> >
>> > <https://kojipkgs.fedoraproject.org//packages/gcc/6.2.1/2.fc25/data/logs/x
>> > 8>
>> >       > 6_64/build.log
>> >       >
>> >
>> > https://kojipkgs.fedoraproject.org//packages/acl/2.2.52/11.fc24/data/logs/x
>> >
>> > <https://kojipkgs.fedoraproject.org//packages/acl/2.2.52/11.fc24/data/logs/
>> > x>
>> >       > 86_64/build.log
>> >       >
>> >       > So far that simple (and not well engineered) parser has found 26
>> >       > “standard” outputs ( and counting ) .
>> >
>> >       This is actually remarkable, as Fuego is only handing the formats
>> > for
>> > the
>> >       standalone tests we ship with Fuego.  As I stated in the BOF, we
>> > have
>> > two
>> >       mechanisms, one for functional tests that uses shell, grep and
>> > diff,
>> > and
>> >       one for benchmark tests that uses a very small python program that
>> > uses
>> >       regexes.   So, currently we only have 50 tests covered, but many
>> > of
>> > these
>> >       parsers use very simple one-line grep regexes.
>> >
>> >       Neither of these Fuego log results parser methods supports
>> > tracking
>> > individual
>> >       subtest results.
>> >
>> >       > The script has the fail that it
>> >       > does not recognize the name of the tests in order to detect
>> >       > regressions. Maybe one test was passing in the previous release
>> > and in
>> >       > the new one is failing, and then the number of failing tests
>> > remains
>> >       > the same.
>> >
>> >       This is a concern with the Fuego log parsing as well.
>> >
>> >       I would like to modify Fuego's parser to not just parse out
>> > counts, but
>> > to
>> >       also convert the results to something where individual sub-tests
>> > can
>> > be
>> >       tracked over time.  Daniel Sangorrin's recent work converting the
>> > output
>> >       of LTP into excel format might be one way to do this (although I'm
>> > not
>> >       that comfortable with using a proprietary format - I would prefer
>> > CSV
>> >       or json, but I think Daniel is going for ease of use first.)
>> >
>> >       I need to do some more research, but I'm hoping that there are
>> > Jenkins
>> >       plugins (maybe xUnit) that will provide tools to automatically
>> > handle
>> >       visualization of test and sub-test results over time.  If so, I
>> > might
>> >       try converting the Fuego parsers to product that format.
>> >
>> >       > To be honest, before presenting at LPC I was very confident that
>> > this
>> >       > script ( or another version of it , much smarter ) could be
>> > beginning
>> >       > of the solution to the problem we have. However, during the
>> > discussion
>> >       > at LPC I understand that this might be a huge effort (not sure
>> > if
>> >       > bigger) in order to solve the nightmare we already have.
>> >
>> >       So far, I think you're solving a bit different problem than Fuego
>> > is,
>> > and in one sense are
>> >       much farther along than Fuego.  I'm hoping we can learn from your
>> >       experience with this.
>> >
>> >       I do think we share the goal of producing a standard, or at least
>> > a
>> > recommendation,
>> >       for a common test log output format.  This would help the industry
>> > going forward.
>> >       Even if individual tests don't produce the standard format, it
>> > will help
>> > 3rd parties
>> >       write parsers that conform the test output to the format, as well
>> > as
>> > encourage the
>> >       development of tools that utilize the format for visualization or
>> > regression checking.
>> >
>> >       Do you feel confident enough to propose a format?  I don't at the
>> > moment.
>> >       I'd like to survey the industry for 1) existing formats produced
>> > by
>> > tests (which you have good experience
>> >       with, which is already maybe capture well by your perl script),
>> > and 2)
>> > existing tools
>> >       that use common formats as input (e.g. the Jenkins xunit plugin).
>> > From this I'd like
>> >       to develop some ideas about the fields that are most commonly
>> > used, and a good language to
>> >       express those fields. My preference would be JSON - I'm something
>> > of an XML naysayer, but
>> >       I could be talked into YAML.  Under no circumstances do I want to
>> > invent a new language for
>> >       this.
>> >
>> >       > Tim Bird participates at the BOF and recommends me to send a
>> > mail
>> > to
>> >       > the Fuego project team in order to look for more inputs and
>> > ideas
>> > bout
>> >       > this topic.
>> >       >
>> >       > I really believe in the importance of attack this problem before
>> > we
>> >       > have a bigger problem
>> >       >
>> >       > All feedback is more than welcome
>> >
>> >       Here is how I propose moving forward on this.  I'd like to get a
>> > group
>> > together to study this
>> >       issue.  I wrote down a list of people at LPC who seem to be
>> > working
>> > on test issues.  I'd like to
>> >       do the following:
>> >        1) perform a survey of the areas I mentioned above
>> >        2) write up a draft spec
>> >        3) send it around for comments (to what individual and lists? is
>> > an
>> > open issue)
>> >        4) discuss it at a future face-to-face meeting (probably at ELC
>> > or
>> > maybe next year's plumbers)
>> >        5) publish it as a standard endorsed by the Linux Foundation
>> >
>> >       Let me know what you think, and if you'd like to be involved.
>> >
>> >       Thanks and regards,
>> >        -- Tim
>> >
>> >
>> >
>> >
>> >
>> >
>> > --
>> >
>> > - Guillermo Ponce
>
>
>
>
> --
> - Guillermo Ponce


More information about the Fuego mailing list