[Fuego] LPC Increase Test Coverage in a Linux-based OS

Victor Rodriguez vm.rod25 at gmail.com
Sat Nov 5 17:15:05 UTC 2016


Hi Fuego team.

This week I presented a case of study for the problem of lack of test
log output standardization in the majority of packages that are used
to build the current Linux distributions. This was presented as a BOF
( https://www.linuxplumbersconf.org/2016/ocw/proposals/3555)  during
the Linux Plumbers Conference.

it was a productive  discussion that let us share the problem that we
have in the current projects that we use every day to build a
distribution ( either in embedded as in a cloud base distribution).
The open source projects don't follow a standard output log format to
print the passing and failing tests that they run during packaging
time ( "make test" or "make check" )

The Clear Linux project is using a simple Perl script that helps them
to count the number of passing and failing tests (which should be
trivial if could have a single standard output among all the projects,
but we don’t):

https://github.com/clearlinux/autospec/blob/master/autospec/count.pl

# perl count.pl <build.log>

Examples of real packages build logs:

https://kojipkgs.fedoraproject.org//packages/gcc/6.2.1/2.fc25/data/logs/x86_64/build.log
https://kojipkgs.fedoraproject.org//packages/acl/2.2.52/11.fc24/data/logs/x86_64/build.log

So far that simple (and not well engineered) parser has found 26
“standard” outputs ( and counting ) .  The script has the fail that it
does not recognize the name of the tests in order to detect
regressions. Maybe one test was passing in the previous release and in
the new one is failing, and then the number of failing tests remains
the same.

To be honest, before presenting at LPC I was very confident that this
script ( or another version of it , much smarter ) could be beginning
of the solution to the problem we have. However, during the discussion
at LPC I understand that this might be a huge effort (not sure if
bigger) in order to solve the nightmare we already have.

Tim Bird participates at the BOF and recommends me to send a mail to
the Fuego project team in order to look for more inputs and ideas bout
this topic.

I really believe in the importance of attack this problem before we
have a bigger problem

All feedback is more than welcome

Regards

Victor Rodriguez

[presentation slides] :
https://drive.google.com/open?id=0B7iKrGdVkDhIcVpncUdGTGhEQTQ
[BOF notes] : https://drive.google.com/open?id=1lOPXQcrhL4AoOBSDnwUlJAKIXsReU8OqP82usZn-DCo


More information about the Fuego mailing list