[lsb-discuss] testing and more

Robert Schweikert robert.schweikert at abaqus.com
Wed Dec 20 10:16:39 PST 2006


Dependent on the point of view one takes this could certainly be viewed
as outside the scope of the LSB, yet it is vital for ISVs and the value
promise of the LSB.

I propose that in some way we have to go beyond simple interface testing
and tests need to be added which stress an implementation. As an example
one could think of testing frpintf with the standard "Hello World"
string. This test will definitely test the interface, i.e. the interface
is there and fprintf prints what is expected. However, in the live of an
application developers print more than "Hello World", what if the
interface fails when the string being printed is very large, I realize
there has to be some kind of cut off point but would it not be the
responsibility of an LSB compliant distribution to have interfaces work
in extreme cases? Shouldn't the LSB test suite make it easy to test
these extreme cases?

You can certainly imagine that this post was not triggered by fprintf
falling over, so here is the real world case. There is one particular
bug in the X.org code which can be triggered by only using LSB
interfaces, one just has to poke a bit. This bug unfortunately shows up
in at least one certified distribution. Now one can with a certified
application make the X-server on the certified distribution crash.

Without thinking too hard I can certainly come up with the first
response one might think of.

- It's a bug, we cannot test the whole system and guarantee no crashes. 

Agreed, it is not possible to test everything for all possible use
cases. The complexity is too great, we'd need more tests than there are
lines of code for implementation.

Yet, the compile once run anywhere (with low incremental testing
overhead) goal has just taken a huge beating. Now I am back to
qualifying everything on every distribution which clearly does not go
along with the spirit of the LSB.

So here we are, clearly we cannot test every interface fro every use
case, but can we define rudimentary tests (such as "Hello World" for
fprintf) and stress tests (such as allocating 10 GB of memory on a 64
bit machine for malloc). Where is the pain threshold for the
distributions? While it may not be a problem for the larger
distributions to run a test that allocates 10 GB of memory where does
this leave the smaller distributions which do not have access to this
type of hardware but still want to be LSB certified? Can the FSG/LSB
work group get funding from AMD, Dell, HP, IBM, Intel and other hardware
vendors to set up a certification center for distributions. The idea is
of course to allow small distributions access to hardware where a 10 GB
allocation test can be performed.

Last but not least, and I know we hit on this topic many times before,
should we not have a frame work that allows a relatively open and easy
contribution of tests to the LSB?

Lets take the X.org bug as an example for this one. Since this bug
effects a large number of people someone is going to come up with a
hopefully relatively simple test case. Since the bug is triggered by
using only LSB interfaces the test case should become part of the LSB
test suite (I understand that we may not be able to add every test for
every LSB component, but lets just continue this train of thought for
now). The submitter of the test case has already done a lot of work,
thus he should not necessarily be required to deal with integration and
documentation. As a submitter I can imagine just giving a basic
description of the problem being tested and a version of the code
showing the problem (if this is a bug fix verification test, in this
case the X.org version). Preferably as a test submitter in I'd just go
to a web page, enter a brief description of the test, a brief
description of the problem observed (in this example an X server crash)
agree to hand copyright of the test code to the FSG, attach the code for
the test case and press submit. As a test submitter I have to of course
be able to be contacted for questions, thus an e-mail address as
submitter information is a must. Done.

Someone on the other side of the web page has work to do now, verify the
problem, check that the code is reasonably commented and that the
resulting code is LSB compliant. Then the test gets integrated, the
submitter gets a thank you, every one benefits and the LSB test suite
just got better.

Yes, there are many questions to be answered, we have to start the
discussion somewhere....

Thanks,
Robert
-- 
Robert Schweikert                   MAY THE SOURCE BE WITH YOU
(Robert.Schweikert at abaqus.com)                 LINUX
ABAQUS Inc.
Phone : 401-276-7190
FAX : 401-276-4408





More information about the lsb-discuss mailing list