[lsb-discuss] testing and more

Robert Schweikert robert.schweikert at abaqus.com
Fri Dec 22 06:28:04 PST 2006


On Fri, 2006-12-22 at 00:25 -0500, Theodore Tso wrote:
> On Wed, Dec 20, 2006 at 01:16:39PM -0500, Robert Schweikert wrote:
> > Lets take the X.org bug as an example for this one. Since this bug
> > effects a large number of people someone is going to come up with a
> > hopefully relatively simple test case. Since the bug is triggered by
> > using only LSB interfaces the test case should become part of the LSB
> > test suite (I understand that we may not be able to add every test for
> > every LSB component, but lets just continue this train of thought for
> > now). 
> 
> Actually, that may not follow.  Just because a bug is triggered only
> using LSB interfaces doesn't necessarily mean that the test case is
> appropriate.  The LSB doesn't necessarily specify all behaviors of a
> particular interface, and it may be that the application may be buggy
> because it makes assumption about the implementation that may not be
> true across all distro's, or may change over time.

I disagree. The value proposition of the LSB from an ISV point of view
is clear. Abide by the programing rules, i.e. use LSB interfaces, and
you application can be expected to work properly on 90%+ of LSB
compliant distributions. 

Without this, and we heard about this from SAP as well at the f2f in
Berlin Linux will not be a supportable platform for ISVs. This is not to
say that all ISVs will jump ship but many ISVs will not port without the
LSB. IMHO the LSB is the single most important effort of the open source
community to get ISVs support on Linux.

About the details, as an ISV I do not care about the underlying
implementation. If distributions want to differentiate themselves by
having faster implementations of one or another interface be my guest.
What an ISV does care about is that the result of the interface call is
the same on all distributions and that I get the same behavior whether I
call the interface once or a million times in a row. Which unfortunately
brings me back to stress testing and the X.org bug. We did produce a
test case (will post in another e-mail) and this test shows that it is
the repetitive nature of the use of the interfaces which crater the X
server. Bummer.

> 
> Or it could be argued that the specification is buggy, because it is
> silent where it needs to say more.  So the test case may be right, but
> it might not be something we can accept without also making spec
> changes.

I am not arguing about bugs. The simple fact of software life is that
every code has bugs. We all try to find the bugs through various test
strategies and if our customers would run our test cases they would
never find a bug and this applies to most software. When I use the
X-server and I only run the test applications and the development tests
I am sure it will not crash. However, as a software provider we all know
that users use the software in a way they see fit to get the job done
and it is our responsibility to address issues they find. If a user
finds a bug lets fix it, add a test case to make sure it doesn't come
back and move on. This was the spirit of my original post. 

When more ISVs certify their applications to the LSB more bugs will be
exposed and there must be a willingness by the LSB community to accept
tests. Most ISVs will not want to make the effort to go a chase down the
individual mailing list and post a bug etc. and a test case. I see the
reasoning more along relatively straight lines as follows:

I have an LSB certified application, when I use and LSB interface the
application crashes and I can prove it is not an application bug. If I
am a somewhat motivated ISV I will go and contact the distribution
vendor (for the distribution where I found the bug) first. If I am less
motivated I might just post a message to this list and say interface X
is broken, what's going on. Now help is needed, and if we have a way to
incorporate a test case to avoid the bug in the future I think the ISV
will be a much happier user of the LSB as compared to the case where the
ISV just gets send to some mailing list of some library development
community.

Thus from my perspective the LSB community is taking on some
responsibility for more than just the interfaces. This responsibility is
implied but it is this implication which makes the LSB so very important
and interesting to an ISV.

> 
> This brings up a related problem that has been gnawing at me for a
> while now, so I'll use this message as an excuse to write my
> concern/rant:
> 
> One of the things which worries me as we start standardizing more
> libraries in order to support X/Desktop applications is that simply
> worrying about interface spceifications may not be enough any more.

Yes, it is in the end the behavior we are after. When I use LSB
interfaces I shouldn't be able to crash the system or parts of it. If I
crash my app, well then I've got to go fix my own bugs before I go
complaining.

> For example, there is a particular application (name omitted to
> protect the guilty, but some folks may be able to guess) which I am
> forced to use every day at work.  Unfortunately, this application is
> incredibly buggy, since the developers only wrote and tested it on
> RHEL 3(!), and it makes assumptions about how the window manager
> works, such that it either crashes or runs at varying levels of
> usefulness (from minor annoyances to not working at all) if you use
> any window manager other than metacity.  It also makes various
> assumptions about how the CUPS libraries work such that if you run it
> on anything other than RHEL 3 and your printer configuration changes,
> it will crash on you.  The amazing thing is that people actually pay
> serious amounts of money for this propietary program....

This appears to be a perfect example where the LSB would help. When CUPS
becomes part of the LSB, or some other printing interface the
assumptions made by the application provider go away and you as the user
get the freedom of running the application on your distribution of
choice, provided that distributions is LSB compliant.

Not to take this ISVs side, it is understandable in the absence of the
LSB that one writes or ports an application to work on a specific
distribution and that everyone else is left out in the cold. Get the
vendor to see the opportunities LSB provides and the problem will most
likely go away.

> 
> Right now the only way I can get it to work on a modern distribution
> is because a colleague was able to figure out how to write a set of
> LD_PRELOAD hacks to intercept the application's buggy use of library
> functions (and I've learned to live within the constraints of metacity
> and dropped my custom sawfish customizations I had written in Lisp).
> Most unfortunately, I don't have source access so I can't fix the bugs
> directly.
> 
> I can easily imagine such a horribly buggy, non-portable application,
> obeying every line of the LSB specification, but still not working on
> anything but the original distribution it was written to run upon.

If this is the case then I'd say the LSB is insufficient in it's
requirements and descriptions.

> The problem is that life is a lot more complicated than just
> interfaces.  Even if you are just talking about the base X libraries,
> the need to obey such messaging protocols such as the ICCCM means that
> just because you use the LSB-blessed interfaces is no guarantee of
> portability if you screw up your ICCCM compliance. 

Then maybe ICCM must be part of the LSB???

>  And GNOME and hal
> makes this even more complicated, because their rules might not
> necessarily be well documented yet.
> 
> How do we fix this?  I'm not sure, but it's something for us to think
> about.  As we start standardizing libraries that have more stateful,
> object-oriented, or messaging-based interfaces, it seems pretty clear
> to me that simply standardizing C/C++ function signatures is not going
> to be enough.
> 
> And even once we create specifications, writing test cases to make
> sure that the libraries are and the applications under a certification
> test are correctly behaving is yet another challenge....
> 
> 						- Ted
> 
-- 
Robert Schweikert                   MAY THE SOURCE BE WITH YOU
(Robert.Schweikert at abaqus.com)                 LINUX
ABAQUS Inc.
Phone : 401-276-7190
FAX : 401-276-4408





More information about the lsb-discuss mailing list