[lsb-discuss] testing and more

Robert Schweikert robert.schweikert at abaqus.com
Fri Dec 22 08:40:40 PST 2006

On Fri, 2006-12-22 at 10:58 -0500, Theodore Tso wrote:

> Sure, but it's "abide by the programming rules" which is the kicker,
> and that means a lot more than just "use the LSB interfaces".  You
> have to use the LSB interfaces _correctly_, and that's a lot harder to
> test for.  In particular, when the LSB specifications that the
> behaviour of an interface is unspecified under certain circumstances,
> this means two things:
> 1) The LSB provider (i.e., the distribution) is free to do whatever
> they want under those circumstances, including causing a core dump.
> (I.e., "the behavior when passing a NULL pointer to strcpy is
> unspecified".)
> 2) The application MUST NOT pass any arguments to an LSB interface
> that results in undefined behaviour.  This includes calling fclose()
> twice in a FILE * handle, passing in freed memory to strcpy (which may
> or may not work), etc., all of which results in "undefined behavior".
> So if an application calls fclose() twice, that's a BUG, and if the
> netscape browser happens to work on Red Hat 2.0 but fail on Red Hat
> 3.0.3, (notice I said Red Hat, not RHEL; this is a very old story),
> while the temptation would be to blame Red Hat and the newer version
> of glibc, because it was now core dumping when a program called
> fclose() on an already closed file handle, glibc() was perfectly
> within its rights to change its behavior over what happens in this
> undefined case (i.e., what happens when you pass an invalid file
> handle to fclose --- with an already closed file handle counting as an
> invalid file handle.)

Yes, but in both cases the app wil crash, thus the first assumption of
any application developer has to be that the problem lies with them. I
am not arguing that everything has to be defined, that's impossible.
Undefined behavior is just that and this has been present for ever. If a
developer uses an interface in a manner which is said to be undefined
and then depends on this behavior the developer is certainly not writing
portable code. (I can without much effort think of a few words I would
call such a developer)

> So your assertion that "the result of the interface call is the same
> on all distributions" can not and most not be absolute.  It can not
> cover behavior which has explicitly been declared to be undefined in
> the specification.  And therefore, just because a test case shows a
> different result on one distribution versus another does not mean that
> we can blindly accept it into the LSB test suite without doing some
> serious analysis.

Of course the tests have to be reviewed and the test submitter (by way
of the test) has to prove that the bug is not in the application.
Frankly I don't care if the app crashes. When the application crashes I
can fix it, when the X-server goes I just ruined my customers work day,
not very good advertisement for our product.

The LSB or upstream tests must be sufficient to avoid this situation.
Once again we cannot future proof our tests and there are new bugs to be
discovered with every version. However, we do have the responsibility to
address issues being uncovered by various people and for this I believ
the LSB should provide a way for ISVs to submit test cases that show
these types of issues. Most ISVs will not care if the test is actually
incorporated into the distribution LSB compliance test or into an
upstream test suite. What ISVs do care about is that the issue gets
fixed and does not show up again.

We must differentiate between application crashes and system crashes.
When the application crashes the first responsibility lies with the
application developer. When the system or part of the system (such as
the X server) decides to take an ungraceful exit the application
developer has the responsibility in trying to find a work around for
their customer but in the end the system developer must fix the code.
The latter part is what the stress testing is all about. Try and catch
as much of the system problems while calling LSB interfaces as possible.

> > > The problem is that life is a lot more complicated than just
> > > interfaces.  Even if you are just talking about the base X libraries,
> > > the need to obey such messaging protocols such as the ICCCM means that
> > > just because you use the LSB-blessed interfaces is no guarantee of
> > > portability if you screw up your ICCCM compliance. 
> > 
> > Then maybe ICCM must be part of the LSB???
> Well, maybe.  But suppose we reference the 75 ICCCM document and the
> 30 page Extended Window Manager Hints document.  How do we assure
> application compliance with these specifications?  These
> specifications don't actually define new interfaces, but a complex set
> of messages that are sent to the Window Manager and other
> participating X Windows System clients. 
> Maybe the answer is that applications shouldn't be using these
> messaging protocols at all, but should be depending on higher level
> libraries and toolkits to do the right thing.  But then how do we
> prevent an application writer from sending a ClientMessage event and
> sending it via SendEvent to the window manager?  And how do we test to
> make sure the application isn't doing anything non-portable?

Don't know. However, at some point the application developer must take
responsibility for their actions. I don't know how to define the line or
even how to shed light onto the gray area. At some point in the chain
there has to be a level of reasonable and open communication where
developers, testers, and other stake holders meet to find a necessary
and sufficient specification level which meets the need of most or all

> So yes, maybe it means we have to include a lot more stuff into the
> LSB; but I'm concerned this won't be enough, and that there won't be a
> substitute for testing an X application against multiple desktops and
> multiple window managers.  Otherwise we'll end up with "LSB certified"
> applications that still only work on one application, and we won't
> have delivered value to either the end-user or the ISV.  The bottom
> line is this is a very non-trivial problem.

We all agree on the bottom line, no question about it.

> 						- Ted
Robert Schweikert                   MAY THE SOURCE BE WITH YOU
(Robert.Schweikert at abaqus.com)                 LINUX
Phone : 401-276-7190
FAX : 401-276-4408

More information about the lsb-discuss mailing list