[lsb-discuss] testing and more
tytso at mit.edu
Fri Dec 22 07:58:22 PST 2006
On Fri, Dec 22, 2006 at 09:28:04AM -0500, Robert Schweikert wrote:
> > Actually, that may not follow. Just because a bug is triggered only
> > using LSB interfaces doesn't necessarily mean that the test case is
> > appropriate. The LSB doesn't necessarily specify all behaviors of a
> > particular interface, and it may be that the application may be buggy
> > because it makes assumption about the implementation that may not be
> > true across all distro's, or may change over time.
> I disagree. The value proposition of the LSB from an ISV point of view
> is clear. Abide by the programing rules, i.e. use LSB interfaces, and
> you application can be expected to work properly on 90%+ of LSB
> compliant distributions.
Sure, but it's "abide by the programming rules" which is the kicker,
and that means a lot more than just "use the LSB interfaces". You
have to use the LSB interfaces _correctly_, and that's a lot harder to
test for. In particular, when the LSB specifications that the
behaviour of an interface is unspecified under certain circumstances,
this means two things:
1) The LSB provider (i.e., the distribution) is free to do whatever
they want under those circumstances, including causing a core dump.
(I.e., "the behavior when passing a NULL pointer to strcpy is
2) The application MUST NOT pass any arguments to an LSB interface
that results in undefined behaviour. This includes calling fclose()
twice in a FILE * handle, passing in freed memory to strcpy (which may
or may not work), etc., all of which results in "undefined behavior".
So if an application calls fclose() twice, that's a BUG, and if the
netscape browser happens to work on Red Hat 2.0 but fail on Red Hat
3.0.3, (notice I said Red Hat, not RHEL; this is a very old story),
while the temptation would be to blame Red Hat and the newer version
of glibc, because it was now core dumping when a program called
fclose() on an already closed file handle, glibc() was perfectly
within its rights to change its behavior over what happens in this
undefined case (i.e., what happens when you pass an invalid file
handle to fclose --- with an already closed file handle counting as an
invalid file handle.)
So your assertion that "the result of the interface call is the same
on all distributions" can not and most not be absolute. It can not
cover behavior which has explicitly been declared to be undefined in
the specification. And therefore, just because a test case shows a
different result on one distribution versus another does not mean that
we can blindly accept it into the LSB test suite without doing some
> > The problem is that life is a lot more complicated than just
> > interfaces. Even if you are just talking about the base X libraries,
> > the need to obey such messaging protocols such as the ICCCM means that
> > just because you use the LSB-blessed interfaces is no guarantee of
> > portability if you screw up your ICCCM compliance.
> Then maybe ICCM must be part of the LSB???
Well, maybe. But suppose we reference the 75 ICCCM document and the
30 page Extended Window Manager Hints document. How do we assure
application compliance with these specifications? These
specifications don't actually define new interfaces, but a complex set
of messages that are sent to the Window Manager and other
participating X Windows System clients.
Maybe the answer is that applications shouldn't be using these
messaging protocols at all, but should be depending on higher level
libraries and toolkits to do the right thing. But then how do we
prevent an application writer from sending a ClientMessage event and
sending it via SendEvent to the window manager? And how do we test to
make sure the application isn't doing anything non-portable?
So yes, maybe it means we have to include a lot more stuff into the
LSB; but I'm concerned this won't be enough, and that there won't be a
substitute for testing an X application against multiple desktops and
multiple window managers. Otherwise we'll end up with "LSB certified"
applications that still only work on one application, and we won't
have delivered value to either the end-user or the ISV. The bottom
line is this is a very non-trivial problem.
More information about the lsb-discuss