[lsb-discuss] testing and more

[Robert Schweikert]

On Fri, 2006-12-22 at 20:01 -0500, Theodore Tso wrote:

> 
> Actually the first assumption of the users and the netscape developers
> was that the application worked just fine with the older version of
> glibc, and but when the system was upgraded to the newer glibc (the
> one that didn't allow you to fclose() a file handle twice), the
> application broke.  Since the last thing that changed on the system
> was the new glibc version, of *course* the application developers and
> the users blamed glibc, even though it was a bug in the Netscape
> browser.  True story!  And if you think about it, it makes perfect sense.

Sad story and I always thought this was the exception, maybe looking
somewhere else first is the norm.

>   
> 
> > Of course the tests have to be reviewed and the test submitter (by way
> > of the test) has to prove that the bug is not in the application.
> > Frankly I don't care if the app crashes. When the application crashes I
> > can fix it, when the X-server goes I just ruined my customers work day,
> > not very good advertisement for our product.
> 
> I guess I have the exact opposite set of priorities.  If the X server
> crashes, that's something which needs to be submitted to the upstream
> developers as part of a bug report, and it will actually probably get
> fixed pretty quickly.  Once it gets fixed, it probably belongs in the
> application's regression test suite, so that if there is ever a
> regression it will get caught quickly. 

Generally I agree. My concern is that many ISVs will not be interested
to follow though on this. Since they certify to the LSB they expect the
LSB system to work. If it breaks <sarcasm>it must be the LSB's fault,
just like it was glibc's fault when netscape cratered</sarcasm>. I am
not advocating that the LSB mailing list becomes the package bug filer
for all ISVs but the LSB web page or list must have a way to deal with
this in an ISV friendly manner.

>  In the case of a userspace
> program that can cause a kernel crash, the right place for that is the
> LTP (Linux Testing Project).  And, typically once the problem is
> fixed, it generally stays fixed (especially if the test is in merged
> into some regression test suite).  So the getting the test into the
> appropriate regression test suite is far more important than whether
> or not it is included in the LSB certification test suite.

The problem is getting the test into the right place, most ISVs will not
know how to go about this and where or how to submit bugs for all the
packages touched by the LSB.

> 
> My much bigger concern is for application programming mistakes that
> lead to a loss of portability.  Because these are problems that don't
> necessarily go away, since the number of X windows applications vastly
> outnumber the X server implementation.  So if even if we fix a bug
> where one application from one vendor

Winblows does this, or has done this in the past, big mess and big
mistake. We should not change system code just because an app developer
is using an interface in a way which per specification triggers
undefined behavior.

>  is making programming mistakes
> such the application works just fine with Metacity, but crashes and
> burns under Sawfish or Kwin, some other application programmer from
> some other vendor might come along make the same mistake next month.
> So how we help application programmers not make these sorts of
> portability mistakes is to me a much more important long-term
> strategic question, 

I would say these are example which should find their way to the LSB
developer network site in some way shape of form. Maybe we need to have
a "What not to do" section filled with examples showing how to break
portability. Each would of course have to be commented appropriately.

> as opposed to an X server bug that will probably
> be fixed within hours or days of bringing it to the attention of the
> correct X.org server developer.

Again the problem for most ISVs, who is the right person or the right
list. IMHO we will have a hard time getting ISVs to expend the effort to
dig many levels below the LSB.

> 
> 						- Ted
> 
-- 
Robert Schweikert                   MAY THE SOURCE BE WITH YOU
(Robert.Schweikert at abaqus.com)                 LINUX
ABAQUS Inc.
Phone : 401-727-4200
FAX : 401-727-4208