[lsb-discuss] In the "new" world

chrubis at suse.cz chrubis at suse.cz
Tue Apr 1 21:30:07 UTC 2014

> > I do maintain Linux Test
> > Project that starts to get more attention lately and is actively used
> > for QA testing for several major distributions.
> Sorry I am unfamiliar with the tests that are part of LTP, do you care 
> to elaborate?

Well, it started as a collection of testsuites around the year 2000. I
cannot tell you much more because I've started to learn how to write
code at that time.

The official pages still says that the goal is:

"To validate reliability, robustness and stability of Linux"

> Do the tests in LTP test against a specification/man page etc.? Is there 
> a general focus of LTP or does LTP simply pick up tests that "come along"?

We have quite a lot of syscall/glibc libcall testcases that validates
that the interface works as described in manual. Then we have stress
tests and regression tests.

I cannot speak much for the past, however it was really non-uniform
collection of various tests and testsuites at the time I've joined. And
I've spend most of my time (more than three years now) making the tests
we allready had reliable. However these days there we have people
contributing new tests that are written solely for LTP.

>  From an LSB perspective any new tests that get developed would be tied 
> to a "Problem Statement" with a defined solution. The test should verify 
> that the solution is implemented and that it behaves as promised.

That goes for quite a lot of LTP tests as well.

LTP contains a well maintained fork of the Open Posix Testsuite that
aims for strict POSIX compatibility testing.

> >
> > So if there are tests that would match the purpose of LTP
> Please elaborate on "purpose of LTP" it's a bit unfair to make us do all 
> the look up and leg work to put the content into your e-mail.


I don't think that we (LTP community) have written down the exact
purpose, but let me try to formulate our goals. Generally that would be
making Linux better and the focus is on automated testing for kernel,
low-level libraries, system tools and daemons.

We are making sure that it's is easy to compile and run the LTP
testsuite. And we roughly have:

* syscalls/glibc libcalls conformance tests + few regression tests
* Open Posix Testsuite
* Relatime Testsuite for realtime kernel patchset
* Some network related tests, fs related tests, stress tests
* A few kernel device driver tests
* A few tests for modern kernel features such as:
  containers, controllers, namespaces, numa...
* A few tests for shell utilities and daemons

Not everything works or is reliable at the moment but we are getting
better with each release.

> > and if the
> > tests were isolated to run separately we may add them to LTP too.
> Thanks having more tests and having them easily accessible to 
> distributions is certainly helpful to the overall quality of Linux 
> distributions. LSB's primary focus is cross distribution compatibility, 
> even in the new world. One could argue that distributions passing LTP 
> tests are compatible, but I am not certain if this is necessary and 
> sufficient.

That depends, in contrast with LSB LTP tries to cope with older
glibc/kernel and with disabled kernel functionality and such situation
is not reported as a failure but rather as "Test not suitable".

> > Some
> > more work would be needed to make this happen. However if the tests
> > would be included in LTP they will get into QA deparments in next LTP
> > release and will be executed as a part of the distribution QA process.
> Which distributions currently use LTP? Based on your statement and 
> e-mail address I must assume that LTP is fully integrated into openQA. 
> Where else is it used?

I'm not really sure of openQA, all I know that is that it was on TODO
but I wasn't the one working on the task...

People from Fujitsu, RedHat and Oracle are quite active in contributing
to upstream. Then we have some patches from Linaro, Twitter, Intel and
from a bunch of embedded companies. And I suspect that there are much
more that simply haven't send anything upstream.

A few kernel commits mentions LTP testcases which implies that there are
kernel developers that use LTP.

Not to mention that we use it quite extensively internally in SUSE.

Cyril Hrubis
chrubis at suse.cz

More information about the lsb-discuss mailing list