[lsb-discuss] unofficial LSB conference call minutes for 23 Apr 2014, post Collab 'New LSB' wiki point in time drop

chrubis at suse.cz chrubis at suse.cz
Thu Apr 24 10:52:11 UTC 2014

> all of which, of course, TET has also, without actually being
> particularly top-heavy. Examples: you report failure (or success) by
> giving a single value to tet_result(), and emit messages by calling
> tet_printf() which has syntax just like printf, not exactly a heavy
> burden. It also has many useful macros, cleanup startup/callback,
> child/parent sync (including the very straightforward tet_fork() and
> tet_wait()), and so on. I could just as easily ask why LTS didn't adopt
> TET :)  Everybody has slightly different needs, and so there are many
> test frameworks in the world.
> For me (unable to attend the summit and so missing plenty of discussion
> I suspect), the problem has always been that ease of use when running
> tens of thousands of tests demands a framework so you can collect
> information and report on it in a way that doesn't drive you insane.
> And then, when something fails, and you need to ask a specialist in that
> area, they want a standalone testcase they can not just run, but also
> quickly tweak and recompile, without the fuss of setting up that
> framework or having to deal with dependencies on headers/libraries that
> belong to it.  SO you end up having to strip out all the stuff that
> takes advantage of the framework.  And that costs /somebody/ time.  I'm
> sure that's not vastly different for LTP either (which is a great
> project I've followed for years, there is zero criticism here).

LTP is more testcase centric, all of the testcases are just executable
files (either binary or shell) the testcase output is printed to stdout
by default, test overall result is propagated via exit value, etc. To
compile a testcase there is no need to install anything besides basic C
devel libraries and testcases can be easily tweaked, recompiled and
executed directly from git checkout. A few of them needs to add PWD to
PATH in order to run subexecutables and some works with block device
that needs to be prepared beforehand and passed in parameters but that
is all it takes to execute the testcase.

Then there is a testdriver which is completly separate entity. The
current one we have is a bit messy and outdated and I have plans to
replace it in future with something very similar with cleaner codebase
and a few more features. However all the testdriver does it to take
runtest files which are files with a list of testcase names and commands
to be executed (usually test binary name optionaly with default
parameters). Once a testcase is executed the test driver then watches
over the processes, writes logs, kills any orphaned children etc. What
the current system lacks are per-test timeouts and few more metadata in
the runtest files.

Cyril Hrubis
chrubis at suse.cz

More information about the lsb-discuss mailing list