[lsb-discuss] unofficial LSB conference call minutes for 23 Apr 2014, post Collab 'New LSB' wiki point in time drop

chrubis at suse.cz chrubis at suse.cz
Mon Apr 28 16:13:21 UTC 2014

> > LTP is more testcase centric, all of the testcases are just executable
> > files (either binary or shell) the testcase output is printed to stdout
> > by default, test overall result is propagated via exit value, etc. To
> > compile a testcase there is no need to install anything besides basic C
> > devel libraries and testcases can be easily tweaked, recompiled and
> > executed directly from git checkout. A few of them needs to add PWD to
> > PATH in order to run subexecutables and some works with block device
> > that needs to be prepared beforehand and passed in parameters but that
> > is all it takes to execute the testcase.
> This sounds very close to what we're thinking of doing for the "new" LSB
> test suites, so it sounds like LTP may be interesting to us.
> That said, I'm also concerned about the other end of what Mats was
> describing--the part where you run lots of tests at once, with nice
> tools for managing the results, formatting, and so on--and it sounds
> like LTP may have some deficiencies there.

Yes, the part that runs tests is not in a good shape and as I've said
I've started to work on replacement however it's not ready yet.

I have a pretty good idea of how it should look like, which is:

* frontend
  - gets a few parameters from the user, i.e. partition to use for
    testing (if any), testcases to run, etc.

  - parses configuration files with testcases to be executed

  - this also includes exit value mappings to success, failure, ...
    because different testsuites maps these differently

  - the frontend will support several file formats, at least the current
    LTP runtest format and something new with more metadata

    - metadata I would like to include are per test timeouts, if the test
      can be executed with background load...

      then some tags, i.e. tag all syscall testcases with syscall tag, all
      IPC testcases with ipc tag so we can run group of testcases by a tag

* test execution framework

  - this part executes tests, watches for timeouts, kills whole test
    process tree if needed, etc.

  - the test results are passed to backend

* backend

  - produces logs and output to terminal

  - writes logs in selected format, so far I've been thinking of using a
    JSON as default log format

I've sketched code for the test execution framework and wrote simple
parser but it's far from being finished.

> Time is always an issue, but if we can cooperate on making that latter
> bit work better for both of us, I'm interested.

As I've said I consider the current system LTP uses to execute testcases
beyond the point at which repair makes sense so we keep it only in
maintenence mode till we have replacement.

And till now the execution framework haven't had much priority because
there were even more broken things and tons of failing testcases.

However if you are interested we can develop something that fits both
LTP and LSB needs.

Cyril Hrubis
chrubis at suse.cz

More information about the lsb-discuss mailing list