[lsb-discuss] LSB 3.2 additions to "desktop" test suite
khoroshilov at ispras.ru
Fri Apr 13 06:37:21 PDT 2007
> Based on contributions from late last year, which I'm
> finally getting around to working on, there are four
> new components to the desktop test suite. These are:
> 1. freedesktop.org specs (menu, "desktop" mime types)
> 2. freetype library
> 3. xft library
> 4. xrender library
> Through the use of autobuilt tests suites, these
> will begin appearing as they're fully integrated.
> For items 2-4, the contribution includes a framework,
> but not a build setup, mainly because the tests cannot
> build properly until the support for these three
> libraries is integrated into the database/build tools.
> That work is just starting up again and hopefully we'll
> start seeing some results in the next couple of weeks.
> So while the packages include directories for these
> three tests, and the test driver (run_tests) will call
> them, they'll fail because there are actually no test
> binaries present - yet.
> For item #1, there's now an lsb-xdg-utils package
> (in snapshots, it was added to the autobuilder last
> week) which the test depends on, to deal with systems which may not
yet have xdg-utils, but I notice the test
> uses the native xdg-utils if it found it. There's a
> big problem though; as we head more towards tests which
> are driven from a single command and operate completely
> non-interactively, this test is very interactive - for
> example, it repeatedly pops a menu up and asks you
> to select an item from it, then confirm that it worked
> right. Since this is at odds with our current automated
> testing philosophy, we'll need to come up with something
> to do here - and this email, if you've gotten this far,
> is a call for suggestions.
> In general, there seem two approaches we could take:
> (1) redesign the test in some way so that it can be
> operated non-interactively (could include using some
> sort of driver which handles the interactive parts)
> (2) split this test out into the small list (all the
> others are currently appbat applications) which can
> not be driven from the test driver.
At the first glance all interactions of the freedesktop.org tests are
important to make decision on correctness of the system under test. So I
believe the second approach is more attractive.
But it would be good to split out really interactive tests only and to
keep non-interactive part of the test suite in the lsb-test-desktop. The
reason is as follows. The LSB Distribution Test Kit supports two basic
use cases: certification test run and "nightly" regression test run. The
first one includes all interactive tests, while the second one requires
fully automated execution.
And we would like to keep all non-interactive tests in the "nightly"
test execution and to have additional interactive tests for bonus and
This can be achieved if the lsb-test-desktop takes a parameter, which
controls the set of tests to be executed:
- non-interactive only;
- interactive only;
The parameter will help to the lsb-dtk-manager to manage test execution.
For "nightly" test runs the lsb-dtk-manager will execute all
non-interactive tests. For certification test runs the lsb-dtk-manager
will execute sequentially: at first all non-interactive tests, and then
all interactive tests.
More information about the lsb-discuss