[Fuego] LTP test split - was RE: Wiki docs for 1.3 release

[Daniel Sangorrin]

Hi Tim,

> -----Original Message-----
> From: fuego-bounces at lists.linuxfoundation.org [mailto:fuego-bounces at lists.linuxfoundation.org] On Behalf Of Tim.Bird at sony.com
> Sent: Tuesday, March 27, 2018 10:13 PM
> To: Tim.Bird at sony.com; fuego at lists.linuxfoundation.org
> Subject: [Fuego] LTP test split - was RE: Wiki docs for 1.3 release
> 
> 
> 
> > -----Original Message-----
> > From: Tim Bird
> > = priorities for 1.3 release =
> > == Free form list of items to do:
> >  - Features:
> >    - resolve toolchain upgrade (switch from emdebian to debian cross
> > toolchains)
> 
> One more thing I'd like to do for the 1.3 release, is to break off one
> of the LTP tests (probably 'smack'), and make a separate test for it.
> e.g. Functional.LTP-smack.
> 
> According to the patch by Liu Wenlong, this test had special setup
> and cleanup issues, separate from the rest of LTP.  I think the code
> for these should go into that test's fuego_test.sh.

OK, I have never tried this one. I am lagging a bit behind Liu's patches.

Side note:
For very complicated setups we should also start thinking about
what Debian CI is doing with QEMU/LXC root filesystem generation[1]. 
For example, suppose that we want to test "smack" on the latest kernel.
Having to reconfigure our board to use smack can take quite a lot of
time an expertise. It would be nicer if Fuego automatically created the
root filesystem and executed the tests on QEMU for example.
Of course, for testing actual products the boards are needed. But if you
only want to test a kernel or do a bisect, QEMU/LXC sounds like an easier
approach.

[1] https://github.com/terceiro/debci/tree/master/backends
 
> I think that trying to handle all of the LTP sub-tests within a single Fuego
> test is not practical.  There are whole sub-systems of LTP that have
> different output formats (the OpenPosix test suite, the realtime tests)
> and different methods of invocation, and it's difficult to manage them.

About the OpenPosix test suite, I wonder how hard it would be to unify 
the output format upstream. About the realtime tests, I feel they are
getting outdated and they are not being used by the PREEMPT_RT maintainers.
In general, I have the impression that there is a lot of work to be done
in LTP, such as maintaining old code and scripts unused.

> LTP may gain additional sub-test areas in the future, so we need to
> deal with these differences - and I'd like to do it using existing Fuego
> mechanisms.

The network tests might also need special attention.
 
> My idea is to figure out a mechanism to share the build directory, so that
> the host is not duplicating  1G of build space for each sub-test.  Any ideas
> for this are appreciated.  We have sharing of build instructions and source
> for netperf, OpenSSL and rt-tests, but I don't think we share the build
> directories for these, when they are used by different tests (e.g
> Benchmark.netperf vs. Functional.netperf).

I like this idea.

Tests sharing the same build folder could specify the build folder name
with a special variable (BUILD_DIR="sharedbuildfolderforrttests"), then
Fuego would also append the "-$ARCH" suffix and build there.
These tests should be sharing the same "test_build" function. And the
test_build function should build the binaries necessary running all tests.

> Let me know what you think.

I think that part of the work, such as output format unification should be done upstream. 
And part of the work, such as separate setup/checks/execution steps, in Fuego.

I feel this is going to be a huge effort. For release 1.3, separating the smack setup sounds
already big enough.

Thanks,
Daniel