[Fuego] [PATCH] LTP_Smack: add a new job to test "smack" module

[Tim.Bird at sony.com]

> -----Original Message-----
> From Daniel Sangorrin
> 
> Hi Li, Tim
> 
> > -----Original Message-----
> > From: fuego-bounces at lists.linuxfoundation.org
> > <fuego-bounces at lists.linuxfoundation.org> On Behalf Of Li, Xiaoming
> [...]
> > diff --git a/engine/tests/Functional.LTP_Smack/test_mount.sh
> > b/engine/tests/Functional.LTP_Smack/test_mount.sh
> > new file mode 100644
> > index 0000000..9d78154
> > --- /dev/null
> > +++ b/engine/tests/Functional.LTP_Smack/test_mount.sh
> > @@ -0,0 +1,28 @@
> > +#!/bin/sh
> > +
> > +if [ "$1" == "start" ]; then
> > +    touch test_mount.log
> > +    mount | grep -v /sys/fs/smackfs | grep /smack > /dev/null
> > +    if [ $? -eq 0 ]; then
> > +        exit 0
> > +    fi
> > +
> > +    if [ ! -d /smack ]; then
> > +        mkdir /smack > /dev/null
> > +        echo "NEW_DIR" >> test_mount.log
> > +    fi
> > +
> > +    mount -t smackfs smackfs /smack
> > +    echo "NEW_MOUNT" >> test_mount.log
> > +fi
> > +
> > +
> > +if [ "$1" == "end" ]; then
> > +    if grep "NEW_MOUNT" test_mount.log > /dev/null; then
> > +        umount /smack
> > +    fi
> > +
> > +    if grep "NEW_DIR" test_mount.log > /dev/null; then
> > +        rmdir /smack
> > +    fi
> > +fi
> 
> This could be added as prechecks in LTP (note that it requires root
> permissions).
> But I think we don't need to mount it for the user, just check that smack is
> ready and report otherwise should be fine.
> Additionally, checking the SMACK Kconfig values in the target board's kernel
> configuration would be helpful.
> What do you think Tim?

There are a few different issues here, that I think are worth discussing.

First, what approach to take with the test depends a bit on the
expected status of the kernel and distribution, and what you are
trying to test.  If it is expected that the kernel will have smack
configured and "running" (mounted), then the best approach is
to add some pre_checks for those conditions, and abort the test if
they are not met.

I don't think it makes much sense for someone to test smack, unless
they are actually using it in normal practice.

So, I think a check of kconfig would be good, as well as possibly a
test that smack is turned on, and already active.  That's probably
what this test should have.

(OK - now waxing a bit philosophical and looking at the bigger picture...
Sorry in advance for the mental detour.)

Another kind of test, however, is of these mechanisms themselves
(configuring the kernel, passing the kernel command line, mounting
the smackfs).  That test of smack (a different one than the one embedded
in LTP) would require a different approach.

Fuego tests usually have a restricted scope of execution, which is 
after the core software is built, the board is provisioned, and the board
is booted.  Specifically, we currently leave provisioning and hardware
booting as an exercise for the user.  If we need to handle these other
steps, and string multiple actions together, we can somewhat do that
with batch jobs.

However, one can imagine certain kinds of testing where we had a different
scope of execution for test_run.  Here are some different scopes:
1) minimal - the board is running, the test is already built and present on the board
(e.g. as part of the base distribution for the board), and we just execute the test.
In this case, we can skip test_build and test_deploy, and just do test_run and
other phases.
2) normal - the board is already running, and fuego builds the test, deploys it, and executes
it on the board.
3) boot test - the board is not already running.  Fuego boots the board as part of
run_test (possibly to test kernel command line parameters, or examine other
boot-time operations)
4) build & boot test - the kernel and/or distribution software are built as part
of the test, the board is provisioned and booted as part of the test, and fuego
records information about this.  This scope of testing allows us to alter 
kernel configuration parameters (possibly from different specs, or using dynamic
vars to do e.g. config bisecting), as well as alter boot parameters.  This type of
testing is currently outside the scope of Fuego.

Fuego currently has an API that can handle rebooting the board, but it does
not have functions for managing boot parameters (varying the kernel
at boot time).  Also, Fuego does not have an API for managing base software
building, or for provisioning the board.  So, there are various kinds of 
variation testing that Fuego is not well-suited for, at the moment.  I hope
to add APIs for Fuego to do these kinds of operations in the future.  At the
moment, you could use Fuego tests or Jenkins jobs to perform these other
operations, and string them together with Fuego batch jobs.  However,
there are a few things that need to be done to support this more effectively,
and especially to be able to share such artifacts with each other.

If we string Fuego tests together in batch jobs to handle these types of
expanded scope testing, we need some clear protocols between tests, to
allow them to coordinate their inputs and outputs.  For example, there
needs to be communication between a test that does kernel build and a test that
does board provisioning, to specify the location of the kernel and modules
to install on the board.

You can do these types of tests with Fuego, but it is more work than it should
be to deal with different board configurations and especially provisioning
methods.  This is one reason I'm working with other groups to formulate
some industry-wide APIs for these things.

</end of detour>

Let's get this test fixed up, and see where to go next on this stuff.
 -- Tim