[Fuego] First steps with fuego

[Bird, Timothy]

> -----Original Message-----
> From: Rafael Gago on Wednesday, May 10, 2017 6:12 AM
> I got some time today to keep investigating today, it was trivial to integrate
> with one of our boards, so the Wiki it's up to date and good. It was a very
> easy and trouble free setup.

Thanks.  That's good to hear.
 
> Then I wanted as an exercise to write test a for rs485 (the same case
> happens with CAN) but I have been looking at "fuego-core/engine/tests"
> and I couldn't answer the questions myself.
> 
> 1.) Our current setup does device poweon/off the device (custom
> commands in LAVA), fetches a kernel, dtb and rootfs from our build servers
> and copies the images to the DUT's RAM for RAM booting (tftp + Uboot
> commands integrated in LAVA). How does these steps are best integrated in
> fuego?
> 
> LAVA has this feature integrated, but we already have an internal Python
> program that handles this, so we don't need fuego to talk with UBoot, just to
> know the best place to launch the tool. Spontaneusly and with what I know I
> only can think about doing it on the "test_pre_check" function or as the first
> (dummy) test of a DUT specific tesplan, but both would have its drawbacks.

It depends on what the actual test is.  If you are trying to determine if something
is going wrong with those steps themselves, then it would be appropriate
to put them in the "test_run()" function.  If these are just pre-cursors to prepare
for the actual testing, and the "actual' test is of some other functionality, then
they should go in "test_pre_check()", probably with something to validate that
they succeeded before proceeding with the actual test.

If this is for setup for a sequence of tests, then I think making a dummy test,
as you indicated, is the right thing.  You could add it as the first step of a Fuego
testplan, or you could make your own Jenkins job to handle the job sequence.
There are two Jenkins plugins which can handle this type of thing:

https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/Join+Plugin

Both of these allow for jobs to be chained together, as precursors or successors
to other jobs.

I'm am not a LAVA expert, but my understanding is that LAVA always does a reboot
between tests.  (I may be confusing this with KernelCI)  Fuego does not have this
model by default, but you can construct Jenkins jobs to perform this type of
system-level sequence of 'build, deploy and reboot' if desired. 
This presentation might be helpful:
Continuous Integration and Autotest Environment Using Fuego - by Kenji Tadano & Kengo Ibe (Mitsubishi Electric), Oct 2016 at ELCE 2016
You can find the links for the PDF and video at:
http://bird.org/fuego/Presentations#2016

If lots of people do testing this way, we should probably improve our support
for it, or come up with guidelines for a good way to do it.

> 2.) Let's say that we have the desired image running on the device and we
> want to test RS485 TX/RX at different baudrates. Now with LAVA we boot,
> wait until both machines are booted (using the "lava-send" and "lava-wait"
> synchronization primitives) set the baudrate in both sides, start the receiver
> side (either host or DUT), synchronize with the sender side (either host or
> DUT) , start sending and then we start the cycle again with another baudrate
> - TX/RX cfg. We do the same type of test for CAN too (using cansequence).

Hmmm.  The 'ftc' command does have a 'wait-for' sub-command for this type
of thing.  However,  I don't know of any tests that use this, and I don't know how
it compares to the lava-send/lava-wait protocols.  When I've used it in the
past I've done simple things like check for the existence of a file, and the "signaling"
side has done something like "touch /tmp/target_ready"

It currently only checks conditions on the host.  This is a deficiency.  It should also
check for conditions on the target.  Something could be jury-rigged, but it would
require some multi-processing in the test_run function.

> I haven't figured out how I would implement such test with fuego. Both
> sender and receiver would need to be able to signal a failure, but there is
> only a "run_test" function and it's running on the DUT. Is there any way to
> implement this or any sane workaround?

test_run() is running on the host, and can start and stop both sides.
But it is not threaded.  It would need to start separate processes
on the sending side and the receiving side of the test, and keep them in
sync.  If one side is the host, then 'ftc wait-for' could be used for this.
You could have a sequence in test_run that looked like the code below.

I'm just brainstorming this, and haven't tried any of this, but here's
some code:  This assumes that the sender on the host can detect
a failure or will time out, and will do "touch /tmp/sender_done" when
it is finished.

# start the test log on the board
report "echo Starting baud_rate test"
for baud_rate in $FUNCTIONAL_RS485_RXTX_BAUD_RATE_LIST ; do 
   # start a listener on the target, and append it's stdout to the test log
   report_append "start_listener $baud_rate"

   # start a sender on the host, and collect it's stdout locally (on the host)
   $TEST_HOME/start_sender $baud_rate >/tmp/sender_log

   # wait (up to 100 seconds) for process on host to signal completion
   ftc wait-for -t 100 "test -f /tmp/sender_done"
   # kill sender on host (should check ftc exit code to see if this is needed)
   pkill sender

   # terminate listener on board, if it hasn't already exited
   kill_procs listener

   # append the log from the host to the board's test log
   put /tmp/sender_log /tmp
   report_append "cat /tmp/sender_log"
done

BAUD_RATE_LIST would be in the spec file for the test
(named here as Functional.RS485_rxtx).

start_sender is a made-up program that runs on the host to 
start one side of the connection.  start_listener is a made-up
program that runs on the target board to start the other side
of the connection.  This code assumes it starts a process called
'listener' on the board, and this code kills that listener when the
test is over, if it is still running. start_listener is deployed to the target
in test_deploy, and start_sender is executed directly from
the test's home directory (e.g. fuego-core/engine/tests/Functional.RS485_rxtx)

OK - just going through this mental exercise has shown some
deficiencies in Fuego's host/target logging model and synchronization
that we should correct.  There should definitely be a method of adding 
something from the host to the log, in one step.  And we should
extend ftc to support checking the condition of something on target.
I'm not sure if the logging shown here will work or not.  start_listener
needs to return immediately, and leave the 'listener' process executing
on the board, still outputting to the log.  If this doesn't work by default,
another layer of output capture and appending to the log could be done,
but that's more awkward.

The function "report()" starts the log on the board, and "report_append()"
adds more material to it.  Since we also want information from the host
(or from a 3rd machine), we have to collect that ourselves and add it to
the log directly.

The tmp file (sender_log) on the host should definitely be put into a more
unique location (test specific, and with a unique temp filename), to avoid
collisions between tests running on multiple boards.

But this shows proof of concept for how this can be done.

> 
> 3.) This one is very easy to add/contribute, but some inbuilt support to
> specify inside a test a git + branch + commit sha instead of a tarball would be
> nice to avoid duplication of tests for different library versions (e.g. when
> testing previous image releases that get backports).
That's a good use case to support.

Actually, we're adding support for getting source for a test from git in Fuego
v1.2.  Some preliminary support is in the 'next' branch, but it still needs a 
bit of work (It doesn't support specifying the commit ID yet).  See the
function "unpack" in fuego-core/engine/scripts/functions.sh.
 
 -- Tim