[Fuego] rt-test related fixes

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on Monday, November 06, 2017 11:21 PM
> I have updated the cyclictest tarball to use the branch unstable/devel/v1.1.1
> which seems to work fine on the Beablebone black (no negative latencies).

Thanks!

> Note that other branches
> can be used by just specifying them on the spec file.

This is directly related to the other discussion we were having about
per_job_build.  Uses of this variable are now starting to pile up.
When I made the variable, I thought that only a few tests would
use it, but if it becomes a regular practice to put alternate source 
repositories into the spec file, then it's use will become more common.
I'm not sure that using multiple sources for the same test is something
that's a good idea in general, but it appears to have its uses and possibly
they are more common than I anticipated.

I think the per_job_build variable is confusing enough that maybe I should look at
handling code sharing automatically.  That is, try to detect when source
sharing is allowable and when not.  Maybe a hash on the tarball or gitrepo
would work for this, to uniquely identify the source used in the
build directory.

>
> I also upgraded the version for other rt-tests that were in Fuego (except
> hackbench): signaltest and pi_tests. Note that signaltest will probably
> cross the 1ms latency threshold and fail unless you change the kernel
> configuration or patch it with real-time support.
OK.  Also, a board without real-time support should have a different
pass criteria, for starters.

> 
> There are actually more test cases inside rt-tests. At first, I thought about
> putting them all together. But then I thought that it would be nice to be
> able to choose the background load for cyclictest from any of the tests
> currently available in Fuego (e.g. a kernel build). That would require
> support for running multiple jobs concurrently on the same board.

Supporting background load is something that's been on my long-range
radar for some time.  Zero-day has support for this, but I'm pretty sure
they are not executing full tests as their background load.  Thinking about this,
I'm not sure this would be very hard.  It might only require a flag to ftc
indicating to 1) run a test without checking the Jenkins status, and 2) without
processing the results, and 3) keeping it in a loop while the test is executing.

Maybe we could do something like this, in the spec:
  "default": {
            "PARAMS": "-S -p 60 -m -D 20 -i 1000 -q",
            "host-load-command": "ftc run-test -b $BOARD -t Functional.mystress --as-load"
        },

The reason to name it "host-load-command" is to distinguish it from "load-command", which
could be something that already exists on the target:
"load-command": "/usr/local/bin/dohell"
(dohell is a test Ingo Molnar wrote for his own RT-Preempt testing.)

> Regards
> Daniel
> 
> [PATCH 1/9] cyclictest: update tarball version (not sent to mailing list)

Where does the tarball come from?  Manually created from the git repo? Or is there an
official tarball somewhere?  If we are creating it manually, it would be good to add
a comment to fuego_test.sh about the steps used to create it.  In general, Fuego
tests currently don't do a good job of documenting where the test program
source comes from. With the introduction of gitrepo, this improves dramatically. 
But I've thought for a while that in the case of tarballs, there should be some
comment or something that indicates the source of the tarball, so future
generations can update them more easily.

I'll review the other patches individually.

Thanks!
 -- Tim

> [PATCH 2/9] cyclictest: update fuego_test to use v1.1.1
> [PATCH 3/9] cyclictest: update twothreads spec
> [PATCH 4/9] functions: fix preference order when unpacking or cloning
> [PATCH 5/9] cyclictest: convert tabs to spaces
> [PATCH 6/9] cyclictest: add assert_define
> BENCHMARK_CYCLICTEST_PARAMS
> [PATCH 7/9] cyclictest: add NEED_ROOT for now
> [PATCH 8/9] signaltest: update to new framework
> [PATCH 9/9] pi_tests: add patch for old toolchains