[Fuego] rt-test related fixes

[Daniel Sangorrin]

> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> Sent: Wednesday, November 08, 2017 10:51 AM
> To: Daniel Sangorrin; fuego at lists.linuxfoundation.org
> Subject: RE: [Fuego] rt-test related fixes
> 
> > -----Original Message-----
> > From: Daniel Sangorrin on Monday, November 06, 2017 11:21 PM
> > I have updated the cyclictest tarball to use the branch unstable/devel/v1.1.1
> > which seems to work fine on the Beablebone black (no negative latencies).
> 
> Thanks!
> 
> > Note that other branches
> > can be used by just specifying them on the spec file.
> 
> This is directly related to the other discussion we were having about
> per_job_build.  Uses of this variable are now starting to pile up.
> When I made the variable, I thought that only a few tests would
> use it, but if it becomes a regular practice to put alternate source
> repositories into the spec file, then it's use will become more common.
> I'm not sure that using multiple sources for the same test is something
> that's a good idea in general, but it appears to have its uses and possibly
> they are more common than I anticipated.

I think there will be more use cases.

For example, kselftests (which for some reason didn't get into your branch) will
probably require having different sources depending on the kernel you want to
test.

Another use case is to use source code that works for relatively old toolchains when
the latest source code requires a more recent compiler.

> I think the per_job_build variable is confusing enough that maybe I should look at
> handling code sharing automatically.  That is, try to detect when source
> sharing is allowable and when not.  Maybe a hash on the tarball or gitrepo
> would work for this, to uniquely identify the source used in the
> build directory.

That's a great idea.

> > I also upgraded the version for other rt-tests that were in Fuego (except
> > hackbench): signaltest and pi_tests. Note that signaltest will probably
> > cross the 1ms latency threshold and fail unless you change the kernel
> > configuration or patch it with real-time support.
> OK.  Also, a board without real-time support should have a different
> pass criteria, for starters.
> 
> >
> > There are actually more test cases inside rt-tests. At first, I thought about
> > putting them all together. But then I thought that it would be nice to be
> > able to choose the background load for cyclictest from any of the tests
> > currently available in Fuego (e.g. a kernel build). That would require
> > support for running multiple jobs concurrently on the same board.
> 
> Supporting background load is something that's been on my long-range
> radar for some time.  Zero-day has support for this, but I'm pretty sure
> they are not executing full tests as their background load.  Thinking about this,
> I'm not sure this would be very hard.  It might only require a flag to ftc
> indicating to 1) run a test without checking the Jenkins status, and 2) without
> processing the results, and 3) keeping it in a loop while the test is executing.
> 
> Maybe we could do something like this, in the spec:
>   "default": {
>             "PARAMS": "-S -p 60 -m -D 20 -i 1000 -q",
>             "host-load-command": "ftc run-test -b $BOARD -t Functional.mystress --as-load"
>         },
> 
> The reason to name it "host-load-command" is to distinguish it from "load-command", which
> could be something that already exists on the target:
> "load-command": "/usr/local/bin/dohell"
> (dohell is a test Ingo Molnar wrote for his own RT-Preempt testing.)

Sounds good. I have to confess that I haven't used ftc run-test yet.

> 
> > Regards
> > Daniel
> >
> > [PATCH 1/9] cyclictest: update tarball version (not sent to mailing list)
> 
> Where does the tarball come from?  Manually created from the git repo? Or is there an
> official tarball somewhere?  If we are creating it manually, it would be good to add
> a comment to fuego_test.sh about the steps used to create it.  In general, Fuego
> tests currently don't do a good job of documenting where the test program
> source comes from. With the introduction of gitrepo, this improves dramatically.
> But I've thought for a while that in the case of tarballs, there should be some
> comment or something that indicates the source of the tarball, so future
> generations can update them more easily.

Sorry, the origin of the tarball is written in the commit log. It comes from the unstable
development v1.1.1 branch (please see the commit id in the commit log).
By the way, all tests belonging to rt-tests (Benchmark.hackbench, Benchmakr.signaltest,
Benchmark.cyclictest, Functional.pi_tests etc..) will be sharing the same source code.
Maybe I should specify a common build folder and just build all binaries at once?

Thanks,
Daniel

> 
> > [PATCH 2/9] cyclictest: update fuego_test to use v1.1.1
> > [PATCH 3/9] cyclictest: update twothreads spec
> > [PATCH 4/9] functions: fix preference order when unpacking or cloning
> > [PATCH 5/9] cyclictest: convert tabs to spaces
> > [PATCH 6/9] cyclictest: add assert_define
> > BENCHMARK_CYCLICTEST_PARAMS
> > [PATCH 7/9] cyclictest: add NEED_ROOT for now
> > [PATCH 8/9] signaltest: update to new framework
> > [PATCH 9/9] pi_tests: add patch for old toolchains
>