[Fuego] Fuego's version up and other changes

[Bird, Timothy]

Sorry for the slow response.  I was out-of-office most of this week.

> From: Daniel Sangorrin on Thursday, January 19, 2017 11:10 PM
> I am assuming a workflow where you have a testplan for each of your targets
> (otherwise you'll get the default one).
> In a testplan you define which tests you want to run on your target. For
> example, you may only want to run
> 8 tests on a specific board. You also define the spec to use for each test (by
> 'spec' I mean the word that gets mapped to a
> specific set of parameters for the test).

OK - that's actually a bit different than how testplans are currently used
in Fuego.  Weirdly (IMHO), they don't actually specify the set of tests
to run, but only the list of specs to use for each test indicated in the plan.

So, the 'testplan_mmc' specifies that each test it mentions should use
the 'mmc' spec, when used with that plan.  But Fuego does not currently
have a feature (that I can see) to run all of the tests mentioned in the testplan.

In terms of separation, I believe the Jenkins job was intended to provide the
actual tests to perform, while the parameters for each test were come from
the testplan/spec system.  But Fuego doesn't actually ship with any pre-defined
Jenkins jobs that run multiple tests that are statically defined.

The only multi-test jobs are the 3 tests (in the "batch runs" tab):
  * run all test on all targets
  * run all tests on selected targets
  * run selected tests on selected targets

The second and third of these are interactive, and I can find no feature
that saves the information so you can re-run the test with your previous
selections whenever you want. That is, you have to manually select
the targets and tests each time.

> 
> When you create the jobs for your target, you pass the corresponding
> testplan as a parameter
> to fuego-create-jobs. This tool will traverse the testplan and create the jobs
> on the jenkins interface.
> 
> [Note] At the moment, if you want to use two different specs for the same
> target-test combination you have to
> define it in a separate testplan. But I will add support for testplans that
> contain the same test with different specs.

That sounds good and superior to what we have now.

> 
> > It looks like you also create a separate Jenkins job for each combination of
> > test and target?  Is that right?
> 
> More exactly, for each combination of target.testplan.test.
> [Note] When I add support for multiple specs inside a testplan it will become
> target.testplan.testspec.test.
>

I've been looking at how testplans are currently defined in Jenkins, and 
frankly they could use an overhaul.  They don't do what you describe
in your refactoring, but I think they should.  The ability to run multiple
tests in a predefined configuration and order should be a feature of
fuego-core, and not reliant on Jenkins.

As part of my work towards supporting test packaging, 
I have modified ovgen.py to not require testplan_default, and to read
all the spec files from their respective test directories (instead of a single
centralized location).

There are two kinds of testplans in current fuego:
 - single-test: this allows you to choose a single spec for an individual test
 - multi-test: this allows you to choose a single spec name for multiple tests

There are currently no instances of testplans that indicate different specs
for different tests.  But this is what you are describing, and I think this
was a feature originally envisioned for this system.

For Jenkins jobs, I think composing them may be the way to go, but I've got
to finish my ovgen.py rewrite first.  I'd like the architecture to support
both jenkins and command line test invocation equally well.

I'd like to do away with the need to edit testplans when a test is installed,
so I think they should be more loosely coupled than they are now.  I'm 
looking at adding a wildcard capability to the test name.  With this, we could have
something like testplan_board_smoketest that would look like:

{
    "testPlanName": "testplan_board_smoketest",
    "tests": [
        {
            "testName": "*",
            "spec": "smoketest"
        }
    ]
}
That way, if someone adds a new smoketest test to the system, all they
need to provide is the spec 'smoketest', and they would run with those
testplans that mention that spec.

Alternatively, when running an individual test, one could just list the available
specs, without the need for a testplan at all.

I had hoped to come up with modifications to ovgen.py that worked with
both the current fuego system and your new one, but to be honest I've been
stumped.  Everything I do ends up requiring edits to the groovy scripts, which
you have eliminated in your system.

Ultimately, I want people to be able to share testplans, specs, and tests
freely.  But since plans mention specs and tests, they are dependent on
those being installed.  So, if someone installed something like "testplan_basic_distro_check",
it would have a list of tests and specs that should be run to validate a basic
embedded Linux distribution.  Maybe an installer for that plan could
automatically find and load the tests that were referenced by it.

I think that specs should obviously be in the test directory, as they are very
test-specific, but it would be nice if people could provide different
specs very easily for the same test (for example, a real-time developer
could provide different specs for cyclictest, to examine or test different
aspects of the system with the same tool).  Having the specs in separate
files is already a nice feature for this decoupling.

Anyway, I'll try to get something that handles the current system, then
quickly move on to supporting your new model.

Any feedback would be welcome.
 -- Tim