[Fuego] Stuff Tim's been working on

[Daniel Sangorrin]

Hi Tim,

> From: fuego-bounces at lists.linuxfoundation.org [mailto:fuego-bounces at lists.linuxfoundation.org] On Behalf Of Bird, Timothy

> My top priorities are:
>  - implementing a system for test dependencies
>  - simplifying the testplan and test spec features
>  - working towards a test package format
> 
> The test dependencies are modeled after something I saw in 0day, that allows a test to indicate
> any dependencies it has on features, kernel config, memory, etc. on the target.
> Theoretically, you can do this in an ad-hoc fashion already with test variables and assert_define
> statements, and shell code.  But I'm implementing something that 1) is easy to use to specify
> some well-known and common dependencies, and 2) allows the dependencies
> to be expressed in a declarative, rather than imperative, fashion.  This is how 0day does it,
> and I feel like that will be useful (sometime in the future) for fuego to be able to automatically
> detect and record test dependencies.   I feel like one of the bigger problem areas that fuego
> has to deal with is making sure we avoid false positives due to missing dependencies on
> the target.
> 
> The testplan and test spec features of fuego are, IMHO, overengineered for what they
> accomplish.  0day does the same thing using simple shell variables.  Fuego is using json.
> I'm not changing that just yet, but I do want to modify ovgen.py so that if a test has
> no test variables (nothing to put in a spec file), and no test plans besides 'testplan_default',
> then no JSON files are required.  We have too many (false) failures based on issues
> around test specs and plans not working correctly.  I'd like to fix this.

Yeah, if a test doesn't need parameters it shouldn't fail. I also think the ovgen.py has some
problems that need to be fixed.

I am using testplans to generate jobs in my upgraded version. One thing I learned is that
they can be useful to define which tests will execute on a board and with which specs. They
also allow specifying timeouts for example.

About 0day, is that the same project as LKP (Linux Kernel Performance)?
As far as I can see, they are defining the parameters through YAML files, not shell variables. Check here:

https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/tree/jobs/hackbench.yaml

# I may have missed something because I only looked at the code briefly.

...
> That's it for now.  I've been preoccupied with work on ELC the last few weeks, but
> hope to have some time to finish up some of the above items shortly (hopefully
> within the next few weeks, but we'll see.  The dependency stuff has taken longer
> than I expected already.)

I will be at ELC as well. 
In my case, I'm going to try to get kernelci working with fuego by then. Although I'm going
to be busy so I can't really promise.

Cheers,
Daniel