[Fuego] Stuff Tim's been working on

Bird, Timothy Tim.Bird at am.sony.com
Fri Jan 13 04:32:18 UTC 2017


Hello all,

Daniel has done some excellent work over the last few weeks. 
Despite my taking a bit of time off for Christmas and New Years,
I've also been working on a few things for Fuego as well.

They're not quite ready to publish, but I thought I'd mention them so you know what's up.

My top priorities are:
 - implementing a system for test dependencies
 - simplifying the testplan and test spec features
 - working towards a test package format

The test dependencies are modeled after something I saw in 0day, that allows a test to indicate
any dependencies it has on features, kernel config, memory, etc. on the target.
Theoretically, you can do this in an ad-hoc fashion already with test variables and assert_define
statements, and shell code.  But I'm implementing something that 1) is easy to use to specify
some well-known and common dependencies, and 2) allows the dependencies
to be expressed in a declarative, rather than imperative, fashion.  This is how 0day does it,
and I feel like that will be useful (sometime in the future) for fuego to be able to automatically
detect and record test dependencies.   I feel like one of the bigger problem areas that fuego
has to deal with is making sure we avoid false positives due to missing dependencies on
the target.

The testplan and test spec features of fuego are, IMHO, overengineered for what they 
accomplish.  0day does the same thing using simple shell variables.  Fuego is using json.
I'm not changing that just yet, but I do want to modify ovgen.py so that if a test has
no test variables (nothing to put in a spec file), and no test plans besides 'testplan_default',
then no JSON files are required.  We have too many (false) failures based on issues
around test specs and plans not working correctly.  I'd like to fix this.

Finally, I think it's important to create a test package system.  I would like, eventually,
for there to be an ecosystem of tests, outside of a single git repository.  Modifying
the tests so that they do not need to be accompanied by binaries (tarfiles), will be
important to this.  So I'm interested in the ideas we've discussed about retrieving
test program source using git (or other fetch mechanisms).  Also, I want to improve
the support for using test programs already present on target because they are
already in the target Linux distribution.

I think formalizing the test packages will also help us isolate the tests themselves
from the framework.  This will help us implement update mechanisms, for when
fundamental framework changes occur.  I want the system to be modular enough
to handle changes to different system parts without stuff breaking.  I strongly applaud
Daniel for his work on refactoring stuff to make it possible to update the Jenkins
front end more easily.

I haven't forgotten about the ideas we've discussed for unified test output
parsing (or a canonical test output format).  Right now, those are on the back-burner
for me.

That's it for now.  I've been preoccupied with work on ELC the last few weeks, but
hope to have some time to finish up some of the above items shortly (hopefully
within the next few weeks, but we'll see.  The dependency stuff has taken longer
than I expected already.)

Regards,
  -- Tim



More information about the Fuego mailing list