[Fuego] Fuego's version up and other changes

[Bird, Timothy]

> -----Original Message-----
> From: Kevin Hilman on Thursday, February 02, 2017 12:52 PM
> "Bird, Timothy" <Tim.Bird at sony.com> writes:
> 
> [...]
> 
> > I haven't seen your lava integration slides yet, but it sounds like
> > they have some elaborate setup and teardown operations as well.
> 
> Speaking of LAVA (and hijacking the thread a bit, since I just
> subscribed...)
> 
> I'm new to fuego and would love to see/hear more about the LAVA
> integration status/plans/future etc. (Unfortunately, I won't be a FOSDEM
> to hear Jan-Simon's talk.)

He'll be giving the talk at ELC, so if you're coming to that you can
see it there (or a few weeks later on video).

> Fuego seems OK in terms of running tests, but actually adding boards is,
> well... less than what I was hoping for.
>
> There are many labs out there with lots of different boards hooked up to
> LAVA already, so having a way for Fuego to submit tests and extract
> results from LAVA seems like a good area for future work.

Ok - now seems like as good a time as any to start rambling on about the
Fuego roadmap and priorities... :-)

Yeah - I've consciously NOT put a lot of work into Fuego's board handling,
since there are other systems (including LAVA) that do this already.
Right now, Fuego supports software reboot, but doesn't have an API
for the kind of hardware control of power, reboot, reflashing, serial console,
etc. needed for kernel bring-up testing, like what KernelCI does.

Since KernelCI is doing a good job in that space (Continuous Integration
kernel boot testing at top of tree on a matrix of boards), I've been trying
to position Fuego as more of a distribution tester.  So, for now we're assuming
that the kernel boot and networking are stable, and Fuego is focused on
testing features and packages.

Right now, we're pretty reliant on a stable ssh, shell and filesystem.  We've
lately been workingon improving our Jenkins integration (decoupling from a
specific version of Jenkins, or even from Jenkins itself, with our own CLI
test tool), and migrating towards a standalone test package format
(so all tests don't have to be in our repository, and we can distribute
tests independently from the Fuego core system).

Big areas to follow are:
 - canonical test output format
 - ability to gather test results from nodes
 - ability to download test packages from a "store"

The area I'd like Fuego to focus on, that I don't see other Linux test
frameworks doing, is actually building out a complement of tests.
I'd like Fuego to be a "test distribution" with lots of tests for different
aspects of the system (file system, networking, realtime, power management,
etc.)  I see it as important to get the package system and test API
right, before doing a concerted push to create all the different tests 
in different areas. This is why the current focus is where it is.

Our first priority is as a distribution tester, where someone (a Linux
Foundation project, a semiconductor vendor, or an internal team
at a big company) has produced a distribution of Linux, and wants to
validate that it has few bugs, and continues working as intended when
delivered to other groups.  The ideal is that the whole test framework
can be delivered with the distribution, and the distro customer can
continue to validate for themselves that nothing has broken in the distro
while thy develop their end product (and modify parts of the distro).

Anyway - I'm also interested in Jan-Simon's LAVA integration.
It would be great to be able to run Fuego tests in current KernelCI
labs.
 -- Tim