[Fuego] Fuego 1.0.0 release

Daniel Sangorrin daniel.sangorrin at toshiba.co.jp
Mon Nov 14 03:25:13 UTC 2016

> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird at am.sony.com]
> Sent: Saturday, November 12, 2016 2:02 AM
> To: Daniel Sangorrin; fuego at lists.linuxfoundation.org
> Subject: RE: [Fuego] Fuego 1.0.0 release
> > -----Original Message-----
> > From: Daniel Sangorrin pm Friday, November 11, 2016 1:42 AM
> > > -----Original Message-----
> > > From: Bird, Timothy on Thursday, November 10, 2016 10:58 AM
> > > > -----Original Message-----
> > > > From: Daniel Sangorrin on Wednesday, November 09, 2016 4:55 PM
> > > > > -----Original Message-----
> > > > > From: Bird, Timothy on Wednesday, November 09, 2016 12:44 PM
> > > > > Let me know if you see any problems.  I've foregone going through an RC
> > > > release
> > > > > series for this release, but I may add that if it looks like I need help from
> > > > outside
> > > > > parties stabilizing releases in the future.
> > > >
> > > > Thanks for the release. I reinstall everything from scratch and let you
> > know if
> > > > I find some problem.
> > >
> > > Thanks.  I would especially like to know if the support for building a
> > container
> > > behind a proxy works right.   I wasn't able to test that.
> >
> > I just installed everything from scratch and didn't have any problems.
> Thanks.  This is really helpful.
No problem!
> > > > > Finally - I know there are pending patches on the mailing list (thanks
> > > > Daniel).
> > > > > I wanted to get this release out before going through those, as this
> > release
> > > > was
> > > > > already complicated enough.  Just by way of information, my upcoming
> > to-
> > > > do list is:
> > > > >  * figure out why there are two tests.info files, and hopefully get rid of
> > one
> > > > >  * evaluate Daniel's latest set of patches (from the last 2 weeks), and
> > > > provide
> > > > >     feedback and/or integrate them into 'next'.
> > > > >  * start work on standards for test pre check functionality (compare
> > with
> > > > 0day
> > > > >    and any other systems that do test pre check
> > > > >  * write a target scan and health_check test
> > > > >  * write the contract for the serial console transport feature
> > > >
> > > > Thanks, I was thinking about preparing a patch to replace tarballs by git
> > > > repositories (using the https protocol so they can be fetched behind
> > > > proxies).
> > > > What do you think about that? If you think there are more urgent
> > changes
> > > > to make let me know.
> > >
> > > I think this would be a nice addition.  Please don't replace tarballs, but just
> > > add the ability to process a git URL, in addition to the current support for
> > > de-tarring (and applying patches).  This feature would make Fuego similar
> > > to other build systems that can retrieve source via repositories, so we
> > might
> > > want to use the same terminology or conventions that other systems do
> > > (maybe copy (at least partially) what Yocto Project does, as some segment
> > > of our user base may have experience with that).
> > >
> > > One thing those systems support is specifying a git commit id to checkout,
> > > along with the repository URL.
> >
> > Actually, I was thinking about using the functionality built into jenkins.
> > Jenkins
> > allows monitoring git repositories and trigger a test when there is an update
> > for example.
> Hmmm.  OK.  I'll be interested to see what this solution looks like.
> To be honest, I'm trying to use as few Jenkins-isms as possible in the
> core script system, to allow people to use the script system with other
> front-ends (including their own proprietary front ends).  However, if
> Jenkins already has this capability, it seems like a good idea to leverage it.
I see. That's a bigger architectural decision that needs to be discussed.

My opinion is that if we are not going to use Jenkins functionality and plugins 
we might as well drop it. For example, we could just focus on the command 
line version (ftc) and then develop a reference front-end on a separate repository.

That front-end could be just Jenkins. But honestly, I don't think that Jenkins 
excels as a front end. The interface is hard to modify (oh Java!)  and not as flexible 
as one could wish. Plus I don't like the idea of using plugins because they come and 
go as they change the interface with new versions.

I know Fuego without Jenkins sounds a bit weird, but if we are going to take
that decision we should do it as soon as possible.

> One thing I worry about, with dynamic update of the test program
> source, is making sure we associate results with a particular version
> of the test source (and build environment).  It's important for regression
> testing to limit the number of variables (things that change) from
> one run to the next.  This is one reason that snapshotting the source is
> useful.  Having different source for test programs at different test
> node diminishes the ability to compare test results from different nodes.
> Or at least it makes it more complicated to determine the source of a
> regression, if there is one.

Right, that's a good point. Actually, using git repositories would solve that
because we can add the commit ID/Date/Author to the output log. Also, we can specify
the particular commit ID that we want to use (e.g. because we know the
test was stable at that point) when building a test that we don't want 
dynamic updates for.

> But I recognize that our current system of bundling the snapshotted
> source is cumbersome.
> > # By the way, because of the size of the tarballs the repository can't be
> > hosted on Github (on a free account).
> Yes - this is one of the results of snapshotting the source for the test programs.
> I've considered different mechanisms to deal with this.  One feature that's
> been requested, and that I'm working towards, is to use test programs already
> on target, from the distribution that's installed.  Another is to create a separate
> location (like a web or ftp server) where test programs (either source or binaries)
> can be downloaded from. But this last option adds more complexity to the image
> build step as well as further complicates the release management for Fuego.
> It adds yet another thing that must be maintained going forward, and something
> that we'd want to automate the management of.

hmm I don't think that tarballs or tests in binary format on a server is a good idea.
Git makes it easier to track which specific commit was used to build a test. And binaries
don't usually work unless they have been compiled statically. 
# Not to mention about the need to maintain that server.

I suggest we use (or mirror) upstream git repositories and create our own for those 
tests that are not distributed with git. Then we can use specific tags or commit ids for
snapshotting the source code of each test.

This has another side advantage: we can download the tests' source code in a "lazy" fashion
(i.e.: when the test is run for the first time) instead of having to download everything 
when installing Fuego from scratch. 
> As an aside, this solution look surprisingly like the one that Yocto Project
> had to create with their mirror system, when they faced similar issues with
> availability of the source for their packages.
> One question I have about this, is how often test programs themselves
> change, vs. the software and hardware that is being tested.  I haven't been
> doing this long enough to determine if things change rapidly or not.
> My suspicion is that some tests are quite stagnant, while other are seeing
> active development.  It is hard to determine how much effort will be
> required to keep test source and/or test source references up-to-date
> in Fuego.  Unfortunately, this is an issue that has been largely unaddressed
> until now.  So maybe a nice side effect of working on the dynamic source
> referencing will be to resolve some issues here.

The LTP test repository is updated every few days. I'm not sure about others
but it might be worth checking that when deciding whether using a specific
tag or forcing dynamic updates for a test.


IoT Technology center
Toshiba Corp. Industrial ICT solutions, 
E-mail:  daniel.sangorrin at toshiba.co.jp

More information about the Fuego mailing list