[Fuego] Patches for merge of Toshiba/Sony work

Bird, Timothy Tim.Bird at sony.com
Wed Mar 22 17:10:24 UTC 2017



> -----Original Message-----
> From: Daniel Sangorrin on Wednesday, March 22, 2017 2:00 AM
> Hi Tim,
> 
> > > -----Original Message-----
> > > From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> > >
> > > Daniel,
> > >
> > > Here are some patches for a bunch of work I've done on the merge of
> the Toshiba
> > > and Sony versions of Fuego.  This is based off of your bitbucket branch.
> ...
> > >
> > > The most significant change is a movement of where and how the logs
> are stored.
> > > All logs for a single run are stored in a single directory:
> > > /fuego-rw/logs/<testname>/<target>.<timestamp>.<build_number>/
> > >
> > > plot data is stored in /fuego-rw/logs/<testname> (plot.data, plot.png,
> etc.)
> 
> It seems that in the new version plot.png is linked. Since plot.png already
> conserves history I don't mind not having them separated but I think creating
> that link in functions.sh is a bit ugly. I will think about how to express that
> through
> the json extralinks property.
Good idea.   plot.png is really not a per-run concept, but something that
applies to multiple runs.  Jenkins should point to the plot.png, not the link.
Arguably, it would be better if the job page pointed at it, but it's quite handy
to have a 'build' page point to it also.
> 
> Yesterday, during the AGL meeting, you mentioned a method for
> distinguishing
> between logs from the same test/board but different specs/testplans.
> Could you explain it again if you don't mind?

Sorry - I wasn't clear in the meeting.  I think you are referring to what I was
saying about the job names.  Currently a job name is:
<board>.<testplan>.<test_name>

Jenkins puts build information (what I call a 'run') under:
/var/lib/jenkins/jobs/<jobname>/builds/<build_num>
(which is the equivalent of:
/var/lib/jenkins/jobs/<board>.<testplan>.<test_name>/builds/<build_num>

I put logs under /fuego-rw/logs/<testname>/<board>.<timestamp>.<build_num>

So there's a slight mismatch here.  I don't have the plan anywhere in my path,
and Jenkins doesn't have the timestamp.  It used to, but they switched BUILD_ID
from timestamp to build number.

In considering the tests, the things that should affect the results are obviously
the test, the board, and the spec.  The spec has a potential to have a dramatic
effect on the results.

I was thinking it would be better to have the spec name be part of the job
name, rather than the testplan name.  So the job name would be:
<board>.<spec>.<testname>

Even if multiple plans contains the same test, if they use the same spec
for that test, the results should be comparable.  Also, using the spec
for the job name means that a single plan can contain the same test, but
with different configurations (e.g. LTP with spec 'syscalls', or LTP with spec 'filesystems')
The spec is really an extension of the test name, whereas the plan is not.

I haven't figured out whether I would add the spec somewhere in the fuego
log path, but it seems like it would be a good idea.  Maybe something like:
/fuego-rw/logs/<test_name>/<spec>/<board>.<timestamp>.<build_num>

It doesn't make sense, IMHO, to plot benchmark data from the same test
but different specs in the same plot.  (well, maybe in some situations it would,
but as a general rule I think the spec essentially makes a different test).

I don't have a solution yet, but I'm thinking about this.

> > > I didn't fix anything about flot with this (although the dataload.py module
> > > has been updated with new paths for the metric data (for benchmarks)).
> 
> Flot isn't working for me. Did you overwrite the plugin?
I'll double-check this.  I don't think flot ever worked for me - I thought it was
broken for you.  I didn't change anything that I know.
> 
> > > ftc still has a few nagging issues with executing tests independent of a
> jenkins
> > > job (that is, from the command line).  However, most things seem to
> work.
> 
> I tried
> $ ftc run-test Benchmark.Dhrystone docker testplan_docker
> and it seems it worked.
Good.  There are a few issues that I didn't finish - like reading the
timeout from the testplan file, but it was mostly working for me.

> 
> However i tried
> $ ftc list-runs
> and got no runs. Is this supposed to be correct?
This is a command I didn't finish fixing.  I'll get to it shortly.
I was working on fixing up ftc, and then realized that the
higher priority was making sure all the existing Jenkins jobs
and workflow worked correctly. 

I saw that ftc list-runs was broken (but should be easy to fix).
I haven’t tested a bunch of the ftc sub-commands, but plan to 
work on those shortly.

> 
> > > A few of the changes I made to ovgen.py don't seem to have made it
> over
> > > to your repository.  If a test is missing a spec (including a default spec),
> > > it causes problems.  I'll try to look at that soon.  I don't think any of the
> > > tests specified in testplan_docker are affected by this, but I have
> problems running
> > > with the testplan_default plan.
> 
> Ok, sorry about that. I thought that testplan_default would be enough, but
> you are
> right because 3rd party tests shouldn't need to add an entry to
> testplan_default.
> 
> > > I didn't do anything with xlsx files, so I'm  pretty sure tools you have to
> create
> > > these files will need to be modified with the new log paths.
> 
> I haven't checked this yet.

Thanks for checking on things.  I'll send another e-mail to the list shortly with
a status update on the post_test changes.
 -- Tim



More information about the Fuego mailing list