[Fuego] Patches for merge of Toshiba/Sony work

Bird, Timothy Tim.Bird at sony.com
Fri Mar 24 01:53:54 UTC 2017


> -----Original Message-----
> From: Daniel Sangorrin on Thursday, March 23, 2017 5:40 PM
> > -----Original Message-----
> > From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> > Sent: Thursday, March 23, 2017 2:10 AM
> > > It seems that in the new version plot.png is linked. Since plot.png already
> > > conserves history I don't mind not having them separated but I think
> creating
> > > that link in functions.sh is a bit ugly. I will think about how to express that
> > > through
> > > the json extralinks property.
> > Good idea.   plot.png is really not a per-run concept, but something that
> > applies to multiple runs.  Jenkins should point to the plot.png, not the link.
> > Arguably, it would be better if the job page pointed at it, but it's quite
> handy
> > to have a 'build' page point to it also.
> 
> Yeah, you're right. It should be in the build page.
> 
> I wonder if we can add a pdf/svg export button to the flot plugin. That
> would create a vector format plot instead of the current png.
That would be nice to have.  I'm not sure how jquery can write to
local files, though, or would it just create it and let a user save
the image from their browser.

> > > Yesterday, during the AGL meeting, you mentioned a method for
> > > distinguishing
> > > between logs from the same test/board but different specs/testplans.
> > > Could you explain it again if you don't mind?
> >
> > Sorry - I wasn't clear in the meeting.  I think you are referring to what I was
> > saying about the job names.  Currently a job name is:
> > <board>.<testplan>.<test_name>
> >
> > Jenkins puts build information (what I call a 'run') under:
> > /var/lib/jenkins/jobs/<jobname>/builds/<build_num>
> > (which is the equivalent of:
> >
> /var/lib/jenkins/jobs/<board>.<testplan>.<test_name>/builds/<build_num
> >
> >
> > I put logs under /fuego-
> rw/logs/<testname>/<board>.<timestamp>.<build_num>
> >
> > So there's a slight mismatch here.  I don't have the plan anywhere in my
> path,
> > and Jenkins doesn't have the timestamp.  It used to, but they switched
> BUILD_ID
> > from timestamp to build number.
> >
> > In considering the tests, the things that should affect the results are
> obviously
> > the test, the board, and the spec.  The spec has a potential to have a
> dramatic
> > effect on the results.
> >
> > I was thinking it would be better to have the spec name be part of the job
> > name, rather than the testplan name.  So the job name would be:
> > <board>.<spec>.<testname>
> 
> Yes, I agree. That's a better solution.
I think so, but I'm hesitating.  There are a few places where Fuego uses
testplan, that I'm not sure would be well-served if we only had the spec.
I might try supporting 'ftc run-test -t <testname> -b <board> -s <spec>
and see what issues that creates.

> 
> > Even if multiple plans contains the same test, if they use the same spec
> > for that test, the results should be comparable.  Also, using the spec
> > for the job name means that a single plan can contain the same test, but
> > with different configurations (e.g. LTP with spec 'syscalls', or LTP with spec
> 'filesystems')
> > The spec is really an extension of the test name, whereas the plan is not.
> 
> Yes, you are right.
> 
> > I haven't figured out whether I would add the spec somewhere in the
> fuego
> > log path, but it seems like it would be a good idea.  Maybe something like:
> > /fuego-
> rw/logs/<test_name>/<spec>/<board>.<timestamp>.<build_num>
> 
> Looks good to me.
> 
> If possible I'd rather have the "timestamp" on a file under the <build_num>
> directory
> to decrease the amount of links.
I'm not sure what you mean.  Something like this:
/fuego-rw/logs/<test_name>/<spec>/<board>.<build_num>/<timestamp>.testlog.txt, <timestamp>.devlog.txt, etc. ?

I would prefer not having the timestamp on individual files.
For some reason (unknown to me) Jenkins backed away from using the timestamp to uniquely identify tests.
I think it's quite handy.  But since Jenkins doesn't provide it to the script, it doesn't know what it is, and can't use
that to access the log directory.

If I can do the link setting from the base script, then Jenkins doesn't have to know the timestamp to create
the path to the logs.  I don't know much about the description-setter, but maybe it can be smarter about
how it creates the links (can it run arbitrary code to get the timestamp for the run?)

> 
> > It doesn't make sense, IMHO, to plot benchmark data from the same test
> > but different specs in the same plot.  (well, maybe in some situations it
> would,
> > but as a general rule I think the spec essentially makes a different test).
> 
> I tend to disagree here.
> 
> I think that comparing the results of a test using different parameters
> can be useful in several scenarios. For example, suppose that you have a
> network
> test and you want to compare a spec that uses normal frames with another
> one
> that uses jumbo frames. Then, suppose you find that jumbo frames are
> much
> slower than normal ones. That may indicate a problem with a driver
> or with the kernel configuration for example.

OK - that's a good use case.  I agree.
 
> At the moment <plot.data> includes information about the board,
> architecture, and kernel.
> Adding an entry indicating the spec's name should be trivial.
I agree.

> What I'm not sure about is how Fuego users should use this functionality.
> I think this is kind of advanced functionality so one option is to restrict it to
> the command line interface so that normal users do not care about it:
> 
> $ ftc compare --testname mytest --spec default --spec optimized
>   -> matplotlib's plot comparing the test results for different specs
> 
> $ ftc compare --testname mytest --spec default --board bbb --board raspi
>   -> matplotlib's plot comparing the test results for different boards

This would be neat - to be able to choose the "axis" to compare.
 
> > I don't have a solution yet, but I'm thinking about this.
> >
> > > > > I didn't fix anything about flot with this (although the dataload.py
> module
> > > > > has been updated with new paths for the metric data (for
> benchmarks)).
> > >
> > > Flot isn't working for me. Did you overwrite the plugin?
> > I'll double-check this.  I don't think flot ever worked for me - I thought it
> was
> > broken for you.  I didn't change anything that I know.
> 
> I fixed flot and was working for me fine. I will fix it and send you a pull
> request.

In thinking about this, since I moved the log directories around
that may be what broke.  I didn't change the flot plugin at all,
and it probably can't find where I put the logs (which is where
the json files with the metrics data are).

 -- Tim


More information about the Fuego mailing list