[Fuego] status update
daniel.sangorrin at toshiba.co.jp
Wed Jun 28 09:02:38 UTC 2017
> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> Sent: Wednesday, June 28, 2017 2:23 PM
> To: Bird, Timothy; Daniel Sangorrin; fuego at lists.linuxfoundation.org
> Subject: RE: [Fuego] status update
> > -----Original Message-----
> > From: Bird, Timothy on Tuesday, June 27, 2017 9:56 PM
> > > -----Original Message-----
> > > From: Daniel Sangorrin on Tuesday, June 27, 2017 6:58 PM
> > > > -----Original Message-----
> > > > From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> > > > Sent: Wednesday, June 28, 2017 9:47 AM
> > > In Fuego, we have a similar concept. If you see bonnie's reference.log 
> > you
> > > will
> > > notice that there are several groups/test_sets and multiple
> > tests/test_cases
> > > inside
> > > each group:
> > >
> > > test_set: Sequential_Output
> > > test_cases: Block, PerChr, Rewrite
> > In this case, is Block a set of measurements, or a single measurement?
> OK - it's a single measurement.
> Ugh. I was confused. reference.log is not the file I was thinking of.
> reference.log is the file that holds thresholds for all of the benchmark
> test measurements. I was thinking of the 'pn log'.
Ouch, I had already forgotten about this guys.
> All the stuff I said about placement of reference.log (well most of it), I meant
> in regard to pn log files.
> OK, getting back to reference.log...
> First, it's terribly named. It's not a 'log' by any stretch of the imagination.
> It's also terribly formatted. It's some weird custom format created just to be
> easy to read and write with 'awk'. It does need to specify all possible measurement
> names (that are relevant for determining benchmark pass or fail status) This
> is usually a small subset of all the measurements made by a benchmark.
> And, I'm not sure it belongs in the test directory, as it should be writable by the
> user to reflect their own notion of values that indicate a regression threshold
> for a measurement.
> However, having said that, I'm not sure I want to change it this release.
OK, that's what I wanted to know. For now, I will stick to what we have.
> Please note one other thing, though, related to test schema and test_cases and
> I strongly believe that every test case/measurement in the system should have
> something that I call a TGUID - a test globally unique identifier (something like an URL in
> World-wide-web parlance). This is a single string that can be used to
> identify a particular measurement from all other measurements. This should be
> able to be constructed using a path-like syntax. Personally, I would use dots
> as delimiters, like bonnie's reference log does (rather than slashes, like URLs or
> filepaths use).
> It is important, IMO, to have a static test identifier, for referring to the test
> independent of host, board, test_plan or spec:
> e.g. "LTP.syscall.kill10"
> We should be able to construct these fully-qualified test identifiers using values
> in the json schemas for a run.
> The reason for having TGUIDs is that I would like for people to be able to share
> meta-data about individual measurements, in a way that is independent of other
> test infrastructure or layout.
In kernel CI they ask for this ID to the centralized server, before submitting the json files.
We can do as them, or create our own identifiers by combining several words.
More information about the Fuego