[Fuego] Discussion about Fuego unified results format

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on Monday, April 24, 2017 11:34 PM> 
> > -----Original Message-----
> > From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> > Sent: Friday, April 14, 2017 4:19 AM
> > To: fuego at lists.linuxfoundation.org; automotive-
> discussions at lists.linuxfoundation.org
> > Cc: 'Cai, Song'; Daniel Sangorrin
> > Subject: Discussion about Fuego unified results format
> ... <removed for clarity>
> >
> > In any event, please be prepared to add a 'host' field to the schema for the
> metadata for
> > the test results.  I will be adding the 'host' definition to the global
> configuration for Fuego, so
> > it will be in the 'run.json' file, and can easily be transferred to the
> results.json file, if desired.
> 
> I haven't added it yet to results.json. Will it be an IP address or some kind of
> registered name?
It will be a registered name.  It will be defined in the user's local configuration
file, and will need to be registered with the central server, if the user wishes
to participate in the public fuego test network, and it will need to be globally unique
within that network.  For now, define it as "<unknown>".
I'll probably export it as FUEGO_HOSTNAME in the environment.

> 
> > We need to decide if we will put all the run meta-data into a single
> results.json file, or leave some
> > of it in the run.json file.  There are pros and cons to both approaches.  The
> run.json file is created
> > at the very end of a test, and it's nice to have it be a separate file from the
> results.json file, in terms
> > of division of responsibility for populating the files by the test tools and
> libraries provided by Fuego.
> > But duplicating fields is kind of annoying, as is having to manage the data
> for a run in two separate files.
> > I think this is a minor point that is easy to change, but we should discuss it
> before we finalize the
> > schema for both files for the 1.2 release.
> 
> I've been scratching my head about this. Except for the test duration (I'll add
> it to results.json),
> most of the information in run.json is already included in the logs (consolelog
> and prolog.sh).
Yes - but it's not machine readable, without a per-test parser.

> For now it doesn't hurt to have it, so I would leave it as it is (I have added
> testspec and fwver information).
> Probably we will need to revisit this when we address the "replayability"
> (what a word!) of Fuego tests.

Well, anything that is collected for every single test should probably go into
run.json.  The AGL tools were collecting these additional properties:
 BOARD_VER
 DEVICE_TEST
 DEVICE_FS

I'm not sure about BOARD_VER and DEVICE_TEST, but I'm guessing those
are duplicates of fields we already have.  DEVICE_FS is so commonly interesting
(even for non-filesystem tests) that it's probably worth recording for every test.

One stray thought.  I'm not liking Jenkins and KernelCIs use of 'seconds' for the start time.
This is not human-friendly.  I much prefer a human-readable number like our timestamp.
I believe each test should have a globally unique, human-readable identifier.  This is useful
for when we send test results between machines, and is why I dislike that Jenkins eliminated
timestamps from their latest releases.  BUILD_NUMBER is not globally unique.  The string
I use for test runs is: "run-<testname>-<timestamp>-on-<host>:<board>"

 -- Tim