[Fuego] JSON schema and examples for Fuego's run.json format

[Daniel Sangorrin]

Hi Tim,

Thanks for your review. Before I modify the schema I want to confirm
a few items.

1) tguids

> >         "tguid":{
> >             "type":"string",
> >             "description":"Fuego's test globally unique identifier"
> >         },
> 
> Hey - thanks!  But this would only be a partial tguid.  Each individual measurement
> should be able to be identified by a fully-qualified tguid (which maybe consists
> of <this-tguid>.<test_set>.<test_case>.<measurement>
...
> >     "test_case":[
> >         {
> >             "tguid":"ab2344cfa34546564s234acfddddaaa2342",
> 
> I want the tguid to be human readable, and invariant over runs.
> In this case, I think the tguid for the single-measurement item
> should be "Dhrystone.Dhrystone"
..
> >                     "tguid":"123a34546564s234acfddddaaa2342",
> >                     "name":"Create",
> 
> tguid would be: "bonnie.Random_Create.Create"
> 
> I think we can synthesize the tguid from the name fields.  If it's unambiguous
> how to generate an invariant name for a test case/measure, then there's
> no need to have a separate field for the tguid.
> 
> The idea with TGUIDs is that we can compare them between test systems.
> As an example, I'd like to be able to compare a 0-day Dhrystone.Dhrystone value
> for the latest kernel with a Fuego Dhrystone.Dhrystone value for the LTSI
> kernel.

OK - based on your comments, I think that I should remove all "tguid" fields from the schema and
instead enforce (with the 'required' token) that all test suites/sets/cases/measures have
a name. This way we can just infer the TGUIDs from the names.
What do you think?

Milo (if you read this): do you see any specific reason for using a hash in Fuego as in Kernel CI?

2) timestamp format

> >                 "start_time":{
> >                     "type":"string",
> >                     "description":"timestamp (YYYY-MM-ddThh:mm:SSZ) when the
> > test suite execution was started",
> >                     "format":"date-time"
> 
> This is a change, but it appears to be iso 8601-compliant.
> I'd rather not be required to express the start time in UTC.  I think
> we should allow TimeZone offsets here (maybe you do).  I like to
> see times in a locally-relevant time format.  This is one of my chief
> gripes about Jenkins.

> >         "start_time":"2017-06-26T09:03:04Z",
> Is this actually in UTC, rather than localtime?
> Where are you getting this from? date? python time.time?

The "date-time" format is a built-in format in the JSON schema specification. It is
explained at [1] and can be used both in UTC and in local time.

UTC example from [1]: 1985-04-12T23:20:50.52Z
   This represents 20 minutes and 50.52 seconds after the 23rd hour of
   April 12th, 1985 in UTC.

Local time example from [1]: 1996-12-19T16:39:57-08:00
   This represents 39 minutes and 57 seconds after the 16th hour of
   December 19th, 1996 with an offset of -08:00 from UTC (Pacific
   Standard Time).  Note that this is equivalent to 1996-12-20T00:39:57Z
   in UTC.

In python you can get local time in two ways:

With tzlocal (pip install tzlocal)
    import tzlocal, datetime
    d = datetime.datetime.now(tz=tzlocal.get_localzone())
    print d.isoformat('T')
        2017-06-30T09:13:27.507494+09:00

With dateutil
    import dateutil.tz, datetime
    d = datetime.datetime.now(dateutil.tz.tzlocal())
    print d.isoformat('T')
        2017-06-30T09:13:27.507494+09:00

[1] https://tools.ietf.org/html/rfc3339


3) Others

> >     "metadata":{
> >         "fuego_version":"0803ee7a2c5a59af159c01f561fc8d30d3d40e75",
> Can we do these with 'git describe' strings instead?
> ex:  "fuego_version":"v1.1-24-g5ad677b"
 
Sure, JSON schema allows expressing the format with regular expressions.
Should I add one?

> >         "fuego_core_version":"b98ae34f3c2199c46cc229ca209fc39faded938b",
> >         "fuego_core_version":"b98ae34f3c2199c46cc229ca209fc39faded938b",
> 
> 
> I assume this duplication is a typo.

Yes, it meant to be testsuite_version.

Thanks,
Daniel