[Fuego] JSON schema and examples for Fuego's run.json format

Bird, Timothy Tim.Bird at sony.com
Fri Jun 30 00:47:57 UTC 2017



> -----Original Message-----
> From: Daniel Sangorrin [mailto:daniel.sangorrin at toshiba.co.jp]
> Sent: Thursday, June 29, 2017 5:24 PM
> To: Bird, Timothy <Tim.Bird at sony.com>; 'Milo Casagrande'
> <milo.casagrande at linaro.org>; fuego at lists.linuxfoundation.org
> Subject: RE: [Fuego] JSON schema and examples for Fuego's run.json format
> 
> Hi Tim,
> 
> Thanks for your review. Before I modify the schema I want to confirm
> a few items.
> 
> 1) tguids
> 
> > >         "tguid":{
> > >             "type":"string",
> > >             "description":"Fuego's test globally unique identifier"
> > >         },
> >
> > Hey - thanks!  But this would only be a partial tguid.  Each individual
> measurement
> > should be able to be identified by a fully-qualified tguid (which maybe
> consists
> > of <this-tguid>.<test_set>.<test_case>.<measurement>
> ...
> > >     "test_case":[
> > >         {
> > >             "tguid":"ab2344cfa34546564s234acfddddaaa2342",
> >
> > I want the tguid to be human readable, and invariant over runs.
> > In this case, I think the tguid for the single-measurement item
> > should be "Dhrystone.Dhrystone"
> ..
> > >                     "tguid":"123a34546564s234acfddddaaa2342",
> > >                     "name":"Create",
> >
> > tguid would be: "bonnie.Random_Create.Create"
> >
> > I think we can synthesize the tguid from the name fields.  If it's
> unambiguous
> > how to generate an invariant name for a test case/measure, then there's
> > no need to have a separate field for the tguid.
> >
> > The idea with TGUIDs is that we can compare them between test systems.
> > As an example, I'd like to be able to compare a 0-day Dhrystone.Dhrystone
> value
> > for the latest kernel with a Fuego Dhrystone.Dhrystone value for the LTSI
> > kernel.
> 
> OK - based on your comments, I think that I should remove all "tguid" fields
> from the schema and
> instead enforce (with the 'required' token) that all test
> suites/sets/cases/measures have
> a name. This way we can just infer the TGUIDs from the names.
> What do you think?

Agreed.  I'm a bit worried about names not being globally invariant, based
on supporting multiple "paths" to the testcase, depending on the structure
of the tests and optional nesting of test_sets.
e.g. Could LTP.syscall.kill10 end up being the same as LTP.RT-tests.functions.kill10?
or could some super-test run LTP tests, resulting in a different string for the
same test, like so: OpenTest.LTP.syscall.kill10?

I'd say that for now, just make 'name' required, and we'll see what issues crop up
and we start using it.

> 
> Milo (if you read this): do you see any specific reason for using a hash in
> Fuego as in Kernel CI?

What is the hash used for in KernelCI?  It almost looks like a per-run value, instead
of a per-test value.

> 
> 2) timestamp format
> 
> > >                 "start_time":{
> > >                     "type":"string",
> > >                     "description":"timestamp (YYYY-MM-ddThh:mm:SSZ) when the
> > > test suite execution was started",
> > >                     "format":"date-time"
> >
> > This is a change, but it appears to be iso 8601-compliant.
> > I'd rather not be required to express the start time in UTC.  I think
> > we should allow TimeZone offsets here (maybe you do).  I like to
> > see times in a locally-relevant time format.  This is one of my chief
> > gripes about Jenkins.
> 
> > >         "start_time":"2017-06-26T09:03:04Z",
> > Is this actually in UTC, rather than localtime?
> > Where are you getting this from? date? python time.time?
> 
> The "date-time" format is a built-in format in the JSON schema specification.
> It is
> explained at [1] and can be used both in UTC and in local time.
> 
> UTC example from [1]: 1985-04-12T23:20:50.52Z
>    This represents 20 minutes and 50.52 seconds after the 23rd hour of
>    April 12th, 1985 in UTC.
> 
> Local time example from [1]: 1996-12-19T16:39:57-08:00
>    This represents 39 minutes and 57 seconds after the 16th hour of
>    December 19th, 1996 with an offset of -08:00 from UTC (Pacific
>    Standard Time).  Note that this is equivalent to 1996-12-20T00:39:57Z
>    in UTC.
> 
> In python you can get local time in two ways:
> 
> With tzlocal (pip install tzlocal)
>     import tzlocal, datetime
>     d = datetime.datetime.now(tz=tzlocal.get_localzone())
>     print d.isoformat('T')
>         2017-06-30T09:13:27.507494+09:00
> 
> With dateutil
>     import dateutil.tz, datetime
>     d = datetime.datetime.now(dateutil.tz.tzlocal())
>     print d.isoformat('T')
>         2017-06-30T09:13:27.507494+09:00
> 
> [1] https://tools.ietf.org/html/rfc3339

The RFC3339 spec only allows 1 digit of fractional seconds,
according to my reading of the spec.  I believe ISO8601 allows
more fractional seconds digits, but I'd like to stick to a strict
subset of ISO8601 for the reasons mentioned in RFC3339.  I dug
into this today (in probably too much gory detail)
and Python and C support for RFC3339 is kind of messed up.

See my notes on this at:
http://bird.org/fuego/Issue_0047

I'm working on a patch that uses timestamps in the following format:
$ date +F_%T%z
2017-06-29_17:44:25-0700

This is not strictly compliant with RFC3339, so might be different than what
kernelCI expects, but I have reasons for my deviations. 
 
> 
> 3) Others
> 
> > >     "metadata":{
> > >         "fuego_version":"0803ee7a2c5a59af159c01f561fc8d30d3d40e75",
> > Can we do these with 'git describe' strings instead?
> > ex:  "fuego_version":"v1.1-24-g5ad677b"
> 
> Sure, JSON schema allows expressing the format with regular expressions.
> Should I add one?

Just a raw string is fine.  I don't want to impose too much structure.
I would just like to have it be more human readable than a raw hash.

 -- Tim



More information about the Fuego mailing list