[Fuego] JSON schema and examples for Fuego's run.json format

Daniel Sangorrin daniel.sangorrin at toshiba.co.jp
Fri Jun 30 03:17:02 UTC 2017


Hi Tim,

Thanks for the review. Here is the new version. Hopefully I included
everything you mentioned.

Note: this time I have used measures at [1] for redacting bonnie's results. By the
way I noticed that bonnie is also a great example for the need of measurements.

I have enforced the "name" property for the test suite, the test sets,
the test cases and the measurements. If we put a rule saying that
"names" should not contain dots ('.') then we can create a TGUI that
makes sense such as:

Benchmark.bonnie. Sequential_Output.PerChr.speed

For simple tests, it's a bit cumbersome but we just repeat the test suite name:

Benchmark.Dhrystone. Dhrystone. Dhrystone

What do you think?

Thanks,
Daniel

[1] http://support.commgate.net/index.php?/Knowledgebase/Article/View/212

*** fuego-schema.json ***

{
    "$schema":"http://json-schema.org/schema#",
    "id":"http://www.fuegotest.org/download/fuego_schema_v1.0.json",
    "title":"test_suite",
    "description":"A test suite JSON object corresponding to a single Fuego job run or execution",
    "definitions":{
        "status":{
            "type":"string",
            "description":"The status of the execution of this test case",
            "enum":[
                "PASS",
                "FAIL",
                "SKIP",
                "ERROR"
            ]
        },
        "measurement":{
            "title":"measurement",
            "description":"A measurement registered by a test case",
            "type":"object",
            "properties":{
                "name":{
                    "type":"string",
                    "description":"The name given to this measurement"
                },
                "unit":{
                    "type":"string",
                    "description":"The unit of this measurement"
                },
                "measure":{
                    "type":[
                        "string",
                        "number",
                        "integer"
                    ],
                    "description":"The data measured during the test case execution; the value will be interpreted based on the $unit field"
                },
                "status":{
                    "$ref":"#/definitions/status"
                },
                "reference":{
                    "type":"object",
                    "description":"Reference measure that is compared to a measure to decide the status",
                    "properties":{
                        "threshold":{
                            "type":[
                                "string",
                                "number",
                                "integer"
                            ],
                            "description":"A threshold value"
                        },
                        "comparison":{
                            "type":"string",
                            "description":"Type of comparison",
                            "enum":[
                                "eq",
                                "ne",
                                "gt",
                                "ge",
                                "lt",
                                "le"
                            ]
                        }
                    },
                    "required":[
                        "threshold",
                        "comparison"
                    ]
                }
            },
            "required":[
                "name",
                "measure"
            ]
        },
        "attachment":{
            "title":"attachment",
            "description":"the location of a file",
            "type":"object",
            "properties":{
                "name":{
                    "type":"string",
                    "description":"The name given to the attachment"
                },
                "server_uri":{
                    "type":"string",
                    "description":"A URI that points to this attachment"
                },
                "path":{
                    "type":"string",
                    "description":"A path in the test results package that identifies this attachment"
                }
            },
            "required":[
                "path"
            ]
        },
        "test_case":{
            "type":"object",
            "description":"A test case is the smallest test program",
            "properties":{
                "name":{
                    "type":"string",
                    "description":"The name given to this test case"
                },
                "status":{
                    "$ref":"#/definitions/status"
                },
                "duration_ms":{
                    "type":"number",
                    "description":"The number of ms it took to execute this test case"
                },
                "measurements":{
                    "type":"array",
                    "description":"Array of measurement objects registered by this test case",
                    "items":{
                        "$ref":"#/definitions/measurement"
                    }
                },
                "attachments":{
                    "type":"array",
                    "description":"List of attachment objects produced by this test case",
                    "items":{
                        "$ref":"#/definitions/attachment"
                    },
                    "additionalItems":true
                }
            },
            "required":[
                "name",
                "status"
            ]
        },
        "test_set":{
            "title":"test_set",
            "description":"A test set is a group of test cases",
            "type":"object",
            "properties":{
                "name":{
                    "type":"string",
                    "description":"The name given to this test set"
                },
                "duration_ms":{
                    "type":"number",
                    "description":"The number of ms it took to execute the test set"
                },
                "test_case":{
                    "type":"array",
                    "description":"The list of test case objects executed by this test set",
                    "items":{
                        "$ref":"#/definitions/test_case"
                    }
                },
                "test_set":{
                    "type":"array",
                    "description":"Nested test set",
                    "items":{
                        "$ref":"#/definitions/test_set"
                    }
                }
            },
            "required":[
                "name"
            ]
        }
    },
    "type":"object",
    "properties":{
        "schema_version":{
            "type":"string",
            "description":"The version number of this JSON schema",
            "enum":[
                "1.0"
            ]
        },
        "name":{
            "type":"string",
            "description":"The name given to this test suite"
        },
        "status":{
            "$ref":"#/definitions/status"
        },
        "duration_ms":{
            "type":"number",
            "description":"The number of ms it took to execute the entire test suite"
        },
        "test_set":{
            "type":"array",
            "description":"The list of test set objects, executed by this test suite",
            "items":{
                "$ref":"#/definitions/test_set"
            }
        },
        "test_case":{
            "type":"array",
            "description":"List of test case objects when they do not belong to a test set",
            "items":{
                "$ref":"#/definitions/test_case"
            }
        },
        "metadata":{
            "type":"object",
            "description":"Object to store Fuego parameters and other accessory data",
            "properties":{
                "fuego_version":{
                    "type":"string",
                    "description":"The version of fuego (e.g. v1.1-9234a234)"
                },
                "fuego_core_version":{
                    "type":"string",
                    "description":"The version of fuego core (e.g. v1.1-234544ac)"
                },
                "testsuite_version":{
                    "type":"string",
                    "description":"string that identifies the testsuite source code (e.g. repo:commit_id, tarball version..)"
                },
                "host_name":{
                    "type":"string",
                    "description":"The name of the host executing this test suite"
                },
                "board_type":{
                    "type":"string",
                    "description":"The name of the board"
                },
                "board_instance":{
                    "type":"string",
                    "description":"The instance identifier of the board (for LAVA)"
                },
                "compiled_on":{
                    "type":"string",
                    "description":"The machine where the test was compiled"
                },
                "job_name":{
                    "type":"string",
                    "description":"The name of the job"
                },
                "kernel_version":{
                    "type":"string",
                    "description":"The version of the kernel running on the board"
                },
                "start_time":{
                    "type":"string",
                    "description":"timestamp (e.g.: 2017-06-29_17:44:25-0700) when the test suite execution was started"
                },
                "test_plan":{
                    "type":"string",
                    "description":"The test plan used"
                },
                "test_spec":{
                    "type":"string",
                    "description":"The test spec used"
                },
                "build_number":{
                    "type":"number",
                    "description":"The build number"
                },
                "keep_log":{
                    "type":"boolean",
                    "description":"The keep_log flag"
                },
                "reboot":{
                    "type":"boolean",
                    "description":"The reboot flag"
                },
                "rebuild":{
                    "type":"boolean",
                    "description":"The rebuild flag"
                },
                "target_precleanup":{
                    "type":"boolean",
                    "description":"The target_precleanup flag"
                },
                "target_postcleanup":{
                    "type":"boolean",
                    "description":"The target_postcleanup flag"
                },
                "workspace":{
                    "type":"string",
                    "description":"The workspace path"
                },
                "attachments":{
                    "type":"array",
                    "description":"List of attachment objects produced by this test suite",
                    "items":{
                        "$ref":"#/definitions/attachment"
                    },
                    "additionalItems":true
                }
            },
            "required":[
                "fuego_version",
                "host_name",
                "board_type",
                "job_name",
                "test_spec",
                "start_time"
            ]
        }
    },
    "required":[
        "schema_version",
        "name",
        "status",
        "metadata"
    ]
}

*** fuego-run-example-bonnie.json ***

{
    "schema_version":"1.0",
    "name":"Benchmark.bonnie",
    "status":"PASS",
    "duration_ms":118352,
    "test_set":[
        {
            "name":"Sequential_Output",
            "status":"PASS",
            "test_case":[
                {
                    "name":"Block",
                    "status":"PASS",
                    "measurements":[
                        {
                            "name":"speed",
                            "measure":31483,
                            "unit":"K/Sec"
                        },
                        {
                            "name":"CPU",
                            "measure":43,
                            "unit":"%CPU"
                        },
                        {
                            "name":"latency",
                            "measure":770,
                            "unit":"ms"
                        }
                    ]
                },
                {
                    "name":"PerChr",
                    "status":"PASS",
                    "measurements":[
                        {
                            "name":"speed",
                            "measure":400,
                            "unit":"K/Sec"
                        },
                        {
                            "name":"CPU",
                            "measure":98,
                            "unit":"%CPU"
                        },
                        {
                            "name":"latency",
                            "measure":24636,
                            "unit":"us"
                        }
                    ]
                },
                {
                    "name":"Rewrite",
                    "status":"PASS",
                    "measurements":[
                        {
                            "name":"speed",
                            "measure":35716,
                            "unit":"K/Sec"
                        },
                        {
                            "name":"CPU",
                            "measure":31,
                            "unit":"%CPU"
                        },
                        {
                            "name":"latency",
                            "measure":686,
                            "unit":"ms"
                        }
                    ]
                }
            ]
        },
        {
            "name":"Sequential_Input",
            "status":"PASS",
            "test_case":[
                {
                    "name":"PerChr",
                    "status":"PASS",
                    "measurements":[
                        {
                            "name":"speed",
                            "measure":31928,
                            "unit":"K/Sec"
                        },
                        {
                            "name":"CPU",
                            "measure":97,
                            "unit":"%CPU"
                        },
                        {
                            "name":"latency",
                            "measure":57343,
                            "unit":"us"
                        }
                    ]
                },
                {
                    "name":"Block",
                    "status":"PASS",
                    "measurements":[
                        {
                            "name":"speed",
                            "measure":110644,
                            "unit":"K/Sec"
                        },
                        {
                            "name":"CPU",
                            "measure":36,
                            "unit":"%CPU"
                        },
                        {
                            "name":"latency",
                            "measure":50657,
                            "unit":"us"
                        }
                    ]
                }
            ]
        },
        {
            "name":"Random_Seeks",
            "status":"PASS",
            "test_case":[
                {
                    "name":"Random_Seeks",
                    "status":"PASS",
                    "measurements":[
                        {
                            "name":"speed",
                            "measure":136.3,
                            "unit":"/Sec"
                        },
                        {
                            "name":"CPU",
                            "measure":792,
                            "unit":"%CPU"
                        },
                        {
                            "name":"latency",
                            "measure":1289,
                            "unit":"ms"
                        }
                    ]
                }
            ]
        },
        {
            "name":"Sequential_Create",
            "status":"PASS",
            "test_case":[
                {
                    "name":"Create",
                    "status":"PASS",
                    "measurements":[
                        {
                            "name":"speed",
                            "measure":23913,
                            "unit":"/sec"
                        },
                        {
                            "name":"CPU",
                            "measure":60,
                            "unit":"%CPU"
                        },
                        {
                            "name":"latency",
                            "measure":1312,
                            "unit":"us"
                        }
                    ]
                },
                {
                    "name":"Read",
                    "status":"PASS",
                    "measurements":[
                        {
                            "name":"latency",
                            "measure":1327,
                            "unit":"us"
                        }
                    ]
                },
                {
                    "name":"Delete",
                    "status":"PASS",
                    "measurements":[
                        {
                            "name":"latency",
                            "measure":1093,
                            "unit":"us"
                        }
                    ]
                }
            ]
        },
        {
            "name":"Random_Create",
            "status":"PASS",
            "test_case":[
                {
                    "name":"Create",
                    "status":"PASS",
                    "measurements":[
                        {
                            "name":"speed",
                            "measure":31928,
                            "unit":"/Sec"
                        },
                        {
                            "name":"CPU",
                            "measure":64,
                            "unit":"%CPU"
                        },
                        {
                            "name":"latency",
                            "measure":346,
                            "unit":"us"
                        }
                    ]
                },
                {
                    "name":"Read",
                    "status":"PASS",
                    "measurements":[
                        {
                            "name":"latency",
                            "measure":126,
                            "unit":"us"
                        }
                    ]
                },
                {
                    "name":"Delete",
                    "status":"PASS",
                    "measurements":[
                        {
                            "name":"latency",
                            "measure":685,
                            "unit":"us"
                        }
                    ]
                }
            ]
        }
    ],
    "metadata":{
        "fuego_version":"v1.1-5ad677b",
        "fuego_core_version":"v1.1-805adb0",
        "testsuite_version":"bonnie++-1.03e.tar.gz",
        "host_name":"daniel_machine_3",
        "board_type":"bbb",
        "compiled_on":"daniel_machine_3.docker",
        "job_name":"bbb.default.Benchmark.bonnie",
        "kernel_version":"3.8.13-bone79",
        "test_plan":"none",
        "test_spec":"default",
        "start_time":"2017-06-30_11:53:50+0900",
        "toolchain":"qemu-armv7hf",
        "build_number":1,
        "keep_log":true,
        "reboot":false,
        "rebuild":true,
        "target_precleanup":true,
        "target_postcleanup":true,
        "workspace":"/fuego-rw/buildzone",
        "attachments":[
            {
                "name":"devlog",
                "path":"devlog.txt"
            },
            {
                "name":"devlog",
                "path":"devlog.txt"
            },
            {
                "name":"syslog.before",
                "path":"syslog.before.txt"
            },
            {
                "name":"syslog.after",
                "path":"syslog.after.txt"
            },
            {
                "name":"testlog",
                "path":"testlog.txt"
            },
            {
                "name":"consolelog",
                "path":"consolelog.txt"
            },
            {
                "name":"test_spec",
                "path":"spec.json"
            }
        ]
    }
}

*** fuego-run-example-dhryston.json ***

{
    "schema_version":"1.0",
    "name":"Benchmark.Dhrystone",
    "status":"PASS",
    "duration_ms":13425,
    "test_case":[
        {
            "name":"Dhrystone",
            "status":"PASS",
            "duration_ms":13400,
            "measurements":[
                {
                    "name": "Dhrystone",
                    "measure":1111111.1,
                    "unit":"Dhrystones per Second",
                    "status":"PASS",
                    "reference":{
                        "threshold":"1",
                        "comparison":"gt"
                    }
                }
            ]
        }
    ],
    "metadata":{
        "fuego_version":"v1.1-5ad677b",
        "fuego_core_version":"v1.1-805adb0",
        "testsuite_version":"Dhrystone.tar.bz2 with dhry_1.c.patch",
        "host_name":"daniel_machine_3",
        "board_type":"bbb",
        "compiled_on":"daniel_machine_3.docker",
        "job_name":"bbb.default.Benchmark.Dhrystone",
        "kernel_version":"3.8.13-bone79",
        "test_plan":"none",
        "test_spec":"default",
        "start_time":"2017-06-30_11:53:50+0900",
        "toolchain":"qemu-armv7hf",
        "build_number":4,
        "keep_log":true,
        "reboot":false,
        "rebuild":true,
        "target_precleanup":true,
        "target_postcleanup":true,
        "workspace":"/fuego-rw/buildzone",
        "attachments":[
            {
                "name":"devlog",
                "path":"devlog.txt"
            },
            {
                "name":"devlog",
                "path":"devlog.txt"
            },
            {
                "name":"syslog.before",
                "path":"syslog.before.txt"
            },
            {
                "name":"syslog.after",
                "path":"syslog.after.txt"
            },
            {
                "name":"testlog",
                "path":"testlog.txt"
            },
            {
                "name":"consolelog",
                "path":"consolelog.txt"
            },
            {
                "name":"test_spec",
                "path":"spec.json"
            }
        ]
    }
}







> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> Sent: Friday, June 30, 2017 9:48 AM
> To: Daniel Sangorrin; 'Milo Casagrande'; fuego at lists.linuxfoundation.org
> Subject: RE: [Fuego] JSON schema and examples for Fuego's run.json format
> 
> 
> 
> > -----Original Message-----
> > From: Daniel Sangorrin [mailto:daniel.sangorrin at toshiba.co.jp]
> > Sent: Thursday, June 29, 2017 5:24 PM
> > To: Bird, Timothy <Tim.Bird at sony.com>; 'Milo Casagrande'
> > <milo.casagrande at linaro.org>; fuego at lists.linuxfoundation.org
> > Subject: RE: [Fuego] JSON schema and examples for Fuego's run.json format
> >
> > Hi Tim,
> >
> > Thanks for your review. Before I modify the schema I want to confirm
> > a few items.
> >
> > 1) tguids
> >
> > > >         "tguid":{
> > > >             "type":"string",
> > > >             "description":"Fuego's test globally unique identifier"
> > > >         },
> > >
> > > Hey - thanks!  But this would only be a partial tguid.  Each individual
> > measurement
> > > should be able to be identified by a fully-qualified tguid (which maybe
> > consists
> > > of <this-tguid>.<test_set>.<test_case>.<measurement>
> > ...
> > > >     "test_case":[
> > > >         {
> > > >             "tguid":"ab2344cfa34546564s234acfddddaaa2342",
> > >
> > > I want the tguid to be human readable, and invariant over runs.
> > > In this case, I think the tguid for the single-measurement item
> > > should be "Dhrystone.Dhrystone"
> > ..
> > > >                     "tguid":"123a34546564s234acfddddaaa2342",
> > > >                     "name":"Create",
> > >
> > > tguid would be: "bonnie.Random_Create.Create"
> > >
> > > I think we can synthesize the tguid from the name fields.  If it's
> > unambiguous
> > > how to generate an invariant name for a test case/measure, then there's
> > > no need to have a separate field for the tguid.
> > >
> > > The idea with TGUIDs is that we can compare them between test systems.
> > > As an example, I'd like to be able to compare a 0-day Dhrystone.Dhrystone
> > value
> > > for the latest kernel with a Fuego Dhrystone.Dhrystone value for the LTSI
> > > kernel.
> >
> > OK - based on your comments, I think that I should remove all "tguid" fields
> > from the schema and
> > instead enforce (with the 'required' token) that all test
> > suites/sets/cases/measures have
> > a name. This way we can just infer the TGUIDs from the names.
> > What do you think?
> 
> Agreed.  I'm a bit worried about names not being globally invariant, based
> on supporting multiple "paths" to the testcase, depending on the structure
> of the tests and optional nesting of test_sets.
> e.g. Could LTP.syscall.kill10 end up being the same as LTP.RT-tests.functions.kill10?
> or could some super-test run LTP tests, resulting in a different string for the
> same test, like so: OpenTest.LTP.syscall.kill10?
> 
> I'd say that for now, just make 'name' required, and we'll see what issues crop up
> and we start using it.
> 
> >
> > Milo (if you read this): do you see any specific reason for using a hash in
> > Fuego as in Kernel CI?
> 
> What is the hash used for in KernelCI?  It almost looks like a per-run value, instead
> of a per-test value.
> 
> >
> > 2) timestamp format
> >
> > > >                 "start_time":{
> > > >                     "type":"string",
> > > >                     "description":"timestamp (YYYY-MM-ddThh:mm:SSZ) when the
> > > > test suite execution was started",
> > > >                     "format":"date-time"
> > >
> > > This is a change, but it appears to be iso 8601-compliant.
> > > I'd rather not be required to express the start time in UTC.  I think
> > > we should allow TimeZone offsets here (maybe you do).  I like to
> > > see times in a locally-relevant time format.  This is one of my chief
> > > gripes about Jenkins.
> >
> > > >         "start_time":"2017-06-26T09:03:04Z",
> > > Is this actually in UTC, rather than localtime?
> > > Where are you getting this from? date? python time.time?
> >
> > The "date-time" format is a built-in format in the JSON schema specification.
> > It is
> > explained at [1] and can be used both in UTC and in local time.
> >
> > UTC example from [1]: 1985-04-12T23:20:50.52Z
> >    This represents 20 minutes and 50.52 seconds after the 23rd hour of
> >    April 12th, 1985 in UTC.
> >
> > Local time example from [1]: 1996-12-19T16:39:57-08:00
> >    This represents 39 minutes and 57 seconds after the 16th hour of
> >    December 19th, 1996 with an offset of -08:00 from UTC (Pacific
> >    Standard Time).  Note that this is equivalent to 1996-12-20T00:39:57Z
> >    in UTC.
> >
> > In python you can get local time in two ways:
> >
> > With tzlocal (pip install tzlocal)
> >     import tzlocal, datetime
> >     d = datetime.datetime.now(tz=tzlocal.get_localzone())
> >     print d.isoformat('T')
> >         2017-06-30T09:13:27.507494+09:00
> >
> > With dateutil
> >     import dateutil.tz, datetime
> >     d = datetime.datetime.now(dateutil.tz.tzlocal())
> >     print d.isoformat('T')
> >         2017-06-30T09:13:27.507494+09:00
> >
> > [1] https://tools.ietf.org/html/rfc3339
> 
> The RFC3339 spec only allows 1 digit of fractional seconds,
> according to my reading of the spec.  I believe ISO8601 allows
> more fractional seconds digits, but I'd like to stick to a strict
> subset of ISO8601 for the reasons mentioned in RFC3339.  I dug
> into this today (in probably too much gory detail)
> and Python and C support for RFC3339 is kind of messed up.
> 
> See my notes on this at:
> http://bird.org/fuego/Issue_0047
> 
> I'm working on a patch that uses timestamps in the following format:
> $ date +F_%T%z
> 2017-06-29_17:44:25-0700
> 
> This is not strictly compliant with RFC3339, so might be different than what
> kernelCI expects, but I have reasons for my deviations.
> 
> >
> > 3) Others
> >
> > > >     "metadata":{
> > > >         "fuego_version":"0803ee7a2c5a59af159c01f561fc8d30d3d40e75",
> > > Can we do these with 'git describe' strings instead?
> > > ex:  "fuego_version":"v1.1-24-g5ad677b"
> >
> > Sure, JSON schema allows expressing the format with regular expressions.
> > Should I add one?
> 
> Just a raw string is fine.  I don't want to impose too much structure.
> I would just like to have it be more human readable than a raw hash.
> 
>  -- Tim

-------------- next part --------------
A non-text attachment was scrubbed...
Name: fuego-run-example-bonnie.json
Type: application/octet-stream
Size: 9383 bytes
Desc: not available
URL: <http://lists.linuxfoundation.org/pipermail/fuego/attachments/20170630/2c440e70/attachment-0003.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fuego-run-example-dhrystone.json
Type: application/octet-stream
Size: 2142 bytes
Desc: not available
URL: <http://lists.linuxfoundation.org/pipermail/fuego/attachments/20170630/2c440e70/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fuego-schema.json
Type: application/octet-stream
Size: 10837 bytes
Desc: not available
URL: <http://lists.linuxfoundation.org/pipermail/fuego/attachments/20170630/2c440e70/attachment-0005.obj>


More information about the Fuego mailing list