[Fuego] [PATCH 02/29] schema: create schema for defining pass criteria

Daniel Sangorrin daniel.sangorrin at toshiba.co.jp
Mon Jul 31 06:25:09 UTC 2017



> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> Sent: Tuesday, July 25, 2017 8:43 AM
> To: Daniel Sangorrin; fuego at lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 02/29] schema: create schema for defining pass criteria
> 
> See comment below.
> 
> > -----Original Message-----
> > From: fuego-bounces at lists.linuxfoundation.org [mailto:fuego-
> > bounces at lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> > Sent: Friday, July 21, 2017 1:02 AM
> > To: fuego at lists.linuxfoundation.org
> > Subject: [Fuego] [PATCH 02/29] schema: create schema for defining pass
> > criteria
> >
> > This schema is used for the creation of criteria.json files.
> > These files are used to specify the pass criteria of a test,
> > test_set, test_case or measure. Default criteria.json files
> > exist under the corresponding directories. Testers will be
> > able to override them with their own.
> > TODO: add option to ftc add-jobs/run for indicating the
> > location of the tester's criteria.json file.
> >
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin at toshiba.co.jp>
> > ---
> >  engine/scripts/parser/fuego-criteria-schema.json | 95
> > ++++++++++++++++++++++++
> >  1 file changed, 95 insertions(+)
> >  create mode 100644 engine/scripts/parser/fuego-criteria-schema.json
> >
> > diff --git a/engine/scripts/parser/fuego-criteria-schema.json
> > b/engine/scripts/parser/fuego-criteria-schema.json
> > new file mode 100644
> > index 0000000..eab4f42
> > --- /dev/null
> > +++ b/engine/scripts/parser/fuego-criteria-schema.json
> > @@ -0,0 +1,95 @@
> > +{
> > +    "$schema":"http://json-schema.org/schema#",
> > +
> > "id":"http://www.fuegotest.org/download/fuego_criteria_schema_v1.0.jso
> > n",
> > +    "title":"criteria",
> > +    "description":"Pass criteria for a test suite",
> > +    "definitions":{
> > +        "criterion":{
> > +            "title":"criterion ",
> > +            "description":"Criterion for deciding if a test (test_set, test_case or
> > measure) passes",
> > +            "type":"object",
> > +            "properties":{
> > +                "tguid":{
> > +                    "type":"string",
> > +                    "description":"unique identifier of a test (e.g.:
> > Sequential_Output.CPU)"
> > +                },
> > +                "min_pass":{
> > +                    "type":"number",
> > +                    "description":"Minimum number of tests that must pass"
> > +                },
> > +                "max_fail":{
> > +                    "type":"number",
> > +                    "description":"Maximum number of tests that can fail"
> > +                },
> > +                "pass_list":{
> > +                    "type":"array",
> > +                    "description":"Detailed list of tests that must pass",
> > +                    "items":{
> > +                        "type":"string"
> > +                    }
> > +                },
> > +                "failok_list":{
> > +                    "type":"array",
> > +                    "description":"Detailed list of tests that can fail",
> > +                    "items":{
> > +                        "type":"string"
> > +                    }
> > +                },
> > +                "reference":{
> > +                    "type":"object",
> > +                    "description":"Reference measure that is compared to a measure
> > to decide the status",
> > +                    "properties":{
> > +                        "threshold":{
> I'd rather this was named "value" or something like that.
> For benchmarks, this is often a threshold, but for the
> equality comparison operator, which you might use with
> functional test measurements, it's not really one.

OK.
 
> > +                            "type":[
> > +                                "string",
> > +                                "number",
> > +                                "integer"
> > +                            ],
> > +                            "description":"A threshold value"
> > +                        },
> > +                        "comparison":{
> > +                            "type":"string",
> > +                            "description":"Type of comparison",
> This looks OK. In the future it might be nice to allow regular expression matching also.

Good idea!

> 
> > +                            "enum":[
> > +                                "eq",
> > +                                "ne",
> > +                                "gt",
> > +                                "ge",
> > +                                "lt",
> > +                                "le"
> > +                            ]
> > +                        }
> > +                    },
> > +                    "required":[
> > +                        "threshold",
> > +                        "comparison"
> > +                    ]
> > +                }
> > +            },
> > +            "required":[
> > +                "tguid"
> > +            ]
> > +        }
> > +    },
> > +    "type":"object",
> > +    "properties":{
> > +        "schema_version":{
> > +            "type":"string",
> > +            "description":"The version number of this JSON schema",
> > +            "enum":[
> > +                "1.0"
> > +            ]
> > +        },
> > +        "criteria":{
> > +            "type":"array",
> > +            "description":"A list of criterion items",
> > +            "items":{
> > +                "$ref":"#/definitions/criterion"
> > +            }
> > +        }
> > +    },
> > +    "required":[
> > +        "schema_version",
> > +        "criteria"
> > +    ]
> > +}
> > --
> > 2.7.4
> 
>  -- Tim
> 





More information about the Fuego mailing list