[Fuego] [PATCH 02/29] schema: create schema for defining pass criteria
Daniel Sangorrin
daniel.sangorrin at toshiba.co.jp
Mon Jul 31 06:25:09 UTC 2017
> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> Sent: Tuesday, July 25, 2017 8:43 AM
> To: Daniel Sangorrin; fuego at lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 02/29] schema: create schema for defining pass criteria
>
> See comment below.
>
> > -----Original Message-----
> > From: fuego-bounces at lists.linuxfoundation.org [mailto:fuego-
> > bounces at lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> > Sent: Friday, July 21, 2017 1:02 AM
> > To: fuego at lists.linuxfoundation.org
> > Subject: [Fuego] [PATCH 02/29] schema: create schema for defining pass
> > criteria
> >
> > This schema is used for the creation of criteria.json files.
> > These files are used to specify the pass criteria of a test,
> > test_set, test_case or measure. Default criteria.json files
> > exist under the corresponding directories. Testers will be
> > able to override them with their own.
> > TODO: add option to ftc add-jobs/run for indicating the
> > location of the tester's criteria.json file.
> >
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin at toshiba.co.jp>
> > ---
> > engine/scripts/parser/fuego-criteria-schema.json | 95
> > ++++++++++++++++++++++++
> > 1 file changed, 95 insertions(+)
> > create mode 100644 engine/scripts/parser/fuego-criteria-schema.json
> >
> > diff --git a/engine/scripts/parser/fuego-criteria-schema.json
> > b/engine/scripts/parser/fuego-criteria-schema.json
> > new file mode 100644
> > index 0000000..eab4f42
> > --- /dev/null
> > +++ b/engine/scripts/parser/fuego-criteria-schema.json
> > @@ -0,0 +1,95 @@
> > +{
> > + "$schema":"http://json-schema.org/schema#",
> > +
> > "id":"http://www.fuegotest.org/download/fuego_criteria_schema_v1.0.jso
> > n",
> > + "title":"criteria",
> > + "description":"Pass criteria for a test suite",
> > + "definitions":{
> > + "criterion":{
> > + "title":"criterion ",
> > + "description":"Criterion for deciding if a test (test_set, test_case or
> > measure) passes",
> > + "type":"object",
> > + "properties":{
> > + "tguid":{
> > + "type":"string",
> > + "description":"unique identifier of a test (e.g.:
> > Sequential_Output.CPU)"
> > + },
> > + "min_pass":{
> > + "type":"number",
> > + "description":"Minimum number of tests that must pass"
> > + },
> > + "max_fail":{
> > + "type":"number",
> > + "description":"Maximum number of tests that can fail"
> > + },
> > + "pass_list":{
> > + "type":"array",
> > + "description":"Detailed list of tests that must pass",
> > + "items":{
> > + "type":"string"
> > + }
> > + },
> > + "failok_list":{
> > + "type":"array",
> > + "description":"Detailed list of tests that can fail",
> > + "items":{
> > + "type":"string"
> > + }
> > + },
> > + "reference":{
> > + "type":"object",
> > + "description":"Reference measure that is compared to a measure
> > to decide the status",
> > + "properties":{
> > + "threshold":{
> I'd rather this was named "value" or something like that.
> For benchmarks, this is often a threshold, but for the
> equality comparison operator, which you might use with
> functional test measurements, it's not really one.
OK.
> > + "type":[
> > + "string",
> > + "number",
> > + "integer"
> > + ],
> > + "description":"A threshold value"
> > + },
> > + "comparison":{
> > + "type":"string",
> > + "description":"Type of comparison",
> This looks OK. In the future it might be nice to allow regular expression matching also.
Good idea!
>
> > + "enum":[
> > + "eq",
> > + "ne",
> > + "gt",
> > + "ge",
> > + "lt",
> > + "le"
> > + ]
> > + }
> > + },
> > + "required":[
> > + "threshold",
> > + "comparison"
> > + ]
> > + }
> > + },
> > + "required":[
> > + "tguid"
> > + ]
> > + }
> > + },
> > + "type":"object",
> > + "properties":{
> > + "schema_version":{
> > + "type":"string",
> > + "description":"The version number of this JSON schema",
> > + "enum":[
> > + "1.0"
> > + ]
> > + },
> > + "criteria":{
> > + "type":"array",
> > + "description":"A list of criterion items",
> > + "items":{
> > + "$ref":"#/definitions/criterion"
> > + }
> > + }
> > + },
> > + "required":[
> > + "schema_version",
> > + "criteria"
> > + ]
> > +}
> > --
> > 2.7.4
>
> -- Tim
>
More information about the Fuego
mailing list