[Fuego] json vs jdiff for custom pass-criteria

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin [mailto:daniel.sangorrin at toshiba.co.jp]
> Sent: Sunday, July 30, 2017 11:05 PM
> To: Bird, Timothy <Tim.Bird at sony.com>; fuego at lists.linuxfoundation.org
> Subject: RE: json vs jdiff for custom pass-criteria
> 
> 
> 
> > -----Original Message-----
> > From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> > Sent: Tuesday, July 25, 2017 7:57 AM
> > To: Daniel Sangorrin; fuego at lists.linuxfoundation.org
> > Subject: RE: json vs jdiff for custom pass-criteria
> >
> >
> >
> > > -----Original Message-----
> > > From: Daniel Sangorrin [mailto:daniel.sangorrin at toshiba.co.jp]
> > > Sent: Wednesday, July 19, 2017 1:33 AM
> > > > -----Original Message-----
> > > > From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> > > > Sent: Wednesday, July 19, 2017 4:26 AM
> > > > To: fuego at lists.linuxfoundation.org
> > > > Cc: Daniel Sangorrin
> > > > Subject: json vs jdiff for custom pass-criteria
> > > >
> > > > In our call, we discussed whether it would be better to use json format
> or
> > > some json diff
> > > > format for the modifications to the reference file.  I'm leaning towards a
> full
> > > json file,
> > > > as that's easier to pass to the existing framework.
> > > >
> > > > I know I wrote previously about readability of the files, and it won't be
> > > obvious to the
> > > > casual observer what's different between two reference files, but with
> a
> > > diff command
> > > > it should be fairly easy to spot the differences between the two.  If we
> use
> > > a diff for the
> > > > input format for the custom pass-criteria, then we'll have to write
> another
> > > processor
> > > > and do this parsing on every test.
> > >
> > > OK, In that case I think we should have two separate files:
> > > - test_definition.json: list of test_sets > test_cases > measurements
> > > - criteria.json: including criteria for  test_sets, test_cases, and thresholds
> for
> > > measurements
> > >
> > > Note: we also need to deal with testers overwriting spec.json and
> > > parameterized builds.
> > >
> > > > My own 'jdiff' command is not quite done yet.  There are some
> > > complexities with
> > > > it handling the current run.json format.  I tried it just now on the
> run.json
> > > files.
> > > >
> > > > In any event, I don't have a strong opinion either way.  As long as the
> result
> > > > is human editable and readable, so someone can make sense of what's
> > > going on,
> > > > it will be OK.  And we can probably convert between the two in a
> backward
> > > > compatible fashion, if we change our mind about a preferred format
> later.
> > >
> > > test_definition.json should be invariant, so we can save it.
> > > If the user provides a criteria.json for a test, then we use his/her.
> Otherwise,
> > > we use the
> > > default criteria.json if there is one. Otherwise, we default to the generic
> rule
> > > that a test
> > > passes if nothing explicitly FAILS.
> >
> > This all sounds good.  I especially like the heuristic that unmentioned
> measurements
> > should be evaluated in the obvious way.  (That is, if a measurement value is
> PASS, and it's
> > not mentioned in the pass-criteria, then it means the test passed.)
> >
> > With this generic rule, then small tests can be evaluated without an explicit
> pass-criteria file.
> > If all measurements pass, then the test passes.
> 
> Just a small remark. Sometimes the status of a test_case or a measurement
> becomes SKIP (not PASS, not FAIL)
> either because the test_case wasn't skipped or because the measurement
> was not within the error margin.
> Currently, the default rule is that SKIPPED test_cases or measurements do
> not cause the corresponding test_set
> to fail, unless there is a specific criteria saying that such test_case must
> actually PASS.

This sounds OK.  I was having a bit of trouble over the weekend with some
code in common.py that set SKIP on some items, causing tests that actually
failed to report SUCCESS.  However, I believe this is an result of my code
being in a non-quiescent (ie hacked-up) state.  I'll first fix my other issues, then
review that if it's still a problem.

(See a future message for my work-in-progress issues)
 -- Tim