[Fuego] json vs jdiff for custom pass-criteria

Daniel Sangorrin daniel.sangorrin at toshiba.co.jp
Wed Aug 2 04:36:24 UTC 2017



> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> Sent: Tuesday, August 01, 2017 8:51 AM
> To: Daniel Sangorrin; fuego at lists.linuxfoundation.org
> Subject: RE: json vs jdiff for custom pass-criteria
> 
> 
> 
> > -----Original Message-----
> > From: Daniel Sangorrin [mailto:daniel.sangorrin at toshiba.co.jp]
> > Sent: Sunday, July 30, 2017 11:05 PM
> > To: Bird, Timothy <Tim.Bird at sony.com>; fuego at lists.linuxfoundation.org
> > Subject: RE: json vs jdiff for custom pass-criteria
> >
> >
> >
> > > -----Original Message-----
> > > From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> > > Sent: Tuesday, July 25, 2017 7:57 AM
> > > To: Daniel Sangorrin; fuego at lists.linuxfoundation.org
> > > Subject: RE: json vs jdiff for custom pass-criteria
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Daniel Sangorrin [mailto:daniel.sangorrin at toshiba.co.jp]
> > > > Sent: Wednesday, July 19, 2017 1:33 AM
> > > > > -----Original Message-----
> > > > > From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> > > > > Sent: Wednesday, July 19, 2017 4:26 AM
> > > > > To: fuego at lists.linuxfoundation.org
> > > > > Cc: Daniel Sangorrin
> > > > > Subject: json vs jdiff for custom pass-criteria
> > > > >
> > > > > In our call, we discussed whether it would be better to use json format
> > or
> > > > some json diff
> > > > > format for the modifications to the reference file.  I'm leaning towards a
> > full
> > > > json file,
> > > > > as that's easier to pass to the existing framework.
> > > > >
> > > > > I know I wrote previously about readability of the files, and it won't be
> > > > obvious to the
> > > > > casual observer what's different between two reference files, but with
> > a
> > > > diff command
> > > > > it should be fairly easy to spot the differences between the two.  If we
> > use
> > > > a diff for the
> > > > > input format for the custom pass-criteria, then we'll have to write
> > another
> > > > processor
> > > > > and do this parsing on every test.
> > > >
> > > > OK, In that case I think we should have two separate files:
> > > > - test_definition.json: list of test_sets > test_cases > measurements
> > > > - criteria.json: including criteria for  test_sets, test_cases, and thresholds
> > for
> > > > measurements
> > > >
> > > > Note: we also need to deal with testers overwriting spec.json and
> > > > parameterized builds.
> > > >
> > > > > My own 'jdiff' command is not quite done yet.  There are some
> > > > complexities with
> > > > > it handling the current run.json format.  I tried it just now on the
> > run.json
> > > > files.
> > > > >
> > > > > In any event, I don't have a strong opinion either way.  As long as the
> > result
> > > > > is human editable and readable, so someone can make sense of what's
> > > > going on,
> > > > > it will be OK.  And we can probably convert between the two in a
> > backward
> > > > > compatible fashion, if we change our mind about a preferred format
> > later.
> > > >
> > > > test_definition.json should be invariant, so we can save it.
> > > > If the user provides a criteria.json for a test, then we use his/her.
> > Otherwise,
> > > > we use the
> > > > default criteria.json if there is one. Otherwise, we default to the generic
> > rule
> > > > that a test
> > > > passes if nothing explicitly FAILS.
> > >
> > > This all sounds good.  I especially like the heuristic that unmentioned
> > measurements
> > > should be evaluated in the obvious way.  (That is, if a measurement value is
> > PASS, and it's
> > > not mentioned in the pass-criteria, then it means the test passed.)
> > >
> > > With this generic rule, then small tests can be evaluated without an explicit
> > pass-criteria file.
> > > If all measurements pass, then the test passes.
> >
> > Just a small remark. Sometimes the status of a test_case or a measurement
> > becomes SKIP (not PASS, not FAIL)
> > either because the test_case wasn't skipped or because the measurement
> > was not within the error margin.
> > Currently, the default rule is that SKIPPED test_cases or measurements do
> > not cause the corresponding test_set
> > to fail, unless there is a specific criteria saying that such test_case must
> > actually PASS.
> 
> This sounds OK.  I was having a bit of trouble over the weekend with some
> code in common.py that set SKIP on some items, causing tests that actually
> failed to report SUCCESS.  However, I believe this is an result of my code
> being in a non-quiescent (ie hacked-up) state.  I'll first fix my other issues, then
> review that if it's still a problem.
> 
> (See a future message for my work-in-progress issues)
>  -- Tim

OK, sorry if it was my fault. I coded it quite quickly after your review, so that we could
continue the discussion and giving it shape together as soon as possible.

Thanks,
Daniel






More information about the Fuego mailing list