[Fuego] json vs jdiff for custom pass-criteria

Bird, Timothy Tim.Bird at sony.com
Wed Aug 2 05:09:41 UTC 2017

> From: Daniel Sangorrin [mailto:daniel.sangorrin at toshiba.co.jp]
> > From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> > This sounds OK.  I was having a bit of trouble over the weekend with some
> > code in common.py that set SKIP on some items, causing tests that actually
> > failed to report SUCCESS.  However, I believe this is an result of my code
> > being in a non-quiescent (ie hacked-up) state.  I'll first fix my other issues,
> then
> > review that if it's still a problem.
> OK, sorry if it was my fault. I coded it quite quickly after your review, so that
> we could
> continue the discussion and giving it shape together as soon as possible.

There may have been some rough edges, but I think more damage was caused
by me not understanding the code well enough to make the changes I attempted.
The good news is that it looks debuggable now.

I've got lots of things working, but lots of things still need to be fixed up.
Some of the issues are clearly not to do with the framework, per se, but
with my environment.
(Possibly I'm seeing some errors in docker tests because of some toolchain issues)

At this point I'd like to plow through all the non-working tests, and see if they
point out any flaws in the core (or are due to other issues).   I'd also like to get
some kind of proof of concept on report generation from the run.json data.
Now that we've got a unified output format, we should be able to leverage it
to do test analysis and reporting that we couldn't previously.

I don't want that to hold up the 1.2 release, though.  That is, I don't think we need
to fully flesh out the report generation support.  I just want to make sure that
our structured output format supports the reports we want to make.
We just need to be sure the parser API won't change again.  Then we
can start promoting more test generation and contributions, while we polish
up other features, and write more of our own tests.
 -- Tim

More information about the Fuego mailing list