[Fuego] json vs jdiff for custom pass-criteria
daniel.sangorrin at toshiba.co.jp
Wed Aug 2 05:41:42 UTC 2017
> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> Sent: Wednesday, August 02, 2017 2:10 PM
> To: Daniel Sangorrin; fuego at lists.linuxfoundation.org
> Subject: RE: json vs jdiff for custom pass-criteria
> > From: Daniel Sangorrin [mailto:daniel.sangorrin at toshiba.co.jp]
> > > From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> > > This sounds OK. I was having a bit of trouble over the weekend with some
> > > code in common.py that set SKIP on some items, causing tests that actually
> > > failed to report SUCCESS. However, I believe this is an result of my code
> > > being in a non-quiescent (ie hacked-up) state. I'll first fix my other issues,
> > then
> > > review that if it's still a problem.
> > OK, sorry if it was my fault. I coded it quite quickly after your review, so that
> > we could
> > continue the discussion and giving it shape together as soon as possible.
> There may have been some rough edges, but I think more damage was caused
> by me not understanding the code well enough to make the changes I attempted.
> The good news is that it looks debuggable now.
> I've got lots of things working, but lots of things still need to be fixed up.
> Some of the issues are clearly not to do with the framework, per se, but
> with my environment.
> (Possibly I'm seeing some errors in docker tests because of some toolchain issues)
Each time I changed something I did "ftc rm-jobs" to remove all jobs including
their logs because the old files were interfering.
> At this point I'd like to plow through all the non-working tests, and see if they
> point out any flaws in the core (or are due to other issues). I'd also like to get
> some kind of proof of concept on report generation from the run.json data.
> Now that we've got a unified output format, we should be able to leverage it
> to do test analysis and reporting that we couldn't previously.
> I don't want that to hold up the 1.2 release, though. That is, I don't think we need
> to fully flesh out the report generation support. I just want to make sure that
> our structured output format supports the reports we want to make.
> We just need to be sure the parser API won't change again. Then we
> can start promoting more test generation and contributions, while we polish
> up other features, and write more of our own tests.
OK, sounds like a good idea.
More information about the Fuego