[Fuego] status update
daniel.sangorrin at toshiba.co.jp
Tue Jun 27 06:04:17 UTC 2017
I've working on the parsing code. Here is a list of things that I managed to complete
and things that need some discussion:
- Output a single run.json file for each job's run/build that captures both the metadata and the test results.
- Merge run.json files for a test_suite into a single results.json file that flot can visualize.
+ I have added concurrency locks to protect results.json from concurrent writes.
- Add HTML output support to flot (similar, but not exactly the same yet, to the one in AGL JTA).
- Fixed the test Functional.tiff (AGL test) and confirmed that it works on docker and beaglebone black.
+ There are many more AGL tests to fix.
- Fixed several bug fixes that occurred when a test fails in an unexpected way.
- Output a jUnit XML file so that the plugin "Test Results Analyzer" can display.
+ This is working but it isn't as flexible as I'd like. The new flot's HTML output support that I added should deprecate it.
- Ability to download a PNG/SVG from the flot plugin directly.
- Add report functionality
+ I have removed the generation of plot.png files. In part because I want to do that directly from the
flot plugin, and in part because I think it is more useful if we integrate it in the future ftc report command,
- Add more information to the run.json files.
+ I am trying to produce a schema that is very close to the one used in Kernel.CI. Probably I can make it compatible.
+ There is some information that needs to be added to Fuego. Unfortunately, I will probably have to fix a lot of files:
1) Each test_case (remember test_suite > test_set > test_case) should be able to store a list of
measurements. Each measurement would consist of a name, value, units, duration and maybe more
such as error messages specific to the test_case or expected values (e.g. expected to fail, or expected to
be greater than 50).
2) Each test_case should have a status (PASS, FAIL..) that is not necessarily the same as the test_set/suite status.
+ Add vcs_commit information (git or tarball information)
- Handle failed runs better. Sometimes the test fails very early, before even building it.
- I am not sure what to do with the "reference.log" files
+ Currently they are used to store thresholds, but these are probably board dependent.
+ This is probably related to the discussion with Rafael about parameterized builds. We should
be able to define the threshold for a specific test_case's measure.
- Remove testplans?
+ I was thinking that we can substitute testplans by custom scripts that call ftc add-jobs
- Create a staging folder for tests that do not work or files that are not used.
+ Or maybe at least list them up on the wiki.
More information about the Fuego