[Fuego] ANNOUNCING Fuego version 1.2.1

Thu Nov 16 01:01:11 UTC 2017

Hello Fuego community,

I am pleased to announce that I have tagged the Fuego 1.2.1 release.  This is a follow-on
to the 1.2.0 release (codenamed "Combustion", in keeping with our fire theme).
However, I never formally announced the 1.2.0 release, and tagged it before I
finished some of the documentation on the wiki.  I got too busy with ELC Europe,
and didn't get around to the formal announcement until now.

This release, while it still has some known issues, should be considered the next
official major release of Fuego.

I have written up more material on the wiki than I can link to here, but a few
pages are of interest:
http://fuegotest.org/wiki/Release_1.2_(Combustion)_Notes

http://fuegotest.org/wiki/Release_1.2.1_Notes

I'm actually going to include the entirety of the "Combustion" release notes
in this message (below), for easier reading and discussion, if desired. 
I'm happy to take feedback on the new features, and their intended use.

You can find the code, as usual, in my master branches on bitbucket:
https://bitbucket.org/tbird20d/fuego
https://bitbucket.org/tbird20d/fuego-core.git

Here are the release notes:
(Note that items in brackets are page names on the fuego wiki at http://fuegotest.org/wiki )
------------
These are the Release Notes for the Fuego 1.2 release, which we are calling "Combustion".

Although this release is numbered as a "minor" release, it has some
major changes in functionality. Here are some things to check out in
this release:

= Major changes =
The major changes to Fuego in this release are as follows:
 * Unified output format
 * Test dependency system
 * Complex pass criteria handling
 * Dynamic board variables
 * Charting core improvements
 * Support for test program source from git repositories
 * Some transport modifications
 * Test improvements

These features are described in the sections that follow.

== Unified output format ==
Fuego now stores the results of all tests in a unified format.

Every test in the system now creates a run.json file, which includes
that test's results, organized into testsets, testcases and measurements.

This format is intentionally modeled after kernelCI APIs, for easier
integration with that test system's front-end.

See [[Unified Output Format]] for more information about this feature.

== Test dependency system ==
Fuego 1.2 introduces a test dependency system, that allows
Fuego to determine whether a test can be run on a board or not,
based on a set of variables declared in the tests base script (fuego_test.sh).

The dependency system allows for short-circuiting of test execution, when
required features are not present.  That is, these dependencies are checked
during a pre_test phase, and if the dependencies are not met, Fuego aborts
the test, before the test is built, deployed and executed on the target.

The dependencies are expressed as "NEED_" variables in fuego_test.sh

In the future, this features will allow for automatically detecting what
tests are applicable to particular boards.

For more information, see [[Dependencies]]

== Complex pass criteria handling ==
Fuego allows for expressing complex pass criteria for a test.
By default, most tests require that all individual testcases
in the test succeed, in order for the test to pass.  However, large
complex tests often have individual testcases that fail.
Fuego allows specifying complex pass criteria, including count-based
pass criteria (number of pass vs. fail), as well as detailed
list-based criteria (testcases whose failure is ignored, or
which are required to pass).  Additionally, the threshold
values used to determine whether a benchmark test passes is
configurable using the same criteria mechanism.

Fuego allows the default pass criteria to be overridden via
per-board criteria definitions, or by a user-supplied criteria file.

This flexibility allows Fuego users to customize results determination
to their boards and unique testing situations, and to share
results analysis in a formalized way between users and sites.

By way of example, the Fuego criteria system allows
a user to specify what the expected filesystem performance, 
or real-time latency should be on their board, in order to meet
their product shipment requirements.  It also allows them to
indicate that certain tests, although they fail, can be safely
ignored for the purposes of overall evaluation of the system.

See [[criteria.json]] for more information.

== Dynamic board variables ==
Fuego allows a user or a test to add additional variables to
a board definition.

This allows persistent information to be saved about a board,
which can be used to share information between tests, or
between invocations of a test.

See the section "Dynamic variables" on the [[Test variables]] page
for the rationale for this feature and examples of potential uses for it.

== Charting core improvements ==
Charting in Fuego was changed in the 1.2 release, to provide
a foundation for future releases.  A new chart configuration
file was introduced, which controls what type of charting to
show by default in the Jenkins interface for a test.
Additionally, Fuego now supports 3 chart types:
 * HTML table of testcases
 * HTML table of test set aggregate results
 * Javascript plots of measurement data, for Benchmark tests

Internally, the charting engine was modified to make it
it more accessible to developers, and to allow (in the future)
the same charting engine to be used for both realtime visualization
and static report generation.

See [[Fuego charting]]

== Test program source from git repositories ==
Fuego has added, in this release, the ability to use
git repositories for test program source.  This is in
addition to the default of providing a  tarball of source
for test programs.

A user can specify a git repository, and a git reference
(commit id, tag, or version), and Fuego will use that source
to build the test program. This provides additional
flexibility in dealing with upstream source for test programs.

See [[function_unpack]] and [[Adding a test]].

== Transport modifications ==
In the 1.2 release, some slight modifications were made to
support different transport mechanisms (particularly to
support boards in LAVA-based board farms).

There are new functions: [[function_ov_transport_connect|ov_transport_connect]]
and [[function_ov_transport_disconnect|ov_transport_disconnect]] for
handling session setup and teardown (or board reservation and release).

Please see the documentation for these functions, for details.

== Test improvements ==
There were numerous adjustments to tests to deal with new
features and fix errant behavior.  Some of the major improvements
to multiple tests were:
 * Added support for aarch64 toolchains
 * Added dependency information
 * Some older tests were fixed to address build issues with newer toolchains
 * The source for some test programs was updated to newer versions
 * Parsers were added for some Functional tests, to give results for individual testcases
 * Parser improvements in general
 * Lots of other bugfixes were made

In addition, Functional.LTP was modified to support LTP
binaries already installed on a target board.  This work
will likely serve as a template for handling additional tests
with binaries already on the board, in future Fuego versions.
See [[Functional.LTP]] for some information about board
variables and specs that can be used in this situation.

== Documentation updates ==
Finally, a lot of these new features required extensive
documentation updates, which have (hopefully) mostly been
completed.

Thanks for your patience, and for contributions from numerous authors.
Special thanks goes to Daniel Sangorrin who authored several of these
major features.

Please test it out (ha ha, pun intended), and let me know how it goes.
 -- Tim