[Fuego] ANNOUNCE: Fuego Release v1.3.0

[Tim.Bird at sony.com]

I am pleased to announce the latest release of Fuego -- version 1.3.0.
The code name for this release is "Draft" - as in a fire "draft" (not a
document draft).

I would like to thank contributors to this release, which has been a long time coming.
I hope that the changes will be useful for your testing needs.

The release notes for this release are at: 
http://fuegotest.org/wiki/Release_1.3_Notes
They are long, but I'm including them below for your reading pleasure. :-)

There are a few known bugs that are documented on the release testing
pages for this release, which are linked from http://fuegotest.org/wiki/Releases
Some of these are old bugs that we haven't gotten around to fixing.
But the tests run OK on some boards.  I wanted to get started on the 1.4 release
before my trip to Japan in June, so I'm releasing 1.3 now.

The release has been tagged and is available in the main Fuego repositories:
     https://bitbucket.org/tbird20d/fuego.git
     https://bitbucket.org/tbird20d/fuego-core.git
(in the "master" branch of each repository)

Thanks again.  And let me know if you see any bugs.  
 -- Tim

------
These are the Release Notes for the Fuego 1.3 release, which we are calling "Draft".

Although this release is numbered as a "minor" release, it has
some major changes in functionality. Here are some things to
check out in this release:

= Major changes =
The major changes to Fuego in this release are as follows:
 * Use of fuego.conf
 * Ability to run ftc outside the docker container
 * New report generation features
   * more output formats
   * customizable field lists
   * more options for --where clauses (tguid, tguid:result)
 * Support for board hardware reboot (ov_board_control_reboot)
 * Log splitting by testcase
 * Can invoke individual test phases with ftc
 * New tests:
   * vuls, LTP_one_test, many RT tests, year2038
   * release test (self test)
 * Addition of web and image comparison features

These features are described in the sections that follow.

== Fuego configuration file ==
The 1.3 release of Fuego introduces a global configuration file for
Fuego.  This file is used to hold global information about the Fuego
installation, and is normally located at fuego-ro/conf/fuego.conf.

The main purpose of this file in this release is to allow for 
ftc to be run outside the docker container.  However, it will be used
to hold additional global configuration variables in the future.

See [[Fuego configuration file]] for more details.

== Running ftc outside docker container ==
Up until this release, all ftc operations were performed inside the docker
container.  There was a convenience feature of ftc that allowed the tool
to be executed outside the docker container, but in this case it would
just re-invoke an instance of 'ftc' inside the docker container to perform
the actual work.

However, some ftc commands only need to access fuego directories, and don't
need to actually execute inside the docker container.  And most commands
operate faster when they are performed directly on the host, rather than
inside the container (for example, 'ftc gen-report').

This release adds the capability to control whether ftc runs outside the
docker container or not.  Two new command line options to ftc support
this functionality.  The '-x' option tells ftc to NOT re-invoke the operation
inside the docker container.  And the '-c' option allows the user to tell
ftc where the fuego configuration file is.  With these two options,
a user can tell ftc to execute the indicated command outside the container.

For more details on this, please see the following page:
[[Running ftc outside the container]]

== New report generation features ==
'ftc gen-report' supports several new features, which make the command more
flexible and useful.
The additions can be divided into three main groups:
 * new filtering (where clause) options
 * new output formats
 * ability to customize the fields reported

The --where clause is used to filter the data that appears in the report.
The --where clause can now filter by test data by the tguid, tguid:result and duration_ms fields.

In addition to text output, you can now specify output in the following
formats:
 * html
 * pdf
 * excel
 * csv
 * rst

Some of these formats are output to a file in the report directory,
which by default is /fuego-rw/reports.  The location of the report
directory can be overridden using the -o option.

Using the options --header_fields and --fields, you can now control
what fields from each run will be put in the report header and in
the body (table) of the report.

See 'ftc gen-report help' to get more information about the report
generation options.

== Hardware board control ==
Previous to 1.3, Fuego only did software rebooting of the board under test.
The 1.3 release introduces a concept called "board control", which allows
Fuego to perform hardware operations on the board.  In the 1.3
release, the board may specify a BOARD_CONTROL variable in the board file,
which indicates what board control system is being used.

In the 1.3 release, only the 'ttc' board control system is supported, and
the only hardware board control operation supported is '''reboot'''.

See [[function_ov_board_control_reboot|ov_board_control_reboot]] for
the function called by Fuego to do a hardware reboot.

Under some circumstances, Fuego will detect a hung board, and try to
perform a hardware reboot of it during the post-test phase of test processing.


== Log splitting by testcase ==
In the 1.3 release, there is new functionality for splitting the testlog
of a run into individual parts, and browsing them from the Jenkins user
interface.

The new functionality is in the parser code, and to use it the test
must have a custom parser (its own parser.py).  That parser must call
the function [[function_split_output_per_testcase|split_output_per_testcase]]

See that function call for more information about adding this
feature to a particular test.

== Support for test phase execution ==
ftc run-test now supports the '-p' option to specify that only a sub-set
of test phases should be executed.  This is useful for quick debugging
of tests, or for special operations like pre-building all tests in a
test plan.

Use 'ftc run-test help' for details.

== New tests in the 1.3 release ==
Some new tests were added to the Fuego 1.3 release.

There are 3 mains groups of tests:
 * New real-time tests,
 * Fuego self-tests
 * Other new tests (including updated versions of tests)

=== New real-time tests ===
This set of tests adds additional Fuego tests for existing programs
in the rt-tests suite of tests.
 * Benchmark.backfire
 * Benchmark.deadlinetest
 * Benchmark.migratetest
 * Benchmark.pmqtest
 * Benchmark.ptsematest
 * Benchmark.sigwaittest
 * Benchmark.svsematest

=== New Fuego self-tests ===
 * Functional.fuego_lint
 * Functional.fuego_releast_test
 * Functional.fuego_tguid_check

=== Other new tests ===
 * Benchmark.dbench4
 * Benchmark.dd
 * Benchmark.iperf3
 * Benchmark.vuls
 * Functional.autopkgtest
 * Functional.fuego_board_status
 * Functional.LTP_one_test
 * Functional.year2038

=== Test Descriptions ===
; Benchmark.backfire: Uses a kernel module (called backfire) to measure signal performance in the kernel.
; Benchmark.deadlinetest: Test the deadline scheduler
; Benchmark.migratetest: Test real-time multiprocessor scheduling of tasks to ensure the highest priority tasks are running on all available CPUs.
; Benchmark.pmqtest: Starts pairs of threads and measures the latency of interprocess communication with POSIX message queues.
; Benchmark.ptsematest: Starts two threads and measures the latency of inteprocess communication with POSIX mutex.
; Benchmark.sigwaittest: Starts two threads or forks two processes, and measures the latency between sending and receiving a signal from one to the other.
; Benchmark.svsematest: Starts two threads or forks two processes, and measures the latency of using SYSV semaphores between the two.

----

; Functional.fuego_lint: Tests that Fuego internal programs follow preferred coding style.
; Functional.fuego_release_test: This test builds the Fuego docker image, and tests basic core Fuego functionality.  This is described more below.
; Functional.fuego_tguid_check: This test verifies that Fuego core routines correctly parse and handle tguids with complicated formats.

----

; Benchmark.dbench4: This is an updated version of the dbench file system performance test.  Dbench was written by the authors of SAMBA, and it generates I/O workloads for filesystem testing.  It uses "loadfiles" to simulate the I/O patterns, including timing, for real applications.
; Benchmark.dd: This test measures the time for simple dd operation on the filesystem.  It tests basic I/O performance.
; Benchmark.iperf3: This is an updated version of the iperf network performance test.
; Benchmark.vuls: Vuls is an agentless vulnerability scanner for Linux and FreeBSD. It uses multiple vulnerability databases. See https://vuls.io/
; Functional.autopkgtest: Runs Debian package tests on the target.  Note that this installs and leaves the package dependencies for the tests on the target board. This means that you should use a board control scheme which re-provisions the board as part of testing (e.g. LAVA).
; Functional.fuego_board_status: Gathers a bunch of information about a board.  This is used to establish a baseline of information about a board, which can be compared with a subsequent run later, to see if something has changed.  Some of the items collected are: cpu information, memory information, loaded modules, usb devices, block devices, mount points, file system disk usage, network devices, running processes, kernel boot messages, and installed packages.
; Functional.LTP_one_test: Allows running a single test program from the LTP test suite.  This is used for isolating individual test cases of interest, with reduced overhead to see results for a single test.
; Functional.year2038: A very simple test of correctly handling dates after the year 2038.

== Release test (self test) ==
Fuego now includes a selftest for itself, called Functional.fuego_release_test.

Hey! We have a test system.  We should be able to use it to test itself.

This test is fairly complicated, but it adds some interesting features to
Fuego.  As an overview, it clones the fuego repositories (fuego and fuego-core),
builds the docker container, starts the container, and checks that various
features are supported (both on the command line and in the Jenkins interface).

The reason this test is complicated is that you end up with two complete
versions of Fuego running at the same time on your system.  The Fuego
driving the test, and the Fuego "under test".  So, it takes a lot of
memory and disk on your host system to be able to run this test.

Probably, this test is most useful to Fuego maintainers, to test releases
of Fuego.  However, the test also utilizes some new functionality that is
now incorporated into the Fuego distribution in the docker container.

For more information about this test, and how to run it,
see the file: fuego-core/engine/tests/Functional.fuego_release_test/README.md


== Web and image comparison features ==
[[https://www.seleniumhq.org/|Selenium]] and Chromium have been added
to the base Fuego distribution, to support testing of results returned
from a web-based interface.

Using techniques demonstrated in Functional.fuego_release_test, you can use
these host-side tools to test web functionality provided by your target board.
This includes the capability for a test to scan a web page for specific
text objects, as well as to check that client-side rendering of a web page
(or part of the page) matches an expected image.

A test developer can take snapshots of a web page, or a region of the page,
and save them for use with a test.  Also, the developer can create image masks.
These image masks can be used to omit sections of the page from comparison.

In this release, these capabilities are contained in a single test (the
fuego_release_test).  However, the intent is to migrate these capabilities
into a generalized library that can be used by other Fuego tests, so that
web page output can be tested easily with Fuego.

= Experimental or unfinished features in this release =
== Sphynx documentation ==
In the 1.3 release, a Sphynx documentation directory and Makefile
were created.  Sphynx is now required on the host machine for building
the documentation.

See fuego/docs/Makefile and the fuego/docs/*.rst files for the initial
(currently stubbed) files for the conversion of the documentation to
Sphynx (reStructuredText) format.

Although the fuego repository was configured for supporting Sphynx
documentation, the pages from the fuegotest wiki that comprise the
current documentation for Fuego have not been converted to reStructuredText
yet.  There is a page with information about markup conversions at
[[Markup Mapping]]

== testcase documentation ==
We have added a convention for storing test, testset and testcase
documentation in reStructuredText.

Tests can now have a 'docs' directory where this documentation
is stored.  The purpose of this documentation is to allow
test users to share information about the test and testcases
among themselves, in a way that will (eventually) be accessible
via the Jenkins interface, and able to be placed in generated
reports.

''Note, however, that this system is still under development.''

See [[Test documentation]] for more information about this system.