[Fuego] RFC - test/set/case documentation generation

Gustavo Sverzut Barbieri barbieri at profusion.mobi
Thu Feb 8 15:47:19 UTC 2018

On Thu, Feb 8, 2018 at 4:28 AM, <Tim.Bird at sony.com> wrote:
> Hello all,
> I'm writing to let you know about a project that I've started, having to do with
> documenting tests, testsets, and test cases, as well as feeding into capabilities
> for report generation.
> It's a bit hard to describe the overall scope of the project, but it's quite big.
> I want to document every testcase that Fuego performs.  This include a mixture
> of human-written documentation, and reports of results from tests performed
> on both local boards and eventually remote boards.
> My plan is to add a 'docs' directory under each test directory.  Inside that
> there will be template files, which describe tests, test sets, and individual test
> cases.  My plan is that the fuego system can combine test result data, with
> human-written text, in both HTML format (for use via the Jenkins intereface),
> and PDF format (for use in PDFs that are generated as test reports).
> The templates are written in reStructuredText, with macro directives
> to indicate what dynamic data to add to the templates.
> The template is processed by Fuego, and is output as an .rst file, which can
> be used as is as a straight text file, or further processed by sphynx to convert
> into either HTML (for access via the Jenkins interface) or latex (for publication
> in PDF test reports).
> It will likely take years to fill in all the testcases, but I expect that the list of
> documented testcases will likely always be sparse.  Only the most important
> testcases will be documented at first, and they can be filled in as we go along.
> Having some testcases documented will be more useful than having no
> testcases documented.
> The first instance of this is not quite ready for usage yet, but you can get an idea
> of what I have planned by looking at:
> /fuego-core/engine/tests/Functional.LTP/docs/Functional.LTP.ftmp
> ('ftmp' stands for 'Fuego template')
> This is processed by Fuego (by a program called gen-page.py) and
> the output is placed in:
> /fuego-rw/docs/Functional.LTP.rst
> The intent is to further process this into
> /fuego-rw/logs/Fucntional.LTP/Functional.LTP.html
> and make that accessible via the Jenkins interface.
> This is in my 'master' branch, at commit 0eba4e6
> Note that the information in Functional.LTP.ftmp is a little out-of-date, and
> the plan is to gather more information automatically (such as dependencies,
> spec list, etc.) over time.
> My idea is that using the same mechanisms, we will be able to generate reports
> from test data, that are a mixture of testcase results, summary data, detailed
> error reports, and human-written explanations of test operation and intent.
> One problem this is intended to solve has to do with report generation.  But
> another problem is that many of our tests are very opaque.  There's no
> human-readable documentation on the operation of the tests, what the results
> are supposed to mean, and what to do to resolve reported failures.  Creating a
> documentation system like this will allow such human-generated text to be
> presented alongside results, hopefully making results interpretation much easier.
> Work on this is progressing slowly, due to other demands on my time.
> But I wanted to present the idea to allow people to give feedback on it.
> Please let me know what you think.

Hi Tim, looks nice but my advice is to avoid pre-processing the rst
(your ftmp format) and instead use reStructuredText and add your own
directives, it's already meant for that.

I'd even go further and say that's better to setup an sphinx for the
project and create your own extension, then you can provide those as
needed AND scan for files in the directory structure. For example
Python's autodoc and the does something like that for python modules.

Initially you can avoid much of the burden by:

 - start sphinx project in the root directory
 - manually link the child .rst until you create your own autoindex
kind of function
 - in the child .rst you .. literalinclude:: results from some
directory until you create your own directives or roles

with that, no gen-pages is needed, just run the tests (to populate the
directories with results you're including as "literalinclude") and it
should produce the contents.

then you can create your sphinx extension (inside this repository,
just list it in the sphinx config) and start to provide the directives
and macros... they can do lots of stuff, like generate tables and
other sphinx formatting (ie: there are modules that does syntax
highlight using pygments). You can receive parameters in them, and you
can get the file/path of the file being processed, so you can discover
the test name and all.

Gustavo Sverzut Barbieri
http://profusion.mobi embedded systems
MSN, GTalk, FaceTime: barbieri at gmail.com
Skype: gsbarbieri
Mobile: +55 (16) 99354-9890

More information about the Fuego mailing list