[Fuego] RFC - test/set/case documentation generation

Tim.Bird at sony.com Tim.Bird at sony.com
Thu Feb 8 18:22:23 UTC 2018

Response at the bottom...

> -----Original Message-----
> From: Gustavo Sverzut Barbieri 
> On Thu, Feb 8, 2018 at 4:28 AM, <Tim.Bird at sony.com> wrote:
> >
> > Hello all,
> >
> > I'm writing to let you know about a project that I've started, having to do
> with
> > documenting tests, testsets, and test cases, as well as feeding into
> capabilities
> > for report generation.
> >
> > It's a bit hard to describe the overall scope of the project, but it's quite big.
> > I want to document every testcase that Fuego performs.  This include a
> mixture
> > of human-written documentation, and reports of results from tests
> performed
> > on both local boards and eventually remote boards.
> >
> > My plan is to add a 'docs' directory under each test directory.  Inside that
> > there will be template files, which describe tests, test sets, and individual
> test
> > cases.  My plan is that the fuego system can combine test result data, with
> > human-written text, in both HTML format (for use via the Jenkins
> intereface),
> > and PDF format (for use in PDFs that are generated as test reports).
> > The templates are written in reStructuredText, with macro directives
> > to indicate what dynamic data to add to the templates.
> >
> > The template is processed by Fuego, and is output as an .rst file, which can
> > be used as is as a straight text file, or further processed by sphynx to
> convert
> > into either HTML (for access via the Jenkins interface) or latex (for
> publication
> > in PDF test reports).
> >
> > It will likely take years to fill in all the testcases, but I expect that the list of
> > documented testcases will likely always be sparse.  Only the most
> important
> > testcases will be documented at first, and they can be filled in as we go
> along.
> > Having some testcases documented will be more useful than having no
> > testcases documented.
> >
> > The first instance of this is not quite ready for usage yet, but you can get an
> idea
> > of what I have planned by looking at:
> > /fuego-core/engine/tests/Functional.LTP/docs/Functional.LTP.ftmp
> > ('ftmp' stands for 'Fuego template')
> > This is processed by Fuego (by a program called gen-page.py) and
> > the output is placed in:
> > /fuego-rw/docs/Functional.LTP.rst
> > The intent is to further process this into
> > /fuego-rw/logs/Fucntional.LTP/Functional.LTP.html
> > and make that accessible via the Jenkins interface.
> >
> > This is in my 'master' branch, at commit 0eba4e6
> >
> > Note that the information in Functional.LTP.ftmp is a little out-of-date, and
> > the plan is to gather more information automatically (such as
> dependencies,
> > spec list, etc.) over time.
> >
> > My idea is that using the same mechanisms, we will be able to generate
> reports
> > from test data, that are a mixture of testcase results, summary data,
> detailed
> > error reports, and human-written explanations of test operation and
> intent.
> >
> > One problem this is intended to solve has to do with report generation.
> But
> > another problem is that many of our tests are very opaque.  There's no
> > human-readable documentation on the operation of the tests, what the
> results
> > are supposed to mean, and what to do to resolve reported failures.
> Creating a
> > documentation system like this will allow such human-generated text to be
> > presented alongside results, hopefully making results interpretation much
> easier.
> >
> > Work on this is progressing slowly, due to other demands on my time.
> > But I wanted to present the idea to allow people to give feedback on it.
> >
> > Please let me know what you think.
> Hi Tim, looks nice but my advice is to avoid pre-processing the rst
> (your ftmp format) and instead use reStructuredText and add your own
> directives, it's already meant for that.

You may notice that my "macro" actually has the same syntax as
an reStructuredText directive.  This was done specifically to allow
the possibility of converting over to the flow of processing that
you mention below.

As an aside, what does Sphynx do with directives it doesn't recognize?
(Sorry - I'm too lazy to go test is right now, but if someone knows
off the top of their head please let me know.)

> I'd even go further and say that's better to setup an sphinx for the
> project and create your own extension, then you can provide those as
> needed AND scan for files in the directory structure. For example
> Python's autodoc and the does something like that for python modules.
> Initially you can avoid much of the burden by:
>  - start sphinx project in the root directory
>  - manually link the child .rst until you create your own autoindex
> kind of function
>  - in the child .rst you .. literalinclude:: results from some
> directory until you create your own directives or roles
> with that, no gen-pages is needed, just run the tests (to populate the
> directories with results you're including as "literalinclude") and it
> should produce the contents.
> then you can create your sphinx extension (inside this repository,
> just list it in the sphinx config) and start to provide the directives
> and macros... they can do lots of stuff, like generate tables and
> other sphinx formatting (ie: there are modules that does syntax
> highlight using pygments). You can receive parameters in them, and you
> can get the file/path of the file being processed, so you can discover
> the test name and all.

Thanks very much for the feedback.  I am by no means sure that the
approach I have taken is optimal, so I really appreciate other ideas
for implementing this.

With regard to sphynx, I definitely want to make it a first-class tool
in our container, and possibly add fuego-specific extensions (directives) to it.
I took a look yesterday at how to write extensions for it, and they
look powerful, and straightforward to write.  I've already started moving
the regular Fuego documentation over to reStructuredText, but I don't
recall if I've made the changes to include sphynx in the Dockerfile yet.

I'll think more about your suggestion as I keep working on this.
 -- Tim

More information about the Fuego mailing list