[Fuego] some consulting about the usage of thresholds in Fuego

[Tim Bird]

On Tue, Aug 7, 2018 at 12:47 AM, zhongl.fnst at cn.fujitsu.com
<zhongl.fnst at cn.fujitsu.com> wrote:
> Hi, I have some consulting about the usage of thresholds in the current
> Fuego.
>
>
>
> For a Fuego test, different prioritized thresholds as below can be used:
>
> 1). FUEGO_CRITERIA_JSON_PATH'(user-specified); [High priority]
>
> 2). ro_board_crit_filename; [Mid priority]
>
> 3). rw_board_crit_filename; [Low priority]
>
> 4). default_criteria_filename; [Lower priority]
>
>
>
> In our testing, the default thresholds of Benchmark.iperf3 supposed to be
> used for a test board equipped with 1000MB ethernet card,
>
> so this test failed for our test board(no 1000MB ethernet card).
>
> Then, we added a customized criteria.josn in /fuego-rw/board/ for our test
> board.

That's correct.

The overall idea is that users can customize the thresholds and other pass
criteria at their own site, but putting items in /fuego-rw/board.

If they want to preserve them more permanently, they can put the files
in /fuego-ro/board.

As an aside, I started work on an ftc command that would update
pass-criteria files
based on existing test results, but that work is not finished.

You can see some of the ideas about this at:
http://fuegotest.org/wiki/Update_Criteria


>
> Maybe it’s better for testers to share thresholds of different cases and
> different boards.
Yes - that's very much the intent.

>
> Can testers share their thresholds and how?
The original idea to share them with others, was to create them in
/fuego-rw/board.  Then, when they were satisfied with the behavior,
 1) move the criteria files to /fuego-ro/board,
 2) make a patch for them, and
 3) send the patch upstream to be included in the fuego git
repository.

New users would get the updated criteria files by doing
git pull in the 'fuego' repository.

Currently, however, no one has actually shared their customized
pass criteria files yet.  And there are no examples of this yet in fuego-ro.
I have intended to do this for the beaglebone, but had not gotten
around to it yet (although I do have some customized criteria files
for that board.)

This system should work OK while there are a small number of labs
using Fuego, but it doesn't scale very well.  In the long run, I think it
would be good to make pass_criteria one of the objects that can be
put on the public fuego server (along with run-requests, ad-hoc tests,
and run-results).  There are a few issues with what meta-data to include
in the pass-criteria, so that users can browse them and decide which ones
make sense to download and use (and possibly customizations) for their
boards. That seems like the right way forward.

We won't know very well how to structure this, and what meta-data to
add to the pass criteria files, until we start sharing them with each other.

Please consider sending your criteria files in the manner described above,
and we'll proceed from there.  For my part, I'll try to prioritize my existing
action item to create more beaglebone criteria files, and put them into
the fuego git repository (as both examples, and for use by other developers
testing on the beaglebone).

Thanks for bringing this up.
 -- Tim