[Fuego] Fuego criteria file documentation

Liu, Wenlong liuwl.fnst at cn.fujitsu.com
Mon Nov 20 06:54:03 UTC 2017


> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> Sent: Tuesday, November 14, 2017 9:53 AM
> To: Liu, Wenlong/刘 文龙 <liuwl.fnst at cn.fujitsu.com>;
> fuego at lists.linuxfoundation.org
> Subject: RE: Fuego criteria file documentation
> 
> 
> 
> > -----Original Message-----
> > From: fuego-bounces at lists.linuxfoundation.org [mailto:fuego-
> > bounces at lists.linuxfoundation.org] On Behalf Of Liu, Wenlong
> > Sent: Monday, November 13, 2017 2:38 AM
> > To: fuego at lists.linuxfoundation.org
> > Subject: Re: [Fuego] Fuego criteria file documentation
> >
> >
> > > -----Original Message-----
> > > From: fuego-bounces at lists.linuxfoundation.org
> > > [mailto:fuego-bounces at lists.linuxfoundation.org] On Behalf Of Bird,
> > > Timothy
> > > Sent: Thursday, November 02, 2017 8:53 AM
> > > To: fuego at lists.linuxfoundation.org
> > > Subject: [Fuego] Fuego criteria file documentation
> > >
> > > Hello all (but especially Daniel),
> > >
> > > I've been writing the documentation for new features in the Fuego
> > > 1.2
> > release,
> > > and I've put together a page about the new criteria file.
> > >
> > > The page is here:
> > > http://fuegotest.org/wiki/criteria.json
> > >
> > > Can you please take a look and see the information there is correct?
> > I have seen the current Fuego criteria documentation.
> > One question as below.
> >
> > As shown in the example of this documentation, {
> >     "schema_version":"1.0",
> >     "criteria": [
> >     {
> >         "tguid": "syscall",
> >         "min_pass": 1000,
> >         "max_fail": 5
> >     },
> >     {
> >         "tguid": "timers",
> >         "fail_ok_list": ["leapsec_timer"]
> >     },
> >     {
> >         "tguid": "pty",
> >         "must_pass_list": ["hangup01"]
> >     }
> >     ]
> > }
> > Yes, the example can works well.
> > However, you know, for LTP test, there are more than 70 sub-testsuites.
> >
> > For sub-testsuite named "net.nfs", the criteria json file cannot works
> > as I expected even though I added "net.nfs" as above.
> > When function "split_test_id(test_case_id)" was executed, the testset
> > name was changed to "default" and we cannot find the correct criteria
> json file.
> > The reason might be that "net.nfs" has a character ".".
> 
> That was indeed the cause.  The full test_case_id included multiple periods,
> which split_test_id did not handle correctly.
> 
> >
> > So, any advice for this situation or it should be fixed?
> I corrected this routine, with commit: f50cff2 in the master branch.
> 
> However, I didn't try to use a criteria file with with net.nfs.  Can you
> check out the latest code and try again?

I have tested this and find one problem.
It'll give the error log as below,

 ... ...
 Writing merged results to  /fuego-rw/logs/Functional.LTP/results.json
 Writing flat results to /fuego-rw/logs/Functional.LTP/flat_plot_data.txt
 Traceback (most recent call last):
   File "/fuego-core/engine/tests/Functional.LTP/parser.py", line 206, in <module>
     sys.exit(plib.process(test_results))
   File "/fuego-core/engine/scripts/parser/common.py", line 577, in process
     make_chart_data(test_logdir, TESTDIR, CHART_CONFIG_JSON, data_lines)
   File "/fuego-core/engine/scripts/parser/prepare_chart_data.py", line 703, in make_chart_data
     chart_list = make_testset_summary_table(test_name, chart_config, entries)
   File "/fuego-core/engine/scripts/parser/prepare_chart_data.py", line 622, in make_testset_summary_table
     ssb_list.sort(ssb_cmp)
   File "/fuego-core/engine/scripts/parser/prepare_chart_data.py", line 543, in ssb_cmp
     spec,ts,bn = ssb.split(".")
 ValueError: too many values to unpack
 ERROR: results did not satisfy the threshold
 ... ...

My test job name is ubuntu.stress.part3.Functional.LTP.
When Fuego try to make the summary table, function "ssb_cmp" cannot handle this situation.

In order to continue my test work, I temporarily used the solution in the attachment.
But I strongly doubt if it'll be a good solution.

> >
> > PS:
> > * Attached is my criteria json file for AGL, although it cannot works
> > as I expected(reference only).
> >   (LTP Version:20170516; Target: R-car M3)
> > * I didn't list all network related tests in this criteria json file
> > because of the issue above.
> 
> Thanks for the bug report.
>  -- Tim
> 
> 



-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-chart-handle-test_set-or-test_spec-names-with-embedd.patch
Type: application/octet-stream
Size: 2601 bytes
Desc: 0001-chart-handle-test_set-or-test_spec-names-with-embedd.patch
URL: <http://lists.linuxfoundation.org/pipermail/fuego/attachments/20171120/10196309/attachment.obj>


More information about the Fuego mailing list