[Fuego] Fuego criteria file documentation
Liu, Wenlong
liuwl.fnst at cn.fujitsu.com
Mon Nov 20 06:54:03 UTC 2017
> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird at sony.com]
> Sent: Tuesday, November 14, 2017 9:53 AM
> To: Liu, Wenlong/刘 文龙 <liuwl.fnst at cn.fujitsu.com>;
> fuego at lists.linuxfoundation.org
> Subject: RE: Fuego criteria file documentation
>
>
>
> > -----Original Message-----
> > From: fuego-bounces at lists.linuxfoundation.org [mailto:fuego-
> > bounces at lists.linuxfoundation.org] On Behalf Of Liu, Wenlong
> > Sent: Monday, November 13, 2017 2:38 AM
> > To: fuego at lists.linuxfoundation.org
> > Subject: Re: [Fuego] Fuego criteria file documentation
> >
> >
> > > -----Original Message-----
> > > From: fuego-bounces at lists.linuxfoundation.org
> > > [mailto:fuego-bounces at lists.linuxfoundation.org] On Behalf Of Bird,
> > > Timothy
> > > Sent: Thursday, November 02, 2017 8:53 AM
> > > To: fuego at lists.linuxfoundation.org
> > > Subject: [Fuego] Fuego criteria file documentation
> > >
> > > Hello all (but especially Daniel),
> > >
> > > I've been writing the documentation for new features in the Fuego
> > > 1.2
> > release,
> > > and I've put together a page about the new criteria file.
> > >
> > > The page is here:
> > > http://fuegotest.org/wiki/criteria.json
> > >
> > > Can you please take a look and see the information there is correct?
> > I have seen the current Fuego criteria documentation.
> > One question as below.
> >
> > As shown in the example of this documentation, {
> > "schema_version":"1.0",
> > "criteria": [
> > {
> > "tguid": "syscall",
> > "min_pass": 1000,
> > "max_fail": 5
> > },
> > {
> > "tguid": "timers",
> > "fail_ok_list": ["leapsec_timer"]
> > },
> > {
> > "tguid": "pty",
> > "must_pass_list": ["hangup01"]
> > }
> > ]
> > }
> > Yes, the example can works well.
> > However, you know, for LTP test, there are more than 70 sub-testsuites.
> >
> > For sub-testsuite named "net.nfs", the criteria json file cannot works
> > as I expected even though I added "net.nfs" as above.
> > When function "split_test_id(test_case_id)" was executed, the testset
> > name was changed to "default" and we cannot find the correct criteria
> json file.
> > The reason might be that "net.nfs" has a character ".".
>
> That was indeed the cause. The full test_case_id included multiple periods,
> which split_test_id did not handle correctly.
>
> >
> > So, any advice for this situation or it should be fixed?
> I corrected this routine, with commit: f50cff2 in the master branch.
>
> However, I didn't try to use a criteria file with with net.nfs. Can you
> check out the latest code and try again?
I have tested this and find one problem.
It'll give the error log as below,
... ...
Writing merged results to /fuego-rw/logs/Functional.LTP/results.json
Writing flat results to /fuego-rw/logs/Functional.LTP/flat_plot_data.txt
Traceback (most recent call last):
File "/fuego-core/engine/tests/Functional.LTP/parser.py", line 206, in <module>
sys.exit(plib.process(test_results))
File "/fuego-core/engine/scripts/parser/common.py", line 577, in process
make_chart_data(test_logdir, TESTDIR, CHART_CONFIG_JSON, data_lines)
File "/fuego-core/engine/scripts/parser/prepare_chart_data.py", line 703, in make_chart_data
chart_list = make_testset_summary_table(test_name, chart_config, entries)
File "/fuego-core/engine/scripts/parser/prepare_chart_data.py", line 622, in make_testset_summary_table
ssb_list.sort(ssb_cmp)
File "/fuego-core/engine/scripts/parser/prepare_chart_data.py", line 543, in ssb_cmp
spec,ts,bn = ssb.split(".")
ValueError: too many values to unpack
ERROR: results did not satisfy the threshold
... ...
My test job name is ubuntu.stress.part3.Functional.LTP.
When Fuego try to make the summary table, function "ssb_cmp" cannot handle this situation.
In order to continue my test work, I temporarily used the solution in the attachment.
But I strongly doubt if it'll be a good solution.
> >
> > PS:
> > * Attached is my criteria json file for AGL, although it cannot works
> > as I expected(reference only).
> > (LTP Version:20170516; Target: R-car M3)
> > * I didn't list all network related tests in this criteria json file
> > because of the issue above.
>
> Thanks for the bug report.
> -- Tim
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-chart-handle-test_set-or-test_spec-names-with-embedd.patch
Type: application/octet-stream
Size: 2601 bytes
Desc: 0001-chart-handle-test_set-or-test_spec-names-with-embedd.patch
URL: <http://lists.linuxfoundation.org/pipermail/fuego/attachments/20171120/10196309/attachment.obj>
More information about the Fuego
mailing list