[Openais] openais: list of tests known to fail ??

[Steven Dake]

On Mon, 2005-07-25 at 20:01 -0700, Preety Puri wrote:
> Hi Steven,
> 
> Thanks for the prompt reply.
> 
> I upgraded openais to version (0.66) as suggested by you.
> 
> I have a few general questions regarding the package supplied tests.
> 
> Are the package supplied tests still valid ? You mentioned that
> saftests were going to be used from now onwards. Does that mean that
> the package supplied tests should not be used ??
> 

A little history..  We initially wrote some tests just to do some
dead-chicken testing on the services..  They were not thourough
(although the event tests are very thourough).  Then the saftest guys
came around and solved the problem for us.  As a result, the current
tests in the test directory have fallen into a bit of disrepair :(  Part
of this is the tests actually violate the specs in many ways, and as we
have fixed conformance-to-spec issues, the tests have broken
accordingly.  The benchmark programs should all work and Muni or I will
have a look at those.

> I did run some of the package supplied test code as mentioned in the
> QUICKSTART guide and had a few queries.
> 
> Can i use the default package supplied configuration file (groups.conf) 
> and be able to run all the tests under openais/test directory or would i 
> need to do something special with this file. I did change openais.conf
> to reflect my correct network settings and was able to run aisexec on
> two nodes and have testclm detect node addition/deletion but i am
> seeing some issues when i run the ckpt/evsbench tests.
> 
> testckpt - throws out a few failures ->
> FAILED expected SA_AIS_OK got SA_AIS_ERR_NOT_EXIST: Opening unlinked checkpoint
> FAILED expected SA_AIS_ERR_EXIST got SA_AIS_OK: creating section 2
> FAILED expected SA_AIS_OK got SA_AIS_ERR_INVALID_PARAM: Testing checkpoint writes
> FAILED expected SA_AIS_OK got SA_AIS_ERR_INVALID_PARAM: initialize section iterator
> FAILED expected SA_AIS_OK got SA_AIS_ERR_BAD_HANDLE: Get next section in iteartion
> The last iteration should fail
> FAILED expected SA_AIS_OK got SA_AIS_ERR_BAD_HANDLE: Finalize iteration
> 

i would say this shouldn't print out these failed errors, but the test
cases are faulty at this time.  for a better feeling of conformance,
have a look at the saftest project.

> 
> ckptbench ->
> prints out the benchmarks for the different write counts, then right 
> after that it prints out the following result ->
> saCkptCheckpointWrite result 7 (should be 1)
> 
> 

never seen this and cannot generate it.  Could you post a log of the
screen output before the checkpoint write fails?

> ckptbenchth -> 
> Thread0 seems to print the write count as 0 for all write_sizes, 
> write_size seems to increase from 100000 to 106000, seconds runtime is 0 
> all along, TP/s (is either 0.000 or nan) , MB/s (is either 0.000 or 
> nan).
> 

we will have a look.

> The result that is finally printed out at the end is ->
> saCkptCheckpointWrite result 9 (should be 1)
> 
> 
> evsbench ->
> prints benchmark values and at the same time it prints a lot of "try 
> again". What does "try again" indicate ?
> 

The try again output is just a junk debug statement that can be deleted.
A try again indicates the executive is "too busy" to handle any new
requests at this time and is used to flow control input to the
executive.  It tells the apis that it is too busy with a "try again"
call.  Then the caller can try again in a busy loop, or try again a
number of times before giving up, or reschedule the work at a later
time.

This is a critical part of ensuring the applications don't overload the
executive causing the executive to consume much more memory then the
system may have available.

regards
-steve

> Are these known issues ?
> 
> Thanks for all the help,
> -Preety.
> 
> 
> On Fri, 22 Jul 2005, Steven Dake wrote:
> 
> > On Fri, 2005-07-22 at 16:38 -0700, Preety Puri wrote:
> >> Hi,
> >>
> >> Does anyone have logs or know a list of failures for
> >> openais's (0.65) package test code.
> >>
> >> The QUICKSTART mentions that testclm, testamf, testckpt,
> >> ckptbench, ckptbenchth, testevt, testevs, evsbench can be run
> >> to validate this package.
> >>
> >> But i could'nt find a list of  known failures for any of these
> >> tests (openais/test - directory).
> >>
> >> If anyone knows a set of known issues/failures for these tests,
> >> please send them across or point me to a location where
> >> i can get them.
> >>
> >> Appreciate all the help.
> >>
> >
> > Preety
> > I'd suggest upgrading to 0.66.  Several bugs have been fixed.
> >
> > Our plan going forward with regards to testing is to use the saftest
> > suite available at www.saf-test.org.  We will publish for each release a
> > code coverage report and test suite results report.
> >
> > The code coverage report is at
> > http://developer.osdl.org/cr/coverage-openais-0.66/index.html
> >
> > The test results for saftest are publlished at:
> > http://developer.osdl.org/dev/openais/conf/conformance-openais-0.66/index.html
> >
> >
> >> Thanks,
> >> -Preety.
> >> _______________________________________________
> >> Openais mailing list
> >> Openais at lists.osdl.org
> >> https://lists.osdl.org/mailman/listinfo/openais
> >