[lsb-discuss] ISP RAS framework issues/feedback/questions
markovtsev at linuxtesting.org
Tue Aug 1 08:15:01 PDT 2006
Thanks for you time spent with OLVER framework. Please, do not hesitate to
contact us in the case of any questions. It does not matter if the question
is covered by the documents or not. I think that live collaboration is much
more productive than reading reference documentation, especially in the
1. pre/post blocks
a) pre/post blocks are connected with function under test. pre block checks
the validity of input parameters and other prerequisites in which the
function call can be issued. pre block can allow or forbid actual target
function invocation. When pre block allows invocation then remote call to
the agent is done. If pre block forbids invocation then this situation is
considered as a bug in the scenario (test case). The basic example when pre
block forbids function invocation is when input parameters satisfy
conditions of the standard which state that in this case function behavior
is undefined. So pre blocks protect us from disruptive calls which can break
the system under test.
pre block is something unnatural for the general testing approach (without
pre/post conditions when we always sure about the aim of the function call
when we do it). In the case of UniTESK it is possible to automatically
generate function calls. Sometimes this can lead to violation of a pre
block. It is worth mention that it is possible to re-use pre block code and
make the check in advance to eliminate the attempt of invalid invocation. In
this case scenario will continue to generate another calls and no error will
Another example of reusing pre block code is checking correctness of the
tests ported. If tests are used as unit tests before porting, they could
check functionality outside of the scope of the standard. Pre blocks help to
catch such cases, which shall be excluded from conformance test suite.
b) post block checks the result of a function call. It compares the actual
result with the result expected by the test suite.
Both pre and post blocks are executed at the test suite side. These blocks
can not be executed at the agent side because they operate with model
representation of function under test. Our intention was to make the agent
lightweight so it is not aware of model representation.
2. porting existing and developing new tests
a) The estimates depend upon what we'll imply by the "porting". I can break
"porting" into several levels:
1. Change the way the test gets configured and executed and the way
it reports the status (PASS, FAIL, .). In this case the original test will
remain almost unchanged.
2. Create the requirements catalogue, annotate assertions in the
code and organize tracing which will allow the framework to build
requirement coverage and error reports for this test. In this case the
original test will be annotated with some macros and some infrastructure
work needs to be done.
3. Make use of pre/post facilities and improve test quality. In this
case the original test will change significantly.
For libxml2 reaching level 1 will require about a week, levels 2 and 3
should be estimated carefully and this requires a pre-study. Roughly
reaching level 2 may require several weeks and level 3 - some more time
depending upon the desired quality of tests.
b) This is a very good question. Indeed, usually formal specifications and
methods have problems with complex data, pointers and so on. Nevertheless
our framework works well here. It is not necessary to transfer big objects
back and fourth in all cases. If an internal structure of an object is not
interesting for us, we don't transfer it and work only with pointers to the
object. But if we want to check the internal state of the object we need to
implement marshalling/unmarshalling. Please, note that
marshalling/unmarshalling is used only along with the remote agent. So it is
not an indispensable part of our framework because it is possible to write
tests that do not use the agent.
It is worth mention that our facilities in test data and test sequences
generation are more effective in the case of complex data. So the additional
efforts can be balanced by additional benefits.
3. We do intend to support all 7 architectures. And there are no any
obstacles to do it. Currently we are limited in the resources and we support
IA32 only. We haven't yet implemented an automatic retargeting mechanism and
if somebody wants to check the test suite on another architecture she needs
to do some work manually.
> -----Original Message-----
> From: Banginwar, Rajesh [mailto:rajesh.banginwar at intel.com]
> Sent: Tuesday, August 01, 2006 9:18 AM
> To: Denis Markovtsev; lsb-discuss at lists.freestandards.org
> Subject: RE: [lsb-discuss] ISP RAS framework issues/feedback/questions
> Thanks for the detailed response. I think things are getting
> clearer now. I think you have answered some of the basic questions we
> were asking till now. Here I will attempt to put together some of our
> observations and comments from what we have understood so far (instead
> of following up on your responses; so nothing inline in this email).
> Please pardon me if I am asking a question that is already answered in
> one of your document as there was a lot to go thr' in such a short
> 1. Are pre/post blocks used for validating the parameters for function
> under test or the test case itself? Many times a function under test can
> only be called after creating a required scenario for it. If the pre
> block is for validating parameters for test case (not the function under
> test), what exactly is its use? Shouldn't this really be done on the
> agent side to validate parameters for function under test? In general,
> the usage of pre-post block was not clear from the docs... Instead of
> taking a simple example like abs, it may be useful to illustrate with
> slightly complex set of parameters.
> 2. I would like to look at this framework from two points of views:
> a. Porting existing tests (e.g. ones taken from upstream and not
> necessarily the ones we have in LSB): In this case, my general
> observation is that the framework will force a lot of re-write. If we
> try to take tests from upstream, then this framework will require a lot
> of changes to the tests. e.g. for libxml2 tests found in LSB CVS, it
> took us about 1 week to "port" it to TET-Lite. What is the estimate for
> converting it to RAS framework?
> b. Developing new tests: For LSB-Core like libraries where many
> of the interfaces are primitives with very little complexity in
> parameters, this framework seems to be a very good choice. As soon as
> the parameters get complex (pointers, structures and combinations of
> those), I think the test developer will spend a lot of time marshaling
> and unmarshaling the parameters while sending test from server to agent.
> Now, I am making this comment based on the assumption that the agent
> side test case will always represent one particular "assert" and tests
> require complex data as input (or globals) which I think is the case for
> many of the higher level libraries. There are many examples in GTK tests
> like the ones in GtkMenu.inp file.
> If this marshalling and unmarshalling is really as I understand
> it will add a lot of overhead for libraries similar to the ones in
> current desktop test suite.
> 3. Does the existing RAS tests cover all 7 LSB supported architectures?
> Any limitations for supporting all these archs... I don't see any, but
> thought I should ask it anyway...
> That's all for now,
More information about the lsb-discuss