[cgl_discussion] Re: Test cases for TIPC

Jon Maloy jon.maloy at ericsson.com
Wed Feb 26 14:30:34 PST 2003


The tests we have been running here were not made specifically
for TIPC, but were rather system tests for our proprietary
 TSP/TelORB system. These have turned out to be very good at
 stressing TIPC and to reveal weaknesses and bugs. Unfortunately
they are very tied to the TSP platform, both regarding build
tools and execution environment. The main focus has also been on
testing the core of TIPC via a propietary character device based
interface, rather than the socket interface. The latter has not been
much used until now.
In short, those test programs are not fit for open source.

So far we had not taken a systematic approach to test TIPC
via the socket interface. What exists is an inter processor
 'benchmark" program and a couple of smaller programs to
test limited parts of the socket interface. The former is a good
test of the behaviour of TIPC under high load, but it is rather
monotonous as a test program. I attach these files to this mail,
since I don't think there is any reason to upload them to 
sourceforge in their current state.

Recently I also wrote an informal spec for a test suite, defining
 a  comprehensive and coherent test of TIPC via the socket
interface. We have a student working on this, but I think this task
is way to demanding for one single person over only a few weeks.
I attach even this spec, so you can have an understanding of what I
am aiming at.
If you have resources to help developing this test suite I would be
very happy, and of course I will do what I can to support such an
effort, both via the phone and via email.

Regards /Jon






Lynch, Rusty wrote:

>Unless I missed something, there are no test cases for TIPC published on
>http://tipc.sourceforge.net/.  Do you have any test cases that you are
>planning on publishing?  Intel has a validation lab in China could help in
>this effort by taking what ever you have already and porting the test cases
>to work within the LTP.
>
>Once the test cases are in LTP then it will be that much easier to utilize
>some of our other resources like the OSDL testing lab via STP.
>
>    --rustyl
>  
>


-------------- next part --------------


                     Testing of TIPC
                     ---------------

General
-------
The test suite should consist of several programs, each testing
a limited part of the TIPC functionality.

These test programs should first be executed one at a time, to
verify that they work within an isolated scope where there are
no disturbances.

Each test program should run through a series of tests, where
it for each test clearly states 1) What is going to be tested,
and if necessary how. 2) The expected outcome. 3) After the test, 
it should print out the outcome, and in case of error (and if 
possible), excactly what went wrong.

Finally the test programs should be run simultaneously, starting
and stopping at random, to verify that they still work in an
environment with other ongoing activity, and even under high load.

I suggest that you start with setting up a cluster of 4 processors
with dual ethernet connectivity, and then start with the test
coordinator program to which the individal tests can register
themselves and receive orders about what to do. Regarding the
individual test programs you should start with a simple one, eg
the Name Table Test Program. You will then get a feeling for
how you can let the coordnator and each test program interact in
a generic way.


Test Coordinator Program
------------------------
The serial/parallel execution of the test programs can be controlled
from a central test coordinator, running at one of the processors.
This should collect all output infomation from the other processes,
and present it on the terminal.
The coordinato will also perform 'disturbing tasks' like  blocking 
links to test changeover, enabling/disabling ethernet cards to
simulate addition/loss of processors etc.


Name Table Test Program
-----------------------

1) Publish 1000 port names, some of them from the same port.
2) Call NAME_AVAILABLE to verify that the names are really
   published.
2) Withdraw these port names in a different order than they 
   were published.
2) Call NAME_AVAILABLE to verify that the names are really
   withdrawn.
3) Publish different types:
     Load shared. Many from same processor, and from
                 different processors
     Preferred load shared.Verify that you are refused a 
                  second publication.
     Illegal suite: lower > upper
     Overlapping suites: Verifiy that these are rejected.
     Publications with different visibility: Zone,Subnet,Processor
              Call NAME_AVAILABLE again to verify publication scope.


Name subscription Test Program
------------------------------
1) Open several subscriptons for <0,0,0xffffffff> from one socket.
   socket. This will in practice work like a network subscription,
   and one should receive an event per subscription each time a 
   processor goes down or comes back. Disable/enable the bearers
   on a processor to verify.
2) Call the syncronous NAME_AVAILABILITY for one name that does
   exist and for one that does not.


Network Subscription Test Program
---------------------------------
Open a subscripton for <0> from a socket. Disable/enable a processor.


Link Subscription Test Program
------------------------------
Open a subscripton for <0> from a socket. Block/deblock a link.


Message Transmission Test Program
---------------------------------
For all tests of message transfer one should verify both message 
contents and lenght. The easiest way to do this is to send the
message back to tye source and do a memcmp() with what was originally
sent out.

When applicable all the tests below should be performed both
between ports on the same processor and on different processors.

All message types should be tested for different message sizes:
0,1,299,301,340,440,1440,1500,6000,16000,66000,66001 octets.

Port Name Addressed Messages:
1) Send a message to an existing port name.
2) Send a message to a non-existing port name. 1024 first bytes 
   should be retured with error code NO_PORT_NAME.
3) Send a message to a port name, but from a connected port. 
   Should give error return value -1;
4) Send a message to an existing port name. The port name is 
   published from several ports on different processors as 
   "load shared". Verify round robin algorithm.
5) Send a message to an existing port name. The port name is 
   published from several ports as 'preferred load shared'. 
   Verify that the one published on the same processor is selected.

Direct Addressed Messages:
1) Send a message to an existing port identity. 
2) Send a message to a non-existing port identity on an existing
   processor. 1024 first bytes should be retured with error code 
   NO_PEER_PORT.
3) Send a message to a port identity on a non-existing processor. 
   1024 first bytes should be retured with error code NO_PEER_PROCESSOR.
4) Send a message to a port identity, but from a connected port. 
   Should give error return value.
5) Send a message to a port identity belonging to a connected port. 
   Should give a returned message with code NO_PEER_PORT (?)

Connection Oriented Messages:
1) Send a message back and forth on an established connection.
2) Try to send a connection oriented message from a non-connected
   port. Should give return value ILLEGAL_ARGS.
3) Do connect towards a non-existing port identity. Try to send
   a message. Should return a CONNECTION_ABORTED event along 
   with MSG_REJECTED and NO_PEER_PORT, and the first 1024 bytes
   of the message.


Connection Management Test Program
----------------------------------
These tests should be performed on one single processor as well
as between different processors.

1) Set up a connection.
2) Do close()  without prior shutdown() on one endpoint socket.
   Verify that you get an immediate CONNECTION_ABORTED with 
   NO_PEER_PORT at the other end. 
3) On a new connection, kill one of the enpoint processes.
   Verify that you get an immediate CONNECTION_ABORTED with 
   NO_PEER_PORT at the other end. 
4) On a new connection, make one of the enpoint processors unavailable.
   Verify that you get an immediate CONNECTION_ABORTED with 
   NO_PEER_PROCESSOR at the other end. 
5) On a new connection do shutdown() and close() on one endpoint.
   Verify that you get a CONNECTION_ABORTED with NO_PEER_PORT at 
   the other end within 2 hours.
6) Create a new connection. Create a third port and do connect()
   on that port towards one of the connection endpoint.
   Verify that you get a CONNECTION_ABORTED with NO_PEER_PORT at 
   the other end within 2 hours.
7) Create a new connection. Create a third port and do connect()
   on that port towards one of the connection endpoint.
   Send a message from the new port.
   Verify that you get an immediate CONNECTION_ABORTED with 
   NO_PEER_PORT(?) along with REJECTED_MSG and MSG_PENDING and 
   the 1024 first bytes of the message.


Link Failover Test Program
--------------------------

1) Make sure there are dual links between all processors. (This can 
   later on be simulated by enabling a second "udp" bearer, but this 
   requires some small changes to the current udp adaptation layer)
2) Start the High Load test program (see below).
3) Via the remote manager, block a link, and make sure that the 
   changeover works smoothly. The High Load test program should itself 
   detect and report disturbances.
4) After a while, deblock the link again. Verify that the changeback
   goes smoothly. The High Load test program should itself detect and
   report disturbances.
5) Disable one of the bearers on a processor. Verify as above.
6) After a while, enable the bearer again. Verify.
7) Repeat 1 through 6 several times, on different processors.


Management Functionality Test Program
--------------------------------------

In general the remote management service should be used by the
test coordinator program to simulate link failures, processors
crashes etc.
But before that, the management service itself must be tested.

Getting network topology:

1) Subscribe for <0,0,0xffffffff>. This will fire with 
   <0,processor_address>  for each processor becoming available
   or disappearing.
   Keep an up-to-date list of all processors available.

Getting some read-only data:

1) Send a TIPC_GET_PORTS to <0,one_of_the_processors>
2) Verify that the list of ports you get is non-zero and correct.
3) Pick one of the ports and send TIPC_GET_PORT_STATISTICS for that
   one.
4) Verify that you get a valid result.
5) Send TIPC_RESET_PORT_STATISTICS for the same port.
6) Verify that you get a valid result.

7) Send a TIPC_GET_NAME_TABLE to <0,one_of_the_processors>
8) Verify that you get a non-zero string as a result, and print
   it out.

9) Send a TIPC_GET_ROUTING_TABLE to <0,one_of_the_processors>
10) Verify that you get a non-zero string as a result, and print
    it out.

11) Send a TIPC_GET_BEARER_TYPES to <0,one_of_the_processors>
12) Verify that you get a non-zero string as a result, and print
    it out.

13) Send a TIPC_GET_BEARERS to the same processor.
14) Verify that the list of bearers you get is non-zero and correct.

15) Send a TIPC_GET_LINKS to <0,one_of_the_processors>
16) Verify that you get a non-zero string as a result, and print
    it out.
17) Send a TIPC_GET_LINK_STATISTICS for one of the links you got.
    Print out the result.
18) Send a TIPC_RESET_LINK_STATISTICS for the same link.Verify 
    reponse.
19) Send another TIPC_GET_LINK_STATISTICS for that link.
    Print out the result.

20) Send a TIPC_GET_PROCESSORS to <0,one_of_the_processors>
21) Verify that you get a list of procesors as result, and print
    it out.


Test of subscriptions, and manipulation of bearers:

1) Send TIPC_LINK_SUBSCRIBE to two of the processors.
2) Send TIPC_NETWORK_SUBSCRIBE to the same processors.
3) Send TIPC_NAME_SUBSCRIBE for <0,0,0xffffffff> to the same 
   processors.
4) Set up a management connection towards all the processors.
   For one of these, try to establish a second connection. It should be
   refused.
5) Pick one of the bearers on one of the two selected processors 
   and send TIPC_DISABLE_BEARER over the previously established 
   connection to that processor.
6) Verify that the link subscription fires with 'down' for all links 
    on that bearer, on that processor, and towards this processor.
7) Pick some other prosessors and send TIPC_DISABLE_BEARER for the
    other bearer on that processor.
    This will provoke a 'lost contact' between these two particular
    processors, while the remaining parts of the network is intact.
8) Verify that both the name subscription and the network subscription
    fires and indicate 'not available' on these two processors.
9) Send TIPC_ENABLE_BEARER for one of the bearers again.
10) Verify that the link subscription fires with 'up' for all links 
    on that bearer, and that the two processors concerned both report
    'available' for their peer. Both the name subscription and the
    network subscription should fire.
11) Send TIPC_ENABLE_BEARER for the other bearer.
12) Verify that the link subscription fires with 'up' for all 
    concerned links.

13) Send a TIPC_SET_LINK_WINDOW to one of the links on a processor.
    Use TIPC_GET_LINKS to obtain link names.
14) Send a TIPC_SET_LINK_PRIORITY to the same link.
15) Send a TIPC_SET_LINK_TOLERANCE to the same link.
16) Send a TIPC_GET_LINK_STATISTICS to verify the result.

17) Do a TIPC_BLOCK_LINK on a link. Verify that the link subsription 
    fires.
18) Do a TIPC_UNBLOCK_LINK on a link. Verify that the link subsription 
    fires.


High Load Test Program
----------------------

Start from the current benchmark program. Make sure that there is a 
process executing this program on each processor, reporting any
kind of problems to the central Test Cordinator Program.
Each of the processes should start with subscribing for all
other processors, and then launch start e.g. 16 processes/connections
for each discovered processor. Each connection should run a ping-pong
style message exchange with variable message size at full speed, i.e.
as soon as a process receives anything it responds with a message.
If any of the connections is aborted or becomes inactive this should
be reported to the coordinator.


Inter Zone Link Test Program
----------------------------

1) Set up two zones of different size, eg one with 3 and one with 
   4 procesors.
2) Establish a pilot link over udp between the two zones using 
   TIPC_CREATE_LINK.
3) The two zones should now automatically establish links so that
   all processor have at least 2 links to different processors in
   the other zone. Some processors in the smaller zone will have
   more than two, to satisfy the demand of the bigger zone, but 
   there should only be as many as strictly necessary. ("Minimum
   entropy algorithm").
4) Let the smaller zone grow first to 4, and then to 5 and 6 
   processors. Verify that the 'mininmum entropy' is maintained.
5) Let the second zone shrink to 3, then grow to four again.
   Verify that the 'mininmum entropy' is maintained.
6) Let the first zone shrink back to 3.
   Verify that the 'mininmum entropy' is maintained.
7) Set up a set of connections between the two zones, and let 
   traffic run at full speed, in  a similar manner as with the
   High Load Test program.
8) Run through the steps 4 through 6 again, with this traffic
   going on.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: tipc-benchmark.tar.gz
Type: application/x-gzip
Size: 7104 bytes
Desc: not available
Url : http://lists.linux-foundation.org/pipermail/cgl_discussion/attachments/20030226/638037b6/tipc-benchmark.tar-0001.bin


More information about the cgl_discussion mailing list