[Openais] corosync trunk - remove hdb in totemsrp and below stacks

Steven Dake sdake at redhat.com
Wed Jul 22 14:58:07 PDT 2009


The HDB is a great mechanism for ensuring concurrent operations on a
context occur in a correct way.  Unfortunately it uses pthread_spin_lock
and pthread_spin_unlock.  For every call into totemsrp, totemrrp,
totemnet, totemudp, and totemiba, there is a matched pair of get and put
operations which result in 4 memory barriers (a very expensive
operation) per layer per call.

This patch removes the use of hdb and instead uses a locally allocated
context.  We rely on totempg's HDB to ensure operations are handled
properly in a concurrent fashion.  After that, the context pointers
never change so there is no reason to add the extra overhead of these
HDB calls.

Performance numbers prior to patch for single node cpgbench on 5530 cpu:
[root at bigiron test]# ./cpgbench
1334392 messages received   100 bytes per write  10.000 Seconds runtime
133439.013 TP/s  13.344 MB/s.
1317752 messages received   400 bytes per write  10.000 Seconds runtime
131774.897 TP/s  52.710 MB/s.
1221195 messages received   700 bytes per write  10.000 Seconds runtime
122119.329 TP/s  85.484 MB/s.
1203771 messages received  1000 bytes per write  10.000 Seconds runtime
120377.004 TP/s 120.377 MB/s.
1176193 messages received  1300 bytes per write  10.000 Seconds runtime
117618.441 TP/s 152.904 MB/s.
1156328 messages received  1600 bytes per write  10.000 Seconds runtime
115632.465 TP/s 185.012 MB/s.
1122111 messages received  1900 bytes per write  10.000 Seconds runtime
112210.786 TP/s 213.200 MB/s.
1097070 messages received  2200 bytes per write  10.000 Seconds runtime
109706.748 TP/s 241.355 MB/s.


Post patch:
[root at bigiron test]# ./cpgbench
1385864 messages received   100 bytes per write  10.000 Seconds runtime
138586.192 TP/s  13.859 MB/s.
1370641 messages received   400 bytes per write  10.000 Seconds runtime
137063.894 TP/s  54.826 MB/s.
1279496 messages received   700 bytes per write  10.000 Seconds runtime
127949.446 TP/s  89.565 MB/s.
1246959 messages received  1000 bytes per write  10.000 Seconds runtime
124695.676 TP/s 124.696 MB/s.
1231739 messages received  1300 bytes per write  10.000 Seconds runtime
123173.691 TP/s 160.126 MB/s.
1211951 messages received  1600 bytes per write  10.000 Seconds runtime
121194.906 TP/s 193.912 MB/s.
1181160 messages received  1900 bytes per write  10.000 Seconds runtime
118115.835 TP/s 224.420 MB/s.
1167084 messages received  2200 bytes per write  10.000 Seconds runtime
116708.295 TP/s 256.758 MB/s.

as can be seen, tps and mb/sec increase roughly 4% on this hardware.  On
my laptop the performance is more like 20% increase (because memory
barriers are such an expensive operation).

Regards
-steve
-------------- next part --------------
A non-text attachment was scrubbed...
Name: corosync-trunk-totem-remove-hdb.patch
Type: text/x-patch
Size: 68103 bytes
Desc: not available
Url : http://lists.linux-foundation.org/pipermail/openais/attachments/20090722/c5684a46/attachment-0001.bin 


More information about the Openais mailing list