[Openais] threaded send operation for openais
Mark Haverkamp
markh at osdl.org
Wed Jul 6 14:45:24 PDT 2005
On Wed, 2005-07-06 at 12:50 -0700, Steven Dake wrote:
> I spent about 3 hours hacking this up so its not polished. Basically
> what it does is create work threads with queues. When the executive
> wants to send a message, it adds the message to the work queue. The
> worker thread then encrypts and sends the message on the wire. The # of
> worker threads are configured via WORKER_THREAD_COUNT. The advantage
> should be better performance on multiprocessors. I find performance
> doesn't improve on my multiprocessor machines. However, the cpus are
> more evenly used on multiprocessors.
>
> I'm interested in people that could test this out and let me know what
> kind of results (with evsbench) they find vs what is currently in
> subversion. I didn't really understand threads when I started this
> patch and just read the man pages. As a result, there may be something
> I am doing that could be done in a more performant way.. Any
> suggestions welcomed.
>
> regards
> -steve
Hi Steve,
I tried this out on my cluster. They are 4 nodes of 2 867mhz processors
each. It looks like the unthreaded test had better numbers. One odd
thing that I noticed about the threaded version is that its size is
around 35Meg as opposed to the unthreaded version less than 4Meg.
threaded:
10987 root 16 0 36828 35M 1488 R 99.9 3.5 0:21 aisexec
unthreaded:
11000 root 16 0 3784 3784 1428 R 74.6 0.3 0:03 aisexec
Mark.
Threaded version:
# ./evsbench
Init result 1
Join result 1
Leave result 1
CONFIGURATION CHANGE
--------------------
New configuration
192.168.1.8
192.168.1.17
192.168.1.18
192.168.1.19
Members Left:
Members Joined:
192.168.1.19
59518 Writes 1 bytes per write 10.002 Seconds runtime 5950.714 TP/s 0.006 MB/s.
27334 Writes 1001 bytes per write 10.003 Seconds runtime 2732.656 TP/s 2.735 MB/s.
18041 Writes 2001 bytes per write 10.003 Seconds runtime 1803.643 TP/s 3.609 MB/s.
14411 Writes 3001 bytes per write 10.002 Seconds runtime 1440.745 TP/s 4.324 MB/s.
10797 Writes 4001 bytes per write 10.002 Seconds runtime 1079.432 TP/s 4.319 MB/s.
9074 Writes 5001 bytes per write 10.005 Seconds runtime 906.979 TP/s 4.536 MB/s.
8107 Writes 6001 bytes per write 10.002 Seconds runtime 810.514 TP/s 4.864 MB/s.
7206 Writes 7001 bytes per write 10.003 Seconds runtime 720.385 TP/s 5.043 MB/s.
6439 Writes 8001 bytes per write 10.003 Seconds runtime 643.729 TP/s 5.150 MB/s.
5283 Writes 9001 bytes per write 10.002 Seconds runtime 528.195 TP/s 4.754 MB/s.
4885 Writes 10001 bytes per write 10.006 Seconds runtime 488.225 TP/s 4.883 MB/s.
4437 Writes 11001 bytes per write 10.006 Seconds runtime 443.413 TP/s 4.878 MB/s.
4214 Writes 12001 bytes per write 10.001 Seconds runtime 421.339 TP/s 5.056 MB/s.
3888 Writes 13001 bytes per write 10.002 Seconds runtime 388.734 TP/s 5.054 MB/s.
3566 Writes 14001 bytes per write 10.007 Seconds runtime 356.353 TP/s 4.989 MB/s.
3333 Writes 15001 bytes per write 10.005 Seconds runtime 333.139 TP/s 4.997 MB/s.
3087 Writes 16001 bytes per write 10.002 Seconds runtime 308.644 TP/s 4.939 MB/s.
2801 Writes 17001 bytes per write 10.003 Seconds runtime 280.030 TP/s 4.761 MB/s.
2728 Writes 18001 bytes per write 10.007 Seconds runtime 272.617 TP/s 4.907 MB/s.
2524 Writes 19001 bytes per write 10.003 Seconds runtime 252.336 TP/s 4.795 MB/s.
2368 Writes 20001 bytes per write 10.002 Seconds runtime 236.762 TP/s 4.735 MB/s.
2253 Writes 21001 bytes per write 10.002 Seconds runtime 225.249 TP/s 4.730 MB/s.
2158 Writes 22001 bytes per write 10.002 Seconds runtime 215.767 TP/s 4.747 MB/s.
2039 Writes 23001 bytes per write 10.002 Seconds runtime 203.862 TP/s 4.689 MB/s.
1979 Writes 24001 bytes per write 10.002 Seconds runtime 197.863 TP/s 4.749 MB/s.
1937 Writes 25001 bytes per write 10.008 Seconds runtime 193.542 TP/s 4.839 MB/s.
1841 Writes 26001 bytes per write 10.002 Seconds runtime 184.072 TP/s 4.786 MB/s.
1770 Writes 27001 bytes per write 10.002 Seconds runtime 176.967 TP/s 4.778 MB/s.
1682 Writes 28001 bytes per write 10.002 Seconds runtime 168.169 TP/s 4.709 MB/s.
1624 Writes 29001 bytes per write 10.008 Seconds runtime 162.271 TP/s 4.706 MB/s.
1568 Writes 30001 bytes per write 10.002 Seconds runtime 156.775 TP/s 4.703 MB/s.
1494 Writes 31001 bytes per write 10.004 Seconds runtime 149.337 TP/s 4.630 MB/s.
1433 Writes 32001 bytes per write 10.006 Seconds runtime 143.212 TP/s 4.583 MB/s.
1400 Writes 33001 bytes per write 10.001 Seconds runtime 139.981 TP/s 4.620 MB/s.
1350 Writes 34001 bytes per write 10.002 Seconds runtime 134.975 TP/s 4.589 MB/s.
1279 Writes 35001 bytes per write 10.007 Seconds runtime 127.809 TP/s 4.473 MB/s.
1272 Writes 36001 bytes per write 10.002 Seconds runtime 127.180 TP/s 4.579 MB/s.
1201 Writes 37001 bytes per write 10.002 Seconds runtime 120.078 TP/s 4.443 MB/s.
1165 Writes 38001 bytes per write 10.002 Seconds runtime 116.476 TP/s 4.426 MB/s.
1074 Writes 39001 bytes per write 10.002 Seconds runtime 107.383 TP/s 4.188 MB/s.
1054 Writes 40001 bytes per write 10.002 Seconds runtime 105.380 TP/s 4.215 MB/s.
1078 Writes 41001 bytes per write 10.001 Seconds runtime 107.789 TP/s 4.419 MB/s.
1051 Writes 42001 bytes per write 10.002 Seconds runtime 105.077 TP/s 4.413 MB/s.
1049 Writes 43001 bytes per write 10.002 Seconds runtime 104.876 TP/s 4.510 MB/s.
1016 Writes 44001 bytes per write 10.001 Seconds runtime 101.591 TP/s 4.470 MB/s.
996 Writes 45001 bytes per write 10.002 Seconds runtime 99.581 TP/s 4.481 MB/s.
966 Writes 46001 bytes per write 10.002 Seconds runtime 96.582 TP/s 4.443 MB/s.
948 Writes 47001 bytes per write 10.008 Seconds runtime 94.727 TP/s 4.452 MB/s.
938 Writes 48001 bytes per write 10.002 Seconds runtime 93.782 TP/s 4.502 MB/s.
912 Writes 49001 bytes per write 10.001 Seconds runtime 91.189 TP/s 4.468 MB/s.
[root at cl008 test]#
Unthreaded version:
# ./evsbench
Init result 1
Join result 1
Leave result 1
CONFIGURATION CHANGE
--------------------
New configuration
192.168.1.8
192.168.1.17
192.168.1.18
192.168.1.19
Members Left:
Members Joined:
192.168.1.19
66435 Writes 1 bytes per write 10.002 Seconds runtime 6641.982 TP/s 0.007 MB/s.
28697 Writes 1001 bytes per write 10.002 Seconds runtime 2869.180 TP/s 2.872 MB/s.
18367 Writes 2001 bytes per write 10.003 Seconds runtime 1836.176 TP/s 3.674 MB/s.
14353 Writes 3001 bytes per write 10.003 Seconds runtime 1434.858 TP/s 4.306 MB/s.
10502 Writes 4001 bytes per write 10.004 Seconds runtime 1049.766 TP/s 4.200 MB/s.
8922 Writes 5001 bytes per write 10.003 Seconds runtime 891.907 TP/s 4.460 MB/s.
7670 Writes 6001 bytes per write 10.001 Seconds runtime 766.927 TP/s 4.602 MB/s.
6733 Writes 7001 bytes per write 11.374 Seconds runtime 591.945 TP/s 4.144 MB/s.
6016 Writes 8001 bytes per write 10.002 Seconds runtime 601.459 TP/s 4.812 MB/s.
5026 Writes 9001 bytes per write 10.001 Seconds runtime 502.556 TP/s 4.524 MB/s.
4650 Writes 10001 bytes per write 10.002 Seconds runtime 464.915 TP/s 4.650 MB/s.
4282 Writes 11001 bytes per write 10.005 Seconds runtime 427.998 TP/s 4.708 MB/s.
3957 Writes 12001 bytes per write 10.004 Seconds runtime 395.548 TP/s 4.747 MB/s.
3698 Writes 13001 bytes per write 10.006 Seconds runtime 369.572 TP/s 4.805 MB/s.
3466 Writes 14001 bytes per write 10.003 Seconds runtime 346.506 TP/s 4.851 MB/s.
3288 Writes 15001 bytes per write 10.007 Seconds runtime 328.561 TP/s 4.929 MB/s.
3064 Writes 16001 bytes per write 10.003 Seconds runtime 306.311 TP/s 4.901 MB/s.
2896 Writes 17001 bytes per write 10.006 Seconds runtime 289.438 TP/s 4.921 MB/s.
2760 Writes 18001 bytes per write 10.007 Seconds runtime 275.796 TP/s 4.965 MB/s.
2596 Writes 19001 bytes per write 10.004 Seconds runtime 259.502 TP/s 4.931 MB/s.
2448 Writes 20001 bytes per write 10.003 Seconds runtime 244.722 TP/s 4.895 MB/s.
2285 Writes 21001 bytes per write 10.003 Seconds runtime 228.425 TP/s 4.797 MB/s.
2178 Writes 22001 bytes per write 10.009 Seconds runtime 217.595 TP/s 4.787 MB/s.
2078 Writes 23001 bytes per write 10.006 Seconds runtime 207.685 TP/s 4.777 MB/s.
2035 Writes 24001 bytes per write 10.001 Seconds runtime 203.475 TP/s 4.884 MB/s.
1944 Writes 25001 bytes per write 10.006 Seconds runtime 194.290 TP/s 4.857 MB/s.
1868 Writes 26001 bytes per write 10.005 Seconds runtime 186.701 TP/s 4.854 MB/s.
1799 Writes 27001 bytes per write 10.005 Seconds runtime 179.814 TP/s 4.855 MB/s.
1704 Writes 28001 bytes per write 10.002 Seconds runtime 170.371 TP/s 4.771 MB/s.
1667 Writes 29001 bytes per write 10.002 Seconds runtime 166.664 TP/s 4.833 MB/s.
1605 Writes 30001 bytes per write 10.006 Seconds runtime 160.408 TP/s 4.812 MB/s.
1549 Writes 31001 bytes per write 10.002 Seconds runtime 154.874 TP/s 4.801 MB/s.
1500 Writes 32001 bytes per write 10.002 Seconds runtime 149.972 TP/s 4.799 MB/s.
1464 Writes 33001 bytes per write 10.002 Seconds runtime 146.373 TP/s 4.830 MB/s.
1364 Writes 34001 bytes per write 10.002 Seconds runtime 136.374 TP/s 4.637 MB/s.
1347 Writes 35001 bytes per write 10.002 Seconds runtime 134.669 TP/s 4.714 MB/s.
1345 Writes 36001 bytes per write 10.003 Seconds runtime 134.463 TP/s 4.841 MB/s.
1295 Writes 37001 bytes per write 10.003 Seconds runtime 129.462 TP/s 4.790 MB/s.
1260 Writes 38001 bytes per write 10.002 Seconds runtime 125.979 TP/s 4.787 MB/s.
1222 Writes 39001 bytes per write 10.002 Seconds runtime 122.180 TP/s 4.765 MB/s.
1171 Writes 40001 bytes per write 10.002 Seconds runtime 117.078 TP/s 4.683 MB/s.
1157 Writes 41001 bytes per write 10.002 Seconds runtime 115.678 TP/s 4.743 MB/s.
1127 Writes 42001 bytes per write 10.001 Seconds runtime 112.691 TP/s 4.733 MB/s.
1115 Writes 43001 bytes per write 10.002 Seconds runtime 111.479 TP/s 4.794 MB/s.
1084 Writes 44001 bytes per write 10.002 Seconds runtime 108.375 TP/s 4.769 MB/s.
1060 Writes 45001 bytes per write 10.001 Seconds runtime 105.985 TP/s 4.769 MB/s.
1015 Writes 46001 bytes per write 10.002 Seconds runtime 101.476 TP/s 4.668 MB/s.
1016 Writes 47001 bytes per write 10.002 Seconds runtime 101.581 TP/s 4.774 MB/s.
1001 Writes 48001 bytes per write 10.001 Seconds runtime 100.087 TP/s 4.804 MB/s.
972 Writes 49001 bytes per write 10.002 Seconds runtime 97.181 TP/s 4.762 MB/s.
[root at cl008 test]#
--
Mark Haverkamp <markh at osdl.org>
More information about the Openais
mailing list