defect 1170 - assert in memb_state_recover_enter (was) Re:
[Openais] looks like the synchronization code is still broke
Fabien THOMAS
fabien.thomas at netasq.com
Thu Apr 6 07:20:16 PDT 2006
Here i've just 4 nodes with latest version. When i kill one node (not
the one that produce this log output) at least one other node
crash (not crash but exit with the exit(1) in the code: do you want a
core at this point ? ).
After some restart the cluster seems to be synchronized again.
Apr 06 8:00:21 [NOTICE ] [CLM ] got nodejoin message 10.2.11.5
Apr 06 8:00:21 [NOTICE ] [CLM ] got nodejoin message 10.2.20.254
Apr 06 8:00:21 [NOTICE ] [CLM ] got nodejoin message 10.2.1.6
Apr 06 8:00:21 [NOTICE ] [CLM ] got nodejoin message 10.2.1.7
Apr 06 8:00:21 [NOTICE ] [SYNC ] Synchronization barrier completed
Apr 06 8:00:21 [NOTICE ] [SYNC ] Synchronization actions starting
for (openais checkpoint service B.01.01)
Apr 06 8:00:21 [NOTICE ] [SYNC ] Synchronization actions done for
(openais checkpoint service B.01.01)
Apr 06 8:00:21 [NOTICE ] [SYNC ] Synchronization barrier completed
Apr 06 8:00:21 [NOTICE ] [SYNC ] Synchronization actions starting
for (openais event service B.01.01)
Apr 06 8:00:21 [NOTICE ] [SYNC ] Synchronization actions done for
(openais event service B.01.01)
Apr 06 14:12:40 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:12:40 [NOTICE ] [TOTEM] Saving state aru 218d3 high seq
received 218d3
Apr 06 14:12:40 [NOTICE ] [TOTEM] Storing new sequence id for ring
72068
Apr 06 14:12:40 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:12:40 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:12:40 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:12:40 [NOTICE ] [TOTEM] Storing new sequence id for ring
72072
Apr 06 14:12:40 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:12:40 [NOTICE ] [TOTEM] HeartBeat Timer expired Invoking
token loss mechanism in state 3
Apr 06 14:12:40 [NOTICE ] [TOTEM] The token was lost in state 3 from
timer 83c6000
Apr 06 14:12:40 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:12:40 [NOTICE ] [TOTEM] Storing new sequence id for ring
72076
Apr 06 14:12:40 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:12:40 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:12:40 [NOTICE ] [TOTEM] Storing new sequence id for ring
72080
Apr 06 14:12:40 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:12:40 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:12:40 [NOTICE ] [TOTEM] position [0] member 10.2.1.6:
Apr 06 14:12:40 [NOTICE ] [TOTEM] previous ring seq 72064 rep 10.2.1.6
Apr 06 14:12:40 [NOTICE ] [TOTEM] aru 218d3 high delivered 218d3
received flag 0
Apr 06 14:12:40 [NOTICE ] [TOTEM] position [1] member 10.2.1.7:
Apr 06 14:12:40 [NOTICE ] [TOTEM] previous ring seq 72064 rep 10.2.1.6
Apr 06 14:12:40 [NOTICE ] [TOTEM] aru 218d3 high delivered 218d3
received flag 0
Apr 06 14:12:40 [NOTICE ] [TOTEM] position [2] member 10.2.20.254:
Apr 06 14:12:40 [NOTICE ] [TOTEM] previous ring seq 72064 rep 10.2.1.6
Apr 06 14:12:40 [NOTICE ] [TOTEM] aru 218d3 high delivered 218d3
received flag 0
Apr 06 14:12:40 [NOTICE ] [TOTEM] copying all old ring messages from
218d4-218d3.
Apr 06 14:12:40 [NOTICE ] [TOTEM] Originated 0 messages in RECOVERY.
Apr 06 14:12:40 [NOTICE ] [TOTEM] Originated for recovery:
Apr 06 14:12:40 [NOTICE ] [TOTEM] Not Originated for recovery:
Apr 06 14:12:40 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:12:40 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:12:40 [NOTICE ] [CLM ] 10.2.1.6
Apr 06 14:12:40 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:12:40 [NOTICE ] [CLM ] 10.2.20.254
Apr 06 14:12:40 [NOTICE ] [CLM ] Members Left:
Apr 06 14:12:40 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:12:40 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:12:40 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:12:40 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:12:40 [NOTICE ] [CLM ] 10.2.1.6
Apr 06 14:12:40 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:12:40 [NOTICE ] [CLM ] 10.2.20.254
Apr 06 14:12:40 [NOTICE ] [CLM ] Members Left:
Apr 06 14:12:40 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:12:40 [NOTICE ] [SYNC ] This node is within the non-
primary component and will NOT provide any services.
Apr 06 14:12:41 [NOTICE ] [TOTEM] entering OPERATIONAL state.
Apr 06 14:12:41 [NOTICE ] [YKD ] This processor is within the
primary component.
Apr 06 14:12:41 [NOTICE ] [SYNC ] This node is within the primary
component and will provide service.
Apr 06 14:12:41 [NOTICE ] [SYNC ] Synchronization barrier completed
Apr 06 14:12:41 [NOTICE ] [SYNC ] Synchronization actions starting
for (openais cluster membership service B.01.01)
Apr 06 14:12:41 [NOTICE ] [SYNC ] Synchronization actions done for
(openais cluster membership service B.01.01)
Apr 06 14:12:41 [NOTICE ] [CLM ] got nodejoin message 10.2.1.6
Apr 06 14:12:41 [NOTICE ] [CLM ] got nodejoin message 10.2.1.7
Apr 06 14:12:41 [NOTICE ] [CLM ] got nodejoin message 10.2.20.254
Apr 06 14:12:41 [NOTICE ] [SYNC ] Synchronization barrier completed
Apr 06 14:12:41 [NOTICE ] [SYNC ] Synchronization actions starting
for (openais checkpoint service B.01.01)
Apr 06 14:12:41 [NOTICE ] [SYNC ] Synchronization actions done for
(openais checkpoint service B.01.01)
Apr 06 14:12:41 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:12:41 [NOTICE ] [TOTEM] Saving state aru c8 high seq
received c8
Apr 06 14:12:41 [NOTICE ] [TOTEM] Storing new sequence id for ring
72084
Apr 06 14:12:41 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:12:41 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:12:41 [NOTICE ] [TOTEM] position [0] member 10.2.1.6:
Apr 06 14:12:41 [NOTICE ] [TOTEM] previous ring seq 72080 rep 10.2.1.6
Apr 06 14:12:41 [NOTICE ] [TOTEM] aru b7 high delivered b7 received
flag 0
Apr 06 14:12:41 [NOTICE ] [TOTEM] position [1] member 10.2.1.7:
Apr 06 14:12:41 [NOTICE ] [TOTEM] previous ring seq 72080 rep 10.2.1.6
Apr 06 14:12:41 [NOTICE ] [TOTEM] aru c8 high delivered ac received
flag 0
Apr 06 14:12:41 [NOTICE ] [TOTEM] position [2] member 10.2.11.5:
Apr 06 14:12:41 [NOTICE ] [TOTEM] previous ring seq 72064 rep 10.2.11.5
Apr 06 14:12:41 [NOTICE ] [TOTEM] aru 0 high delivered 0 received
flag 0
Apr 06 14:12:41 [NOTICE ] [TOTEM] position [3] member 10.2.20.254:
Apr 06 14:12:41 [NOTICE ] [TOTEM] previous ring seq 72080 rep 10.2.1.6
Apr 06 14:12:41 [NOTICE ] [TOTEM] aru d9 high delivered ac received
flag 0
Apr 06 14:12:41 [NOTICE ] [TOTEM] copying all old ring messages from
b8-c8.
Apr 06 14:12:41 [NOTICE ] [TOTEM] Originated 17 messages in RECOVERY.
Apr 06 14:12:41 [NOTICE ] [TOTEM] Originated for recovery: b8 b9 ba
bb bc bd be bf c0 c1 c2 c3 c4 c5 c6 c7 c8
Apr 06 14:12:41 [NOTICE ] [TOTEM] Not Originated for recovery:
Apr 06 14:12:41 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:42 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:42 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:42 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:42 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:42 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:42 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:42 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:42 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:42 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:42 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:42 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:42 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:42 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:43 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:43 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:43 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:43 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:43 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:43 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:43 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:43 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:43 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:43 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:43 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:43 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:43 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:44 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:44 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:44 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:44 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:44 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:44 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:45 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:45 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:45 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:45 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:46 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:46 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:46 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:46 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:47 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:47 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:47 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:47 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:48 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:48 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:48 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:48 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:49 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:49 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e
Apr 06 14:12:49 [ERROR ] [TOTEM] FAILED TO RECEIVE
Apr 06 14:12:51 [NOTICE ] [MAIN ] AIS Executive Service: Copyright
(C) 2002-2006 MontaVista Software, Inc. and contributors.
Apr 06 14:12:51 [WARNING ] [MAIN ] Could not lock memory of service
to avoid page faults
Apr 06 14:12:51 [NOTICE ] [TOTEM] Token Timeout (1000 ms) retransmit
timeout (238 ms)
Apr 06 14:12:51 [NOTICE ] [TOTEM] token hold (180 ms) retransmits
before loss (4 retrans)
Apr 06 14:12:51 [NOTICE ] [TOTEM] join (100 ms) consensus (200 ms)
merge (200 ms)
Apr 06 14:12:51 [NOTICE ] [TOTEM] downcheck (1000 ms) fail to recv
const (50 msgs)
Apr 06 14:12:51 [NOTICE ] [TOTEM] seqno unchanged const (30
rotations) Maximum network MTU 1500
Apr 06 14:12:51 [NOTICE ] [TOTEM] window size per rotation (50
messages) maximum messages per rotation (17 messages)
Apr 06 14:12:51 [NOTICE ] [TOTEM] send threads (0 threads)
Apr 06 14:12:51 [NOTICE ] [TOTEM] heartbeat_failures_allowed (3)
Apr 06 14:12:51 [NOTICE ] [TOTEM] max_network_delay (50 ms)
Apr 06 14:12:51 [NOTICE ] [TOTEM] total heartbeat_timeout (764 ms)
Apr 06 14:12:51 [NOTICE ] [TOTEM] Receive multicast socket recv
buffer size (144000 bytes).
Apr 06 14:12:51 [NOTICE ] [TOTEM] Transmit multicast socket send
buffer size (144000 bytes).
Apr 06 14:12:51 [NOTICE ] [TOTEM] The network interface [10.2.1.7]
is now up.
Apr 06 14:12:51 [NOTICE ] [TOTEM] Created or loaded sequence id
72084.10.2.1.7 for this ring.
Apr 06 14:12:51 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:12:51 [NOTICE ] [SERV ] Initialising service handler
'openais cluster membership service B.01.01'
Apr 06 14:12:51 [NOTICE ] [SERV ] Initialising service handler
'openais availability management framework B.01.01'
Apr 06 14:12:51 [NOTICE ] [SERV ] Initialising service handler
'openais checkpoint service B.01.01'
Apr 06 14:12:51 [NOTICE ] [SERV ] Initialising service handler
'openais event service B.01.01'
Apr 06 14:12:51 [NOTICE ] [SERV ] Initialising service handler
'openais distributed locking service B.01.01'
Apr 06 14:12:51 [NOTICE ] [SERV ] Initialising service handler
'openais message service B.01.01'
Apr 06 14:12:51 [NOTICE ] [SERV ] Initialising service handler
'openais configuration service'
Apr 06 14:12:51 [NOTICE ] [SERV ] Initialising service handler
'openais cluster closed process group service v1.01'
Apr 06 14:12:51 [NOTICE ] [MAIN ] AIS Executive Service: started and
ready to receive connections.
Apr 06 14:12:51 [NOTICE ] [TOTEM] Creating commit token because I am
the rep.
Apr 06 14:12:51 [NOTICE ] [TOTEM] Saving state aru 0 high seq
received 0
Apr 06 14:12:51 [NOTICE ] [TOTEM] Storing new sequence id for ring
72088
Apr 06 14:12:51 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:12:51 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:12:51 [NOTICE ] [TOTEM] position [0] member 10.2.1.7:
Apr 06 14:12:51 [NOTICE ] [TOTEM] previous ring seq 72084 rep 10.2.1.7
Apr 06 14:12:51 [NOTICE ] [TOTEM] aru 0 high delivered 0 received
flag 0
Apr 06 14:12:51 [NOTICE ] [TOTEM] Did not need to originate any
messages in recovery.
Apr 06 14:12:51 [NOTICE ] [TOTEM] Sending initial ORF token
Apr 06 14:12:51 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:12:51 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:12:51 [NOTICE ] [CLM ] Members Left:
Apr 06 14:12:51 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:12:51 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:12:51 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:12:51 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:12:51 [NOTICE ] [CLM ] Members Left:
Apr 06 14:12:51 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:12:51 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:12:51 [NOTICE ] [SYNC ] This node is within the non-
primary component and will NOT provide any services.
Apr 06 14:12:51 [NOTICE ] [TOTEM] entering OPERATIONAL state.
Apr 06 14:12:51 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:12:51 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:12:51 [NOTICE ] [TOTEM] Saving state aru 11 high seq
received 11
Apr 06 14:12:51 [NOTICE ] [TOTEM] Storing new sequence id for ring
72104
Apr 06 14:12:51 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:12:51 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:12:51 [NOTICE ] [TOTEM] position [0] member 10.2.1.6:
Apr 06 14:12:51 [NOTICE ] [TOTEM] previous ring seq 72100 rep 10.2.1.6
Apr 06 14:12:51 [NOTICE ] [TOTEM] aru 22 high delivered 0 received
flag 0
Apr 06 14:12:51 [NOTICE ] [TOTEM] position [1] member 10.2.1.7:
Apr 06 14:12:51 [NOTICE ] [TOTEM] previous ring seq 72088 rep 10.2.1.7
Apr 06 14:12:51 [NOTICE ] [TOTEM] aru 11 high delivered 0 received
flag 0
Apr 06 14:12:51 [NOTICE ] [TOTEM] position [2] member 10.2.20.254:
Apr 06 14:12:51 [NOTICE ] [TOTEM] previous ring seq 72100 rep 10.2.1.6
Apr 06 14:12:51 [NOTICE ] [TOTEM] aru 32 high delivered 0 received
flag 0
Apr 06 14:12:51 [NOTICE ] [TOTEM] copying all old ring messages from
12-11.
Apr 06 14:12:51 [NOTICE ] [TOTEM] Originated 0 messages in RECOVERY.
Apr 06 14:12:51 [NOTICE ] [TOTEM] Originated for recovery:
Apr 06 14:12:51 [NOTICE ] [TOTEM] Not Originated for recovery:
Apr 06 14:12:51 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:51 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:51 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:51 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:51 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:52 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:53 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:53 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:53 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:53 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:53 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:53 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:53 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:53 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:54 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:54 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:54 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:54 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:54 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:55 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:55 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:55 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:55 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:56 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:56 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:56 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:56 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:56 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:57 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:57 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:57 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:57 [NOTICE ] [TOTEM] Retransmit List: 1 2 3 4 5 6 7 8 9
a b c d e f 10
Apr 06 14:12:57 [ERROR ] [TOTEM] FAILED TO RECEIVE
Apr 06 14:12:59 [NOTICE ] [MAIN ] AIS Executive Service: Copyright
(C) 2002-2006 MontaVista Software, Inc. and contributors.
Apr 06 14:12:59 [WARNING ] [MAIN ] Could not lock memory of service
to avoid page faults
Apr 06 14:12:59 [NOTICE ] [TOTEM] Token Timeout (1000 ms) retransmit
timeout (238 ms)
Apr 06 14:12:59 [NOTICE ] [TOTEM] token hold (180 ms) retransmits
before loss (4 retrans)
Apr 06 14:12:59 [NOTICE ] [TOTEM] join (100 ms) consensus (200 ms)
merge (200 ms)
Apr 06 14:12:59 [NOTICE ] [TOTEM] downcheck (1000 ms) fail to recv
const (50 msgs)
Apr 06 14:12:59 [NOTICE ] [TOTEM] seqno unchanged const (30
rotations) Maximum network MTU 1500
Apr 06 14:12:59 [NOTICE ] [TOTEM] window size per rotation (50
messages) maximum messages per rotation (17 messages)
Apr 06 14:12:59 [NOTICE ] [TOTEM] send threads (0 threads)
Apr 06 14:12:59 [NOTICE ] [TOTEM] heartbeat_failures_allowed (3)
Apr 06 14:12:59 [NOTICE ] [TOTEM] max_network_delay (50 ms)
Apr 06 14:12:59 [NOTICE ] [TOTEM] total heartbeat_timeout (764 ms)
Apr 06 14:12:59 [NOTICE ] [TOTEM] Receive multicast socket recv
buffer size (144000 bytes).
Apr 06 14:12:59 [NOTICE ] [TOTEM] Transmit multicast socket send
buffer size (144000 bytes).
Apr 06 14:12:59 [NOTICE ] [TOTEM] The network interface [10.2.1.7]
is now up.
Apr 06 14:12:59 [NOTICE ] [TOTEM] Created or loaded sequence id
72104.10.2.1.7 for this ring.
Apr 06 14:12:59 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:12:59 [NOTICE ] [SERV ] Initialising service handler
'openais cluster membership service B.01.01'
Apr 06 14:12:59 [NOTICE ] [SERV ] Initialising service handler
'openais availability management framework B.01.01'
Apr 06 14:12:59 [NOTICE ] [SERV ] Initialising service handler
'openais checkpoint service B.01.01'
Apr 06 14:12:59 [NOTICE ] [SERV ] Initialising service handler
'openais event service B.01.01'
Apr 06 14:12:59 [NOTICE ] [SERV ] Initialising service handler
'openais distributed locking service B.01.01'
Apr 06 14:12:59 [NOTICE ] [SERV ] Initialising service handler
'openais message service B.01.01'
Apr 06 14:12:59 [NOTICE ] [SERV ] Initialising service handler
'openais configuration service'
Apr 06 14:12:59 [NOTICE ] [SERV ] Initialising service handler
'openais cluster closed process group service v1.01'
Apr 06 14:12:59 [NOTICE ] [MAIN ] AIS Executive Service: started and
ready to receive connections.
Apr 06 14:12:59 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:12:59 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:00 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:00 [NOTICE ] [TOTEM] Creating commit token because I am
the rep.
Apr 06 14:13:00 [NOTICE ] [TOTEM] Saving state aru 0 high seq
received 0
Apr 06 14:13:00 [NOTICE ] [TOTEM] Storing new sequence id for ring
72120
Apr 06 14:13:00 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:13:00 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:13:00 [NOTICE ] [TOTEM] position [0] member 10.2.1.7:
Apr 06 14:13:00 [NOTICE ] [TOTEM] previous ring seq 72104 rep 10.2.1.7
Apr 06 14:13:00 [NOTICE ] [TOTEM] aru 0 high delivered 0 received
flag 0
Apr 06 14:13:00 [NOTICE ] [TOTEM] position [1] member 10.2.11.5:
Apr 06 14:13:00 [NOTICE ] [TOTEM] previous ring seq 72116 rep 10.2.11.5
Apr 06 14:13:00 [NOTICE ] [TOTEM] aru 2b high delivered 2b received
flag 0
Apr 06 14:13:00 [NOTICE ] [TOTEM] Did not need to originate any
messages in recovery.
Apr 06 14:13:00 [NOTICE ] [TOTEM] Sending initial ORF token
Apr 06 14:13:00 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:00 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:00 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:00 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:00 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:00 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:00 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:00 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:00 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:00 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:00 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:00 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:00 [NOTICE ] [SYNC ] This node is within the non-
primary component and will NOT provide any services.
Apr 06 14:13:00 [NOTICE ] [TOTEM] entering OPERATIONAL state.
Apr 06 14:13:00 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:00 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:00 [NOTICE ] [TOTEM] Creating commit token because I am
the rep.
Apr 06 14:13:00 [NOTICE ] [TOTEM] Saving state aru 46 high seq
received 46
Apr 06 14:13:00 [NOTICE ] [TOTEM] Storing new sequence id for ring
72124
Apr 06 14:13:00 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:13:00 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:13:00 [NOTICE ] [TOTEM] position [0] member 10.2.1.7:
Apr 06 14:13:00 [NOTICE ] [TOTEM] previous ring seq 72120 rep 10.2.1.7
Apr 06 14:13:00 [NOTICE ] [TOTEM] aru 46 high delivered 32 received
flag 0
Apr 06 14:13:00 [NOTICE ] [TOTEM] position [1] member 10.2.11.5:
Apr 06 14:13:00 [NOTICE ] [TOTEM] previous ring seq 72120 rep 10.2.1.7
Apr 06 14:13:00 [NOTICE ] [TOTEM] aru 46 high delivered 32 received
flag 0
Apr 06 14:13:00 [NOTICE ] [TOTEM] copying all old ring messages from
47-46.
Apr 06 14:13:00 [NOTICE ] [TOTEM] Originated 0 messages in RECOVERY.
Apr 06 14:13:00 [NOTICE ] [TOTEM] Originated for recovery:
Apr 06 14:13:00 [NOTICE ] [TOTEM] Not Originated for recovery:
Apr 06 14:13:00 [NOTICE ] [TOTEM] Sending initial ORF token
Apr 06 14:13:00 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:00 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:00 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:00 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:00 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:00 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:00 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:00 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:00 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:00 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:00 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:00 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:00 [NOTICE ] [SYNC ] This node is within the non-
primary component and will NOT provide any services.
Apr 06 14:13:00 [NOTICE ] [TOTEM] entering OPERATIONAL state.
Apr 06 14:13:00 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:01 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:01 [NOTICE ] [TOTEM] Creating commit token because I am
the rep.
Apr 06 14:13:01 [NOTICE ] [TOTEM] Saving state aru 46 high seq
received 46
Apr 06 14:13:01 [NOTICE ] [TOTEM] Storing new sequence id for ring
72128
Apr 06 14:13:01 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:13:01 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:13:01 [NOTICE ] [TOTEM] position [0] member 10.2.1.7:
Apr 06 14:13:01 [NOTICE ] [TOTEM] previous ring seq 72124 rep 10.2.1.7
Apr 06 14:13:01 [NOTICE ] [TOTEM] aru 46 high delivered 11 received
flag 0
Apr 06 14:13:01 [NOTICE ] [TOTEM] position [1] member 10.2.11.5:
Apr 06 14:13:01 [NOTICE ] [TOTEM] previous ring seq 72124 rep 10.2.1.7
Apr 06 14:13:01 [NOTICE ] [TOTEM] aru 46 high delivered 11 received
flag 0
Apr 06 14:13:01 [NOTICE ] [TOTEM] copying all old ring messages from
47-46.
Apr 06 14:13:01 [NOTICE ] [TOTEM] Originated 0 messages in RECOVERY.
Apr 06 14:13:01 [NOTICE ] [TOTEM] Originated for recovery:
Apr 06 14:13:01 [NOTICE ] [TOTEM] Not Originated for recovery:
Apr 06 14:13:01 [NOTICE ] [TOTEM] Sending initial ORF token
Apr 06 14:13:01 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:01 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:01 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:01 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:01 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:01 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:01 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:01 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:01 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:01 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:01 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:01 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:01 [NOTICE ] [SYNC ] This node is within the non-
primary component and will NOT provide any services.
Apr 06 14:13:01 [NOTICE ] [TOTEM] entering OPERATIONAL state.
Apr 06 14:13:01 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:01 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:01 [NOTICE ] [TOTEM] Creating commit token because I am
the rep.
Apr 06 14:13:01 [NOTICE ] [TOTEM] Saving state aru 46 high seq
received 46
Apr 06 14:13:01 [NOTICE ] [TOTEM] Storing new sequence id for ring
72132
Apr 06 14:13:01 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:13:01 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:13:01 [NOTICE ] [TOTEM] position [0] member 10.2.1.7:
Apr 06 14:13:01 [NOTICE ] [TOTEM] previous ring seq 72128 rep 10.2.1.7
Apr 06 14:13:01 [NOTICE ] [TOTEM] aru 46 high delivered 6 received
flag 0
Apr 06 14:13:01 [NOTICE ] [TOTEM] position [1] member 10.2.11.5:
Apr 06 14:13:01 [NOTICE ] [TOTEM] previous ring seq 72128 rep 10.2.1.7
Apr 06 14:13:01 [NOTICE ] [TOTEM] aru 46 high delivered 11 received
flag 0
Apr 06 14:13:01 [NOTICE ] [TOTEM] copying all old ring messages from
47-46.
Apr 06 14:13:01 [NOTICE ] [TOTEM] Originated 0 messages in RECOVERY.
Apr 06 14:13:01 [NOTICE ] [TOTEM] Originated for recovery:
Apr 06 14:13:01 [NOTICE ] [TOTEM] Not Originated for recovery:
Apr 06 14:13:01 [NOTICE ] [TOTEM] Sending initial ORF token
Apr 06 14:13:01 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:01 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:01 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:01 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:01 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:01 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:01 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:01 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:01 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:01 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:01 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:01 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:01 [NOTICE ] [SYNC ] This node is within the non-
primary component and will NOT provide any services.
Apr 06 14:13:01 [NOTICE ] [TOTEM] entering OPERATIONAL state.
Apr 06 14:13:01 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:02 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:02 [NOTICE ] [TOTEM] Creating commit token because I am
the rep.
Apr 06 14:13:02 [NOTICE ] [TOTEM] Saving state aru 46 high seq
received 46
Apr 06 14:13:02 [NOTICE ] [TOTEM] Storing new sequence id for ring
72136
Apr 06 14:13:02 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:13:02 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:13:02 [NOTICE ] [TOTEM] position [0] member 10.2.1.7:
Apr 06 14:13:02 [NOTICE ] [TOTEM] previous ring seq 72132 rep 10.2.1.7
Apr 06 14:13:02 [NOTICE ] [TOTEM] aru 46 high delivered 22 received
flag 0
Apr 06 14:13:02 [NOTICE ] [TOTEM] position [1] member 10.2.11.5:
Apr 06 14:13:02 [NOTICE ] [TOTEM] previous ring seq 72132 rep 10.2.1.7
Apr 06 14:13:02 [NOTICE ] [TOTEM] aru 46 high delivered 32 received
flag 0
Apr 06 14:13:02 [NOTICE ] [TOTEM] copying all old ring messages from
47-46.
Apr 06 14:13:02 [NOTICE ] [TOTEM] Originated 0 messages in RECOVERY.
Apr 06 14:13:02 [NOTICE ] [TOTEM] Originated for recovery:
Apr 06 14:13:02 [NOTICE ] [TOTEM] Not Originated for recovery:
Apr 06 14:13:02 [NOTICE ] [TOTEM] Sending initial ORF token
Apr 06 14:13:02 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:02 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:02 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:02 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:02 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:02 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:02 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:02 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:02 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:02 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:02 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:02 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:02 [NOTICE ] [SYNC ] This node is within the non-
primary component and will NOT provide any services.
Apr 06 14:13:02 [NOTICE ] [TOTEM] entering OPERATIONAL state.
Apr 06 14:13:02 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:02 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:02 [NOTICE ] [TOTEM] Creating commit token because I am
the rep.
Apr 06 14:13:02 [NOTICE ] [TOTEM] Saving state aru 46 high seq
received 46
Apr 06 14:13:02 [NOTICE ] [TOTEM] Storing new sequence id for ring
72140
Apr 06 14:13:02 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:13:02 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:13:02 [NOTICE ] [TOTEM] position [0] member 10.2.1.7:
Apr 06 14:13:02 [NOTICE ] [TOTEM] previous ring seq 72136 rep 10.2.1.7
Apr 06 14:13:02 [NOTICE ] [TOTEM] aru 46 high delivered 22 received
flag 0
Apr 06 14:13:02 [NOTICE ] [TOTEM] position [1] member 10.2.11.5:
Apr 06 14:13:02 [NOTICE ] [TOTEM] previous ring seq 72136 rep 10.2.1.7
Apr 06 14:13:02 [NOTICE ] [TOTEM] aru 46 high delivered 22 received
flag 0
Apr 06 14:13:02 [NOTICE ] [TOTEM] copying all old ring messages from
47-46.
Apr 06 14:13:02 [NOTICE ] [TOTEM] Originated 0 messages in RECOVERY.
Apr 06 14:13:02 [NOTICE ] [TOTEM] Originated for recovery:
Apr 06 14:13:02 [NOTICE ] [TOTEM] Not Originated for recovery:
Apr 06 14:13:02 [NOTICE ] [TOTEM] Sending initial ORF token
Apr 06 14:13:02 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:02 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:02 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:02 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:02 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:02 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:02 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:02 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:02 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:02 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:02 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:02 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:02 [NOTICE ] [SYNC ] This node is within the non-
primary component and will NOT provide any services.
Apr 06 14:13:02 [NOTICE ] [TOTEM] entering OPERATIONAL state.
Apr 06 14:13:02 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:02 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:02 [NOTICE ] [TOTEM] Creating commit token because I am
the rep.
Apr 06 14:13:02 [NOTICE ] [TOTEM] Saving state aru 46 high seq
received 46
Apr 06 14:13:02 [NOTICE ] [TOTEM] Storing new sequence id for ring
72144
Apr 06 14:13:02 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:13:02 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:13:02 [NOTICE ] [TOTEM] position [0] member 10.2.1.7:
Apr 06 14:13:02 [NOTICE ] [TOTEM] previous ring seq 72140 rep 10.2.1.7
Apr 06 14:13:02 [NOTICE ] [TOTEM] aru 46 high delivered 0 received
flag 0
Apr 06 14:13:02 [NOTICE ] [TOTEM] position [1] member 10.2.11.5:
Apr 06 14:13:02 [NOTICE ] [TOTEM] previous ring seq 72140 rep 10.2.1.7
Apr 06 14:13:02 [NOTICE ] [TOTEM] aru 46 high delivered 11 received
flag 0
Apr 06 14:13:02 [NOTICE ] [TOTEM] copying all old ring messages from
47-46.
Apr 06 14:13:02 [NOTICE ] [TOTEM] Originated 0 messages in RECOVERY.
Apr 06 14:13:02 [NOTICE ] [TOTEM] Originated for recovery:
Apr 06 14:13:02 [NOTICE ] [TOTEM] Not Originated for recovery:
Apr 06 14:13:02 [NOTICE ] [TOTEM] Sending initial ORF token
Apr 06 14:13:03 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:03 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:03 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:03 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:03 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:03 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:03 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:03 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:03 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:03 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:03 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:03 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:03 [NOTICE ] [SYNC ] This node is within the non-
primary component and will NOT provide any services.
Apr 06 14:13:03 [NOTICE ] [TOTEM] entering OPERATIONAL state.
Apr 06 14:13:03 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:03 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:03 [NOTICE ] [TOTEM] Creating commit token because I am
the rep.
Apr 06 14:13:03 [NOTICE ] [TOTEM] Saving state aru 46 high seq
received 46
Apr 06 14:13:03 [NOTICE ] [TOTEM] Storing new sequence id for ring
72148
Apr 06 14:13:03 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:13:03 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:13:03 [NOTICE ] [TOTEM] position [0] member 10.2.1.7:
Apr 06 14:13:03 [NOTICE ] [TOTEM] previous ring seq 72144 rep 10.2.1.7
Apr 06 14:13:03 [NOTICE ] [TOTEM] aru 46 high delivered 0 received
flag 0
Apr 06 14:13:03 [NOTICE ] [TOTEM] position [1] member 10.2.11.5:
Apr 06 14:13:03 [NOTICE ] [TOTEM] previous ring seq 72144 rep 10.2.1.7
Apr 06 14:13:03 [NOTICE ] [TOTEM] aru 46 high delivered 11 received
flag 0
Apr 06 14:13:03 [NOTICE ] [TOTEM] copying all old ring messages from
47-46.
Apr 06 14:13:03 [NOTICE ] [TOTEM] Originated 0 messages in RECOVERY.
Apr 06 14:13:03 [NOTICE ] [TOTEM] Originated for recovery:
Apr 06 14:13:03 [NOTICE ] [TOTEM] Not Originated for recovery:
Apr 06 14:13:03 [NOTICE ] [TOTEM] Sending initial ORF token
Apr 06 14:13:03 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:03 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:03 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:03 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:03 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:03 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:03 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:03 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:03 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:03 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:03 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:03 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:03 [NOTICE ] [SYNC ] This node is within the non-
primary component and will NOT provide any services.
Apr 06 14:13:03 [NOTICE ] [TOTEM] entering OPERATIONAL state.
Apr 06 14:13:03 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:03 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:03 [NOTICE ] [TOTEM] Creating commit token because I am
the rep.
Apr 06 14:13:03 [NOTICE ] [TOTEM] Saving state aru 46 high seq
received 46
Apr 06 14:13:03 [NOTICE ] [TOTEM] Storing new sequence id for ring
72152
Apr 06 14:13:03 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:13:03 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:13:03 [NOTICE ] [TOTEM] position [0] member 10.2.1.7:
Apr 06 14:13:03 [NOTICE ] [TOTEM] previous ring seq 72148 rep 10.2.1.7
Apr 06 14:13:03 [NOTICE ] [TOTEM] aru 46 high delivered 22 received
flag 0
Apr 06 14:13:03 [NOTICE ] [TOTEM] position [1] member 10.2.11.5:
Apr 06 14:13:03 [NOTICE ] [TOTEM] previous ring seq 72148 rep 10.2.1.7
Apr 06 14:13:03 [NOTICE ] [TOTEM] aru 46 high delivered 32 received
flag 0
Apr 06 14:13:03 [NOTICE ] [TOTEM] copying all old ring messages from
47-46.
Apr 06 14:13:03 [NOTICE ] [TOTEM] Originated 0 messages in RECOVERY.
Apr 06 14:13:03 [NOTICE ] [TOTEM] Originated for recovery:
Apr 06 14:13:03 [NOTICE ] [TOTEM] Not Originated for recovery:
Apr 06 14:13:03 [NOTICE ] [TOTEM] Sending initial ORF token
Apr 06 14:13:03 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:03 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:03 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:03 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:03 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:03 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:03 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:03 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:03 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:03 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:03 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:03 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:03 [NOTICE ] [SYNC ] This node is within the non-
primary component and will NOT provide any services.
Apr 06 14:13:04 [NOTICE ] [TOTEM] entering OPERATIONAL state.
Apr 06 14:13:04 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:04 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:04 [NOTICE ] [TOTEM] Creating commit token because I am
the rep.
Apr 06 14:13:04 [NOTICE ] [TOTEM] Saving state aru 46 high seq
received 46
Apr 06 14:13:04 [NOTICE ] [TOTEM] Storing new sequence id for ring
72156
Apr 06 14:13:04 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:13:04 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:13:04 [NOTICE ] [TOTEM] position [0] member 10.2.1.7:
Apr 06 14:13:04 [NOTICE ] [TOTEM] previous ring seq 72152 rep 10.2.1.7
Apr 06 14:13:04 [NOTICE ] [TOTEM] aru 46 high delivered 22 received
flag 0
Apr 06 14:13:04 [NOTICE ] [TOTEM] position [1] member 10.2.11.5:
Apr 06 14:13:04 [NOTICE ] [TOTEM] previous ring seq 72152 rep 10.2.1.7
Apr 06 14:13:04 [NOTICE ] [TOTEM] aru 46 high delivered 22 received
flag 0
Apr 06 14:13:04 [NOTICE ] [TOTEM] copying all old ring messages from
47-46.
Apr 06 14:13:04 [NOTICE ] [TOTEM] Originated 0 messages in RECOVERY.
Apr 06 14:13:04 [NOTICE ] [TOTEM] Originated for recovery:
Apr 06 14:13:04 [NOTICE ] [TOTEM] Not Originated for recovery:
Apr 06 14:13:04 [NOTICE ] [TOTEM] Sending initial ORF token
Apr 06 14:13:04 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:04 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:04 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:04 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:04 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:04 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:04 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:04 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:04 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:04 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:04 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:04 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:04 [NOTICE ] [SYNC ] This node is within the non-
primary component and will NOT provide any services.
Apr 06 14:13:04 [NOTICE ] [TOTEM] entering OPERATIONAL state.
Apr 06 14:13:04 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:04 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:04 [NOTICE ] [TOTEM] Creating commit token because I am
the rep.
Apr 06 14:13:04 [NOTICE ] [TOTEM] Saving state aru 46 high seq
received 46
Apr 06 14:13:04 [NOTICE ] [TOTEM] Storing new sequence id for ring
72160
Apr 06 14:13:04 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:13:04 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:13:04 [NOTICE ] [TOTEM] position [0] member 10.2.1.7:
Apr 06 14:13:04 [NOTICE ] [TOTEM] previous ring seq 72156 rep 10.2.1.7
Apr 06 14:13:04 [NOTICE ] [TOTEM] aru 46 high delivered 11 received
flag 0
Apr 06 14:13:04 [NOTICE ] [TOTEM] position [1] member 10.2.11.5:
Apr 06 14:13:04 [NOTICE ] [TOTEM] previous ring seq 72156 rep 10.2.1.7
Apr 06 14:13:04 [NOTICE ] [TOTEM] aru 46 high delivered 11 received
flag 0
Apr 06 14:13:04 [NOTICE ] [TOTEM] copying all old ring messages from
47-46.
Apr 06 14:13:04 [NOTICE ] [TOTEM] Originated 0 messages in RECOVERY.
Apr 06 14:13:04 [NOTICE ] [TOTEM] Originated for recovery:
Apr 06 14:13:04 [NOTICE ] [TOTEM] Not Originated for recovery:
Apr 06 14:13:04 [NOTICE ] [TOTEM] Sending initial ORF token
Apr 06 14:13:04 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:04 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:04 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:04 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:04 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:04 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:04 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:04 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:04 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:04 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:04 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:04 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:04 [NOTICE ] [SYNC ] This node is within the non-
primary component and will NOT provide any services.
Apr 06 14:13:04 [NOTICE ] [TOTEM] entering OPERATIONAL state.
Apr 06 14:13:05 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:05 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:05 [NOTICE ] [TOTEM] Creating commit token because I am
the rep.
Apr 06 14:13:05 [NOTICE ] [TOTEM] Saving state aru 46 high seq
received 46
Apr 06 14:13:05 [NOTICE ] [TOTEM] Storing new sequence id for ring
72164
Apr 06 14:13:05 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:13:05 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:13:05 [NOTICE ] [TOTEM] position [0] member 10.2.1.7:
Apr 06 14:13:05 [NOTICE ] [TOTEM] previous ring seq 72160 rep 10.2.1.7
Apr 06 14:13:05 [NOTICE ] [TOTEM] aru 46 high delivered 22 received
flag 0
Apr 06 14:13:05 [NOTICE ] [TOTEM] position [1] member 10.2.11.5:
Apr 06 14:13:05 [NOTICE ] [TOTEM] previous ring seq 72160 rep 10.2.1.7
Apr 06 14:13:05 [NOTICE ] [TOTEM] aru 46 high delivered 22 received
flag 0
Apr 06 14:13:05 [NOTICE ] [TOTEM] copying all old ring messages from
47-46.
Apr 06 14:13:05 [NOTICE ] [TOTEM] Originated 0 messages in RECOVERY.
Apr 06 14:13:05 [NOTICE ] [TOTEM] Originated for recovery:
Apr 06 14:13:05 [NOTICE ] [TOTEM] Not Originated for recovery:
Apr 06 14:13:05 [NOTICE ] [TOTEM] Sending initial ORF token
Apr 06 14:13:05 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:05 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:05 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:05 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:05 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:05 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:05 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:05 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:05 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:05 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:05 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:05 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:05 [NOTICE ] [SYNC ] This node is within the non-
primary component and will NOT provide any services.
Apr 06 14:13:05 [NOTICE ] [TOTEM] entering OPERATIONAL state.
Apr 06 14:13:05 [NOTICE ] [YKD ] This processor is within the
primary component.
Apr 06 14:13:05 [NOTICE ] [SYNC ] This node is within the primary
component and will provide service.
Apr 06 14:13:05 [NOTICE ] [SYNC ] Synchronization barrier completed
Apr 06 14:13:05 [NOTICE ] [SYNC ] Synchronization actions starting
for (openais cluster membership service B.01.01)
Apr 06 14:13:05 [NOTICE ] [SYNC ] Synchronization actions done for
(openais cluster membership service B.01.01)
Apr 06 14:13:05 [NOTICE ] [CLM ] got nodejoin message 10.2.1.7
Apr 06 14:13:05 [NOTICE ] [CLM ] got nodejoin message 10.2.11.5
Apr 06 14:13:05 [NOTICE ] [SYNC ] Synchronization barrier completed
Apr 06 14:13:05 [NOTICE ] [SYNC ] Synchronization actions starting
for (openais checkpoint service B.01.01)
Apr 06 14:13:05 [NOTICE ] [SYNC ] Synchronization actions done for
(openais checkpoint service B.01.01)
Apr 06 14:13:05 [NOTICE ] [SYNC ] Synchronization barrier completed
Apr 06 14:13:05 [NOTICE ] [SYNC ] Synchronization actions starting
for (openais event service B.01.01)
Apr 06 14:13:05 [NOTICE ] [SYNC ] Synchronization actions done for
(openais event service B.01.01)
Apr 06 14:13:06 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:06 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:06 [NOTICE ] [TOTEM] Creating commit token because I am
the rep.
Apr 06 14:13:06 [NOTICE ] [TOTEM] Saving state aru 55 high seq
received 55
Apr 06 14:13:06 [NOTICE ] [TOTEM] Storing new sequence id for ring
72168
Apr 06 14:13:06 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:13:06 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:13:06 [NOTICE ] [TOTEM] position [0] member 10.2.1.7:
Apr 06 14:13:06 [NOTICE ] [TOTEM] previous ring seq 72164 rep 10.2.1.7
Apr 06 14:13:06 [NOTICE ] [TOTEM] aru 55 high delivered 55 received
flag 0
Apr 06 14:13:06 [NOTICE ] [TOTEM] position [1] member 10.2.11.5:
Apr 06 14:13:06 [NOTICE ] [TOTEM] previous ring seq 72164 rep 10.2.1.7
Apr 06 14:13:06 [NOTICE ] [TOTEM] aru 55 high delivered 55 received
flag 0
Apr 06 14:13:06 [NOTICE ] [TOTEM] copying all old ring messages from
56-55.
Apr 06 14:13:06 [NOTICE ] [TOTEM] Originated 0 messages in RECOVERY.
Apr 06 14:13:06 [NOTICE ] [TOTEM] Originated for recovery:
Apr 06 14:13:06 [NOTICE ] [TOTEM] Not Originated for recovery:
Apr 06 14:13:06 [NOTICE ] [TOTEM] Sending initial ORF token
Apr 06 14:13:06 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:06 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:06 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:06 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:06 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:06 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:06 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:06 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:06 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:06 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:06 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:06 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:06 [NOTICE ] [SYNC ] This node is within the non-
primary component and will NOT provide any services.
Apr 06 14:13:06 [NOTICE ] [TOTEM] entering OPERATIONAL state.
Apr 06 14:13:06 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:06 [NOTICE ] [TOTEM] Saving state aru 43 high seq
received 43
Apr 06 14:13:06 [NOTICE ] [TOTEM] Storing new sequence id for ring
72172
Apr 06 14:13:06 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:13:06 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:13:06 [NOTICE ] [TOTEM] position [0] member 10.2.1.6:
Apr 06 14:13:06 [NOTICE ] [TOTEM] previous ring seq 72168 rep 10.2.1.6
Apr 06 14:13:06 [NOTICE ] [TOTEM] aru 6e high delivered 6e received
flag 0
Apr 06 14:13:06 [NOTICE ] [TOTEM] position [1] member 10.2.1.7:
Apr 06 14:13:06 [NOTICE ] [TOTEM] previous ring seq 72168 rep 10.2.1.7
Apr 06 14:13:06 [NOTICE ] [TOTEM] aru 43 high delivered 3 received
flag 0
Apr 06 14:13:06 [NOTICE ] [TOTEM] position [2] member 10.2.11.5:
Apr 06 14:13:06 [NOTICE ] [TOTEM] previous ring seq 72168 rep 10.2.1.7
Apr 06 14:13:06 [NOTICE ] [TOTEM] aru 45 high delivered 11 received
flag 0
Apr 06 14:13:06 [NOTICE ] [TOTEM] copying all old ring messages from
44-43.
Apr 06 14:13:06 [NOTICE ] [TOTEM] Originated 0 messages in RECOVERY.
Apr 06 14:13:06 [NOTICE ] [TOTEM] Originated for recovery:
Apr 06 14:13:06 [NOTICE ] [TOTEM] Not Originated for recovery:
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:06 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:07 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:07 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:07 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:07 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:07 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:08 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:08 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:08 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:08 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:08 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:08 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:09 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:09 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:09 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:09 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:09 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:10 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:10 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:10 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:10 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:10 [NOTICE ] [TOTEM] Retransmit List: 1
Apr 06 14:13:10 [ERROR ] [TOTEM] FAILED TO RECEIVE
Apr 06 14:13:12 [NOTICE ] [MAIN ] AIS Executive Service: Copyright
(C) 2002-2006 MontaVista Software, Inc. and contributors.
Apr 06 14:13:12 [WARNING ] [MAIN ] Could not lock memory of service
to avoid page faults
Apr 06 14:13:12 [NOTICE ] [TOTEM] Token Timeout (1000 ms) retransmit
timeout (238 ms)
Apr 06 14:13:12 [NOTICE ] [TOTEM] token hold (180 ms) retransmits
before loss (4 retrans)
Apr 06 14:13:12 [NOTICE ] [TOTEM] join (100 ms) consensus (200 ms)
merge (200 ms)
Apr 06 14:13:12 [NOTICE ] [TOTEM] downcheck (1000 ms) fail to recv
const (50 msgs)
Apr 06 14:13:12 [NOTICE ] [TOTEM] seqno unchanged const (30
rotations) Maximum network MTU 1500
Apr 06 14:13:12 [NOTICE ] [TOTEM] window size per rotation (50
messages) maximum messages per rotation (17 messages)
Apr 06 14:13:12 [NOTICE ] [TOTEM] send threads (0 threads)
Apr 06 14:13:12 [NOTICE ] [TOTEM] heartbeat_failures_allowed (3)
Apr 06 14:13:12 [NOTICE ] [TOTEM] max_network_delay (50 ms)
Apr 06 14:13:12 [NOTICE ] [TOTEM] total heartbeat_timeout (764 ms)
Apr 06 14:13:12 [NOTICE ] [TOTEM] Receive multicast socket recv
buffer size (144000 bytes).
Apr 06 14:13:12 [NOTICE ] [TOTEM] Transmit multicast socket send
buffer size (144000 bytes).
Apr 06 14:13:12 [NOTICE ] [TOTEM] The network interface [10.2.1.7]
is now up.
Apr 06 14:13:12 [NOTICE ] [TOTEM] Created or loaded sequence id
72172.10.2.1.7 for this ring.
Apr 06 14:13:12 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:12 [NOTICE ] [SERV ] Initialising service handler
'openais cluster membership service B.01.01'
Apr 06 14:13:12 [NOTICE ] [SERV ] Initialising service handler
'openais availability management framework B.01.01'
Apr 06 14:13:12 [NOTICE ] [SERV ] Initialising service handler
'openais checkpoint service B.01.01'
Apr 06 14:13:12 [NOTICE ] [SERV ] Initialising service handler
'openais event service B.01.01'
Apr 06 14:13:12 [NOTICE ] [SERV ] Initialising service handler
'openais distributed locking service B.01.01'
Apr 06 14:13:12 [NOTICE ] [SERV ] Initialising service handler
'openais message service B.01.01'
Apr 06 14:13:12 [NOTICE ] [SERV ] Initialising service handler
'openais configuration service'
Apr 06 14:13:12 [NOTICE ] [SERV ] Initialising service handler
'openais cluster closed process group service v1.01'
Apr 06 14:13:12 [NOTICE ] [MAIN ] AIS Executive Service: started and
ready to receive connections.
Apr 06 14:13:12 [NOTICE ] [TOTEM] Creating commit token because I am
the rep.
Apr 06 14:13:12 [NOTICE ] [TOTEM] Saving state aru 0 high seq
received 0
Apr 06 14:13:12 [NOTICE ] [TOTEM] Storing new sequence id for ring
72176
Apr 06 14:13:12 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:13:12 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:13:12 [NOTICE ] [TOTEM] position [0] member 10.2.1.7:
Apr 06 14:13:12 [NOTICE ] [TOTEM] previous ring seq 72172 rep 10.2.1.7
Apr 06 14:13:12 [NOTICE ] [TOTEM] aru 0 high delivered 0 received
flag 0
Apr 06 14:13:12 [NOTICE ] [TOTEM] Did not need to originate any
messages in recovery.
Apr 06 14:13:12 [NOTICE ] [TOTEM] Sending initial ORF token
Apr 06 14:13:12 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:12 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:12 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:12 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:12 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:12 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:12 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:12 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:12 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:12 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:12 [NOTICE ] [SYNC ] This node is within the non-
primary component and will NOT provide any services.
Apr 06 14:13:13 [NOTICE ] [TOTEM] entering OPERATIONAL state.
Apr 06 14:13:13 [NOTICE ] [YKD ] This processor is within the
primary component.
Apr 06 14:13:13 [NOTICE ] [SYNC ] This node is within the primary
component and will provide service.
Apr 06 14:13:13 [NOTICE ] [SYNC ] Synchronization barrier completed
Apr 06 14:13:13 [NOTICE ] [SYNC ] Synchronization actions starting
for (openais cluster membership service B.01.01)
Apr 06 14:13:13 [NOTICE ] [SYNC ] Synchronization actions done for
(openais cluster membership service B.01.01)
Apr 06 14:13:13 [NOTICE ] [CLM ] got nodejoin message 10.2.1.7
Apr 06 14:13:13 [NOTICE ] [SYNC ] Synchronization barrier completed
Apr 06 14:13:13 [NOTICE ] [SYNC ] Synchronization actions starting
for (openais checkpoint service B.01.01)
Apr 06 14:13:13 [NOTICE ] [SYNC ] Synchronization actions done for
(openais checkpoint service B.01.01)
Apr 06 14:13:13 [NOTICE ] [TOTEM] entering GATHER state.
Apr 06 14:13:13 [NOTICE ] [TOTEM] Saving state aru 28 high seq
received 28
Apr 06 14:13:13 [NOTICE ] [TOTEM] Storing new sequence id for ring
72184
Apr 06 14:13:13 [NOTICE ] [TOTEM] entering COMMIT state.
Apr 06 14:13:13 [NOTICE ] [TOTEM] entering RECOVERY state.
Apr 06 14:13:13 [NOTICE ] [TOTEM] position [0] member 10.2.1.6:
Apr 06 14:13:13 [NOTICE ] [TOTEM] previous ring seq 72180 rep 10.2.1.6
Apr 06 14:13:13 [NOTICE ] [TOTEM] aru e8 high delivered e8 received
flag 0
Apr 06 14:13:13 [NOTICE ] [TOTEM] position [1] member 10.2.1.7:
Apr 06 14:13:13 [NOTICE ] [TOTEM] previous ring seq 72176 rep 10.2.1.7
Apr 06 14:13:13 [NOTICE ] [TOTEM] aru 28 high delivered 27 received
flag 0
Apr 06 14:13:13 [NOTICE ] [TOTEM] position [2] member 10.2.11.5:
Apr 06 14:13:13 [NOTICE ] [TOTEM] previous ring seq 72180 rep 10.2.1.6
Apr 06 14:13:13 [NOTICE ] [TOTEM] aru e8 high delivered e8 received
flag 0
Apr 06 14:13:13 [NOTICE ] [TOTEM] position [3] member 10.2.20.254:
Apr 06 14:13:13 [NOTICE ] [TOTEM] previous ring seq 72180 rep 10.2.1.6
Apr 06 14:13:13 [NOTICE ] [TOTEM] aru e8 high delivered e8 received
flag 0
Apr 06 14:13:13 [NOTICE ] [TOTEM] copying all old ring messages from
29-28.
Apr 06 14:13:13 [NOTICE ] [TOTEM] Originated 0 messages in RECOVERY.
Apr 06 14:13:13 [NOTICE ] [TOTEM] Originated for recovery:
Apr 06 14:13:13 [NOTICE ] [TOTEM] Not Originated for recovery:
Apr 06 14:13:13 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:13 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:13 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:13 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:13 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:13 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
Apr 06 14:13:13 [NOTICE ] [CLM ] New Configuration:
Apr 06 14:13:13 [NOTICE ] [CLM ] 10.2.1.6
Apr 06 14:13:13 [NOTICE ] [CLM ] 10.2.1.7
Apr 06 14:13:13 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:13 [NOTICE ] [CLM ] 10.2.20.254
Apr 06 14:13:13 [NOTICE ] [CLM ] Members Left:
Apr 06 14:13:13 [NOTICE ] [CLM ] Members Joined:
Apr 06 14:13:13 [NOTICE ] [CLM ] 10.2.1.6
Apr 06 14:13:13 [NOTICE ] [CLM ] 10.2.11.5
Apr 06 14:13:13 [NOTICE ] [CLM ] 10.2.20.254
Apr 06 14:13:13 [NOTICE ] [SYNC ] This node is within the non-
primary component and will NOT provide any services.
Apr 06 14:13:13 [NOTICE ] [TOTEM] entering OPERATIONAL state.
Apr 06 14:13:13 [NOTICE ] [YKD ] This processor is within the
primary component.
Apr 06 14:13:13 [NOTICE ] [SYNC ] This node is within the primary
component and will provide service.
Apr 06 14:13:13 [NOTICE ] [SYNC ] Synchronization barrier completed
Apr 06 14:13:13 [NOTICE ] [SYNC ] Synchronization actions starting
for (openais cluster membership service B.01.01)
Apr 06 14:13:13 [NOTICE ] [SYNC ] Synchronization actions done for
(openais cluster membership service B.01.01)
Apr 06 14:13:13 [NOTICE ] [CLM ] got nodejoin message 10.2.11.5
Apr 06 14:13:13 [NOTICE ] [CLM ] got nodejoin message 10.2.20.254
Apr 06 14:13:13 [NOTICE ] [CLM ] got nodejoin message 10.2.1.6
Apr 06 14:13:13 [NOTICE ] [CLM ] got nodejoin message 10.2.1.7
Apr 06 14:13:13 [NOTICE ] [SYNC ] Synchronization barrier completed
Apr 06 14:13:13 [NOTICE ] [SYNC ] Synchronization actions starting
for (openais checkpoint service B.01.01)
Apr 06 14:13:13 [NOTICE ] [SYNC ] Synchronization actions done for
(openais checkpoint service B.01.01)
Apr 06 14:13:14 [NOTICE ] [SYNC ] Synchronization barrier completed
Apr 06 14:13:14 [NOTICE ] [SYNC ] Synchronization actions starting
for (openais event service B.01.01)
Apr 06 14:13:14 [NOTICE ] [SYNC ] Synchronization actions done for
(openais event service B.01.01)
More information about the Openais
mailing list