[Openais] Trunk broken?

Fabien THOMAS fabien.thomas at netasq.com
Tue Apr 25 09:10:02 PDT 2006


>
> It appears one of your nodes is dropping the token.  I have not  
> made any
> changes to the token sending routines that I can recall.  Are you  
> still
> using the "netmtu" command to reduce the size of the network mtu?
>
yes

totem {
     version: 2
     secauth: on
     threads: 0
     heartbeat_failures_allowed: 3
     max_network_delay: 50
     interface {
         ringnumber: 0
         bindnetaddr: 10.2.0.0
         mcastaddr: 226.94.1.10
         mcastport: 5406
         netmtu: 1400
     }
}


> Try an svn update to get all the latest bits.  If that doesn't  
> work, if
> you can identify the revision that breaks the code for you, that would
> really help.
>
At revision 1008.

ok i will do that.

> (It works here with 4 nodes).
>
> Regards
> -steve
>
>
> On Tue, 2006-04-25 at 15:27 +0200, Fabien THOMAS wrote:
>> i've updated to latest trunk and cannot get it to works with two
>> nodes it loops :
>>
>> Apr 25 13:15:05.975526 [MAIN ] Could not lock memory of service to
>> avoid page faults
>> Apr 25 13:15:05.976208 [TOTEM] Token Timeout (1000 ms) retransmit
>> timeout (238 ms)
>> Apr 25 13:15:05.976322 [TOTEM] token hold (180 ms) retransmits before
>> loss (4 retrans)
>> Apr 25 13:15:05.976558 [TOTEM] join (100 ms) consensus (200 ms) merge
>> (200 ms)
>> Apr 25 13:15:05.976632 [TOTEM] downcheck (1000 ms) fail to recv const
>> (50 msgs)
>> Apr 25 13:15:05.976704 [TOTEM] seqno unchanged const (30 rotations)
>> Maximum network MTU 1500
>> Apr 25 13:15:05.976784 [TOTEM] window size per rotation (50 messages)
>> maximum messages per rotation (17 messages)
>> Apr 25 13:15:05.977016 [TOTEM] send threads (0 threads)
>> Apr 25 13:15:05.977084 [TOTEM] heartbeat_failures_allowed (0)
>> Apr 25 13:15:05.977151 [TOTEM] max_network_delay (50 ms)
>> Apr 25 13:15:05.977831 [TOTEM] HeartBeat is Disabled. To enable set
>> heartbeat_failures_allowed > 0
>> Apr 25 13:15:05.981252 [TOTEM] Receive multicast socket recv buffer
>> size (144000 bytes).
>> Apr 25 13:15:05.981616 [TOTEM] Transmit multicast socket send buffer
>> size (144000 bytes).
>> Apr 25 13:15:05.982116 [TOTEM] The network interface [10.2.1.7] is
>> now up.
>> Apr 25 13:15:05.982346 [TOTEM] Created or loaded sequence id
>> 63728.10.2.1.7 for this ring.
>> Apr 25 13:15:05.983214 [TOTEM] entering GATHER state.
>> Apr 25 13:15:05.983694 [SERV ] Initialising service handler 'openais
>> cluster membership service B.01.01'
>> Apr 25 13:15:05.984103 [SERV ] Initialising service handler 'openais
>> availability management framework B.01.01'
>> Apr 25 13:15:05.984221 [SERV ] Initialising service handler 'openais
>> checkpoint service B.01.01'
>> Apr 25 13:15:05.984308 [SERV ] Initialising service handler 'openais
>> event service B.01.01'
>> Apr 25 13:15:05.984578 [SERV ] Initialising service handler 'openais
>> distributed locking service B.01.01'
>> Apr 25 13:15:05.984672 [SERV ] Initialising service handler 'openais
>> message service B.01.01'
>> Apr 25 13:15:05.984757 [SERV ] Initialising service handler 'openais
>> configuration service'
>> Apr 25 13:15:05.984841 [SERV ] Initialising service handler 'openais
>> cluster closed process group service v1.01'
>> Apr 25 13:15:05.987150 [MAIN ] AIS Executive Service: started and
>> ready to provide service.
>> Apr 25 13:15:05.988130 [TOTEM] Creating commit token because I am the
>> rep.
>> Apr 25 13:15:05.988277 [TOTEM] Saving state aru 0 high seq received 0
>> Apr 25 13:15:05.989269 [TOTEM] Storing new sequence id for ring 63732
>> Apr 25 13:15:05.989792 [TOTEM] entering COMMIT state.
>> Apr 25 13:15:05.991214 [TOTEM] entering RECOVERY state.
>> Apr 25 13:15:05.992591 [TOTEM] position [0] member 10.2.1.7:
>> Apr 25 13:15:05.992747 [TOTEM] previous ring seq 63728 rep 10.2.1.7
>> Apr 25 13:15:05.992821 [TOTEM] aru 0 high delivered 0 received flag 0
>> Apr 25 13:15:05.993051 [TOTEM] Did not need to originate any messages
>> in recovery.
>> Apr 25 13:15:05.993661 [TOTEM] Sending initial ORF token
>> Apr 25 13:15:05.999589 [CLM  ] CLM CONFIGURATION CHANGE
>> Apr 25 13:15:05.999746 [CLM  ] New Configuration:
>> Apr 25 13:15:05.999813 [CLM  ] Members Left:
>> Apr 25 13:15:06.000289 [CLM  ] Members Joined:
>> Apr 25 13:15:06.001124 [CLM  ] CLM CONFIGURATION CHANGE
>> Apr 25 13:15:06.001230 [CLM  ] New Configuration:
>> Apr 25 13:15:06.001304 [CLM  ] 	10.2.1.7
>> Apr 25 13:15:06.001525 [CLM  ] Members Left:
>> Apr 25 13:15:06.001591 [CLM  ] Members Joined:
>> Apr 25 13:15:06.001663 [CLM  ] 	10.2.1.7
>> Apr 25 13:15:06.001776 [SYNC ] This node is within the non-primary
>> component and will NOT provide any services.
>> Apr 25 13:15:06.064051 [TOTEM] entering OPERATIONAL state.
>> Apr 25 13:15:06.075652 [YKD  ] This processor is within the primary
>> component.
>> Apr 25 13:15:06.076080 [SYNC ] This node is within the primary
>> component and will provide service.
>> Apr 25 13:15:06.077375 [SYNC ] Synchronization barrier completed
>> Apr 25 13:15:06.077681 [SYNC ] Synchronization actions starting for
>> (openais cluster membership service B.01.01)
>> Apr 25 13:15:06.077831 [SYNC ] Synchronization actions done for
>> (openais cluster membership service B.01.01)
>> Apr 25 13:15:06.079122 [CLM  ] got nodejoin message 10.2.1.7
>> Apr 25 13:15:06.080331 [SYNC ] Synchronization barrier completed
>> Apr 25 13:15:06.080675 [SYNC ] Synchronization actions starting for
>> (openais checkpoint service B.01.01)
>> Apr 25 13:15:06.080805 [SYNC ] Synchronization actions done for
>> (openais checkpoint service B.01.01)
>> Apr 25 13:15:06.082876 [SYNC ] Synchronization barrier completed
>> Apr 25 13:15:06.083162 [SYNC ] Synchronization actions starting for
>> (openais event service B.01.01)
>> Apr 25 13:15:06.087210 [SYNC ] Synchronization actions done for
>> (openais event service B.01.01)
>> Apr 25 13:15:06.088651 [TOTEM] entering GATHER state.
>> Apr 25 13:15:06.092611 [TOTEM] Saving state aru 2a high seq  
>> received 2a
>> Apr 25 13:15:06.093620 [TOTEM] Storing new sequence id for ring 63736
>> Apr 25 13:15:06.093816 [TOTEM] entering COMMIT state.
>> Apr 25 13:15:06.097226 [TOTEM] entering RECOVERY state.
>> Apr 25 13:15:06.098654 [TOTEM] position [0] member 10.2.1.6:
>> Apr 25 13:15:06.098812 [TOTEM] previous ring seq 63728 rep 10.2.1.6
>> Apr 25 13:15:06.098886 [TOTEM] aru 2a high delivered 2a received  
>> flag 0
>> Apr 25 13:15:06.099119 [TOTEM] position [1] member 10.2.1.7:
>> Apr 25 13:15:06.099197 [TOTEM] previous ring seq 63732 rep 10.2.1.7
>> Apr 25 13:15:06.099268 [TOTEM] aru 2a high delivered 2a received  
>> flag 0
>> Apr 25 13:15:06.099347 [TOTEM] copying all old ring messages from  
>> 2b-2a.
>> Apr 25 13:15:06.099424 [TOTEM] Originated 0 messages in RECOVERY.
>> Apr 25 13:15:06.099667 [TOTEM] Originated for recovery:
>> Apr 25 13:15:06.099736 [TOTEM] Not Originated for recovery:
>> Apr 25 13:15:06.117182 [CLM  ] CLM CONFIGURATION CHANGE
>> Apr 25 13:15:06.117377 [CLM  ] New Configuration:
>> Apr 25 13:15:06.117621 [CLM  ] 	10.2.1.7
>> Apr 25 13:15:06.117691 [CLM  ] Members Left:
>> Apr 25 13:15:06.117756 [CLM  ] Members Joined:
>> Apr 25 13:15:06.117938 [CLM  ] CLM CONFIGURATION CHANGE
>> Apr 25 13:15:06.118169 [CLM  ] New Configuration:
>> Apr 25 13:15:06.118243 [CLM  ] 	10.2.1.6
>> Apr 25 13:15:06.118315 [CLM  ] 	10.2.1.7
>> Apr 25 13:15:06.118381 [CLM  ] Members Left:
>> Apr 25 13:15:06.118596 [CLM  ] Members Joined:
>> Apr 25 13:15:06.118671 [CLM  ] 	10.2.1.6
>> Apr 25 13:15:06.118763 [SYNC ] This node is within the non-primary
>> component and will NOT provide any services.
>> Apr 25 13:15:06.170331 [TOTEM] entering OPERATIONAL state.
>> Apr 25 13:15:07.172328 [TOTEM] The token was lost in state 1 from
>> timer 83ce000
>> Apr 25 13:15:07.174080 [TOTEM] Receive multicast socket recv buffer
>> size (144000 bytes).
>> Apr 25 13:15:07.174418 [TOTEM] Transmit multicast socket send buffer
>> size (144000 bytes).
>> Apr 25 13:15:07.175429 [TOTEM] entering GATHER state.
>> Apr 25 13:15:07.382250 [TOTEM] entering GATHER state.
>> Apr 25 13:15:07.406583 [TOTEM] Creating commit token because I am the
>> rep.
>> Apr 25 13:15:07.406788 [TOTEM] Saving state aru 10 high seq  
>> received 10
>> Apr 25 13:15:07.407777 [TOTEM] Storing new sequence id for ring 63744
>> Apr 25 13:15:07.408128 [TOTEM] entering COMMIT state.
>> Apr 25 13:15:07.408720 [TOTEM] entering RECOVERY state.
>> Apr 25 13:15:07.409974 [TOTEM] position [0] member 10.2.1.7:
>> Apr 25 13:15:07.410128 [TOTEM] previous ring seq 63736 rep 10.2.1.6
>> Apr 25 13:15:07.410203 [TOTEM] aru 10 high delivered 0 received  
>> flag 0
>> Apr 25 13:15:07.410285 [TOTEM] copying all old ring messages from  
>> 11-10.
>> Apr 25 13:15:07.410520 [TOTEM] Originated 0 messages in RECOVERY.
>> Apr 25 13:15:07.410593 [TOTEM] Originated for recovery:
>> Apr 25 13:15:07.410659 [TOTEM] Not Originated for recovery:
>> Apr 25 13:15:07.411246 [TOTEM] Sending initial ORF token
>> Apr 25 13:15:07.418268 [CLM  ] CLM CONFIGURATION CHANGE
>> Apr 25 13:15:07.418654 [CLM  ] New Configuration:
>> Apr 25 13:15:07.418734 [CLM  ] 	10.2.1.7
>> Apr 25 13:15:07.418804 [CLM  ] Members Left:
>> Apr 25 13:15:07.419025 [CLM  ] 	10.2.1.6
>> Apr 25 13:15:07.419092 [CLM  ] Members Joined:
>> Apr 25 13:15:07.419181 [CKPT ] clean_checkpoint_list: List is empty
>> Apr 25 13:15:07.419280 [CLM  ] CLM CONFIGURATION CHANGE
>> Apr 25 13:15:07.419502 [CLM  ] New Configuration:
>> Apr 25 13:15:07.419577 [CLM  ] 	10.2.1.7
>> Apr 25 13:15:07.419644 [CLM  ] Members Left:
>> Apr 25 13:15:07.419708 [CLM  ] Members Joined:
>> Apr 25 13:15:07.419794 [SYNC ] This node is within the non-primary
>> component and will NOT provide any services.
>> Apr 25 13:15:07.471209 [TOTEM] entering OPERATIONAL state.
>> Apr 25 13:15:07.482646 [YKD  ] This processor is within the primary
>> component.
>> Apr 25 13:15:07.483046 [SYNC ] This node is within the primary
>> component and will provide service.
>> Apr 25 13:15:07.483189 [YKD  ] This processor is within the primary
>> component.
>> Apr 25 13:15:07.483264 [SYNC ] This node is within the primary
>> component and will provide service.
>> Apr 25 13:15:07.484664 [SYNC ] Synchronization barrier completed
>> Apr 25 13:15:07.484811 [SYNC ] Synchronization actions starting for
>> (openais cluster membership service B.01.01)
>> Apr 25 13:15:07.485107 [SYNC ] Synchronization actions done for
>> (openais cluster membership service B.01.01)
>> Apr 25 13:15:07.486871 [CLM  ] got nodejoin message 10.2.1.7
>> Apr 25 13:15:07.488347 [SYNC ] Synchronization barrier completed
>> Apr 25 13:15:07.488651 [SYNC ] Synchronization actions starting for
>> (openais checkpoint service B.01.01)
>> Apr 25 13:15:07.488777 [SYNC ] Synchronization actions done for
>> (openais checkpoint service B.01.01)
>> Apr 25 13:15:07.490859 [SYNC ] Synchronization barrier completed
>> Apr 25 13:15:07.491146 [SYNC ] Synchronization actions starting for
>> (openais event service B.01.01)
>> Apr 25 13:15:07.491245 [MAIN ] Can't find cluster node at 10.2.1.6
>> Apr 25 13:15:07.495344 [SYNC ] Synchronization actions done for
>> (openais event service B.01.01)
>> Apr 25 13:15:08.368732 [TOTEM] entering GATHER state.
>> Apr 25 13:15:08.371716 [TOTEM] Saving state aru 3e high seq  
>> received 3e
>> Apr 25 13:15:08.372794 [TOTEM] Storing new sequence id for ring 63748
>> Apr 25 13:15:08.373171 [TOTEM] entering COMMIT state.
>> Apr 25 13:15:08.375755 [TOTEM] entering RECOVERY state.
>> Apr 25 13:15:08.379433 [TOTEM] position [0] member 10.2.1.6:
>> Apr 25 13:15:08.379870 [TOTEM] previous ring seq 63736 rep 10.2.1.6
>> Apr 25 13:15:08.379946 [TOTEM] aru 0 high delivered 0 received flag 0
>> Apr 25 13:15:08.380232 [TOTEM] position [1] member 10.2.1.7:
>> Apr 25 13:15:08.380311 [TOTEM] previous ring seq 63744 rep 10.2.1.7
>> Apr 25 13:15:08.380383 [TOTEM] aru 3e high delivered 3e received  
>> flag 0
>> Apr 25 13:15:08.380467 [TOTEM] copying all old ring messages from  
>> 3f-3e.
>> Apr 25 13:15:08.381226 [TOTEM] Originated 0 messages in RECOVERY.
>> Apr 25 13:15:08.381323 [TOTEM] Originated for recovery:
>> Apr 25 13:15:08.381391 [TOTEM] Not Originated for recovery:
>> Apr 25 13:15:08.396649 [CLM  ] CLM CONFIGURATION CHANGE
>> Apr 25 13:15:08.396844 [CLM  ] New Configuration:
>> Apr 25 13:15:08.396925 [CLM  ] 	10.2.1.7
>> Apr 25 13:15:08.397138 [CLM  ] Members Left:
>> Apr 25 13:15:08.397205 [CLM  ] Members Joined:
>> Apr 25 13:15:08.397317 [CLM  ] CLM CONFIGURATION CHANGE
>> Apr 25 13:15:08.397387 [CLM  ] New Configuration:
>> Apr 25 13:15:08.397460 [CLM  ] 	10.2.1.6
>> Apr 25 13:15:08.397678 [CLM  ] 	10.2.1.7
>> Apr 25 13:15:08.397746 [CLM  ] Members Left:
>> Apr 25 13:15:08.397811 [CLM  ] Members Joined:
>> Apr 25 13:15:08.397883 [CLM  ] 	10.2.1.6
>> Apr 25 13:15:08.397973 [SYNC ] This node is within the non-primary
>> component and will NOT provide any services.
>> Apr 25 13:15:08.448519 [TOTEM] entering OPERATIONAL state.
>> Apr 25 13:15:09.766290 [YKD  ] This processor is within the primary
>> component.
>> Apr 25 13:15:09.766747 [SYNC ] This node is within the primary
>> component and will provide service.
>> Apr 25 13:15:09.767387 [YKD  ] This processor is within the primary
>> component.
>> Apr 25 13:15:09.767632 [SYNC ] This node is within the primary
>> component and will provide service.
>> Apr 25 13:15:09.780704 [SYNC ] Synchronization barrier completed
>> Apr 25 13:15:09.780914 [SYNC ] Synchronization actions starting for
>> (openais cluster membership service B.01.01)
>> Apr 25 13:15:09.781237 [SYNC ] Synchronization actions done for
>> (openais cluster membership service B.01.01)
>> Apr 25 13:15:09.785445 [CLM  ] got nodejoin message 10.2.1.7
>> Apr 25 13:15:09.789360 [CLM  ] got nodejoin message 10.2.1.6
>> Apr 25 13:15:09.794258 [SYNC ] Synchronization barrier completed
>> Apr 25 13:15:09.794597 [SYNC ] Synchronization actions starting for
>> (openais checkpoint service B.01.01)
>> Apr 25 13:15:09.795775 [SYNC ] Synchronization actions done for
>> (openais checkpoint service B.01.01)
>> Apr 25 13:15:09.803874 [SYNC ] Synchronization barrier completed
>> Apr 25 13:15:09.804301 [SYNC ] Synchronization actions starting for
>> (openais event service B.01.01)
>> Apr 25 13:15:10.804287 [TOTEM] The token was lost in state 1 from
>> timer 83ce000
>> Apr 25 13:15:10.805974 [TOTEM] Receive multicast socket recv buffer
>> size (144000 bytes).
>> Apr 25 13:15:10.806289 [TOTEM] Transmit multicast socket send buffer
>> size (144000 bytes).
>> Apr 25 13:15:10.807411 [TOTEM] entering GATHER state.
>> Apr 25 13:15:11.012684 [TOTEM] Saving state aru 73 high seq  
>> received 73
>> Apr 25 13:15:11.016427 [TOTEM] Storing new sequence id for ring 63752
>> Apr 25 13:15:11.016672 [TOTEM] entering COMMIT state.
>> Apr 25 13:15:11.020004 [TOTEM] entering RECOVERY state.
>> Apr 25 13:15:11.021161 [TOTEM] position [0] member 10.2.1.6:
>> Apr 25 13:15:11.021443 [TOTEM] previous ring seq 63748 rep 10.2.1.6
>> Apr 25 13:15:11.021523 [TOTEM] aru 73 high delivered 73 received  
>> flag 0
>> Apr 25 13:15:11.021609 [TOTEM] position [1] member 10.2.1.7:
>> Apr 25 13:15:11.021686 [TOTEM] previous ring seq 63748 rep 10.2.1.6
>> Apr 25 13:15:11.021760 [TOTEM] aru 73 high delivered 73 received  
>> flag 0
>> Apr 25 13:15:11.021969 [TOTEM] copying all old ring messages from  
>> 74-73.
>> Apr 25 13:15:11.022048 [TOTEM] Originated 0 messages in RECOVERY.
>> Apr 25 13:15:11.022120 [TOTEM] Originated for recovery:
>> Apr 25 13:15:11.022188 [TOTEM] Not Originated for recovery:
>> Apr 25 13:15:12.022718 [TOTEM] The token was lost in state 4 from
>> timer 83ce000
>> Apr 25 13:15:12.022950 [TOTEM] Restoring instance->my_aru 73 my high
>> seq received 73
>> Apr 25 13:15:12.023855 [TOTEM] entering GATHER state.
>> Apr 25 13:15:12.225043 [TOTEM] entering GATHER state.
>> Apr 25 13:15:12.225916 [TOTEM] Creating commit token because I am the
>> rep.
>> Apr 25 13:15:12.226887 [TOTEM] Storing new sequence id for ring 63756
>> Apr 25 13:15:12.227084 [TOTEM] entering COMMIT state.
>> Apr 25 13:15:12.228023 [TOTEM] entering RECOVERY state.
>> Apr 25 13:15:12.229495 [TOTEM] position [0] member 10.2.1.7:
>> Apr 25 13:15:12.229831 [TOTEM] previous ring seq 63748 rep 10.2.1.6
>> Apr 25 13:15:12.229910 [TOTEM] aru 73 high delivered 73 received  
>> flag 0
>> Apr 25 13:15:12.230040 [TOTEM] copying all old ring messages from  
>> 74-73.
>> Apr 25 13:15:12.230266 [TOTEM] Originated 0 messages in RECOVERY.
>> Apr 25 13:15:12.230339 [TOTEM] Originated for recovery:
>> Apr 25 13:15:12.230406 [TOTEM] Not Originated for recovery:
>> Apr 25 13:15:12.230996 [TOTEM] Sending initial ORF token
>> Apr 25 13:15:12.237299 [CLM  ] CLM CONFIGURATION CHANGE
>> Apr 25 13:15:12.237481 [CLM  ] New Configuration:
>> Apr 25 13:15:12.237563 [CLM  ] 	10.2.1.7
>> Apr 25 13:15:12.237792 [CLM  ] Members Left:
>> Apr 25 13:15:12.237866 [CLM  ] 	10.2.1.6
>> Apr 25 13:15:12.237933 [CLM  ] Members Joined:
>> Apr 25 13:15:12.238044 [CKPT ] clean_checkpoint_list: List is empty
>> Apr 25 13:15:12.238294 [CLM  ] CLM CONFIGURATION CHANGE
>> Apr 25 13:15:12.238367 [CLM  ] New Configuration:
>> Apr 25 13:15:12.238441 [CLM  ] 	10.2.1.7
>> Apr 25 13:15:12.238508 [CLM  ] Members Left:
>> Apr 25 13:15:12.238575 [CLM  ] Members Joined:
>> Apr 25 13:15:12.238971 [SYNC ] This node is within the non-primary
>> component and will NOT provide any services.
>> Apr 25 13:15:12.290823 [TOTEM] entering OPERATIONAL state.
>> Apr 25 13:15:12.301039 [YKD  ] This processor is within the primary
>> component.
>> Apr 25 13:15:12.301487 [SYNC ] This node is within the primary
>> component and will provide service.
>> Apr 25 13:15:12.303028 [SYNC ] Synchronization barrier completed
>> Apr 25 13:15:12.303346 [SYNC ] Synchronization actions starting for
>> (openais cluster membership service B.01.01)
>> Apr 25 13:15:12.303501 [SYNC ] Synchronization actions done for
>> (openais cluster membership service B.01.01)
>> Apr 25 13:15:12.304808 [CLM  ] got nodejoin message 10.2.1.7
>> Apr 25 13:15:12.305985 [SYNC ] Synchronization barrier completed
>> Apr 25 13:15:12.306116 [SYNC ] Synchronization actions starting for
>> (openais checkpoint service B.01.01)
>> Apr 25 13:15:12.306824 [SYNC ] Synchronization actions done for
>> (openais checkpoint service B.01.01)
>> Apr 25 13:15:12.309345 [SYNC ] Synchronization barrier completed
>> Apr 25 13:15:12.309499 [SYNC ] Synchronization actions starting for
>> (openais event service B.01.01)
>> Apr 25 13:15:12.314866 [TOTEM] entering GATHER state.
>> Apr 25 13:15:12.317401 [TOTEM] Saving state aru 2c high seq  
>> received 2c
>> Apr 25 13:15:12.318473 [TOTEM] Storing new sequence id for ring 63760
>> Apr 25 13:15:12.318853 [TOTEM] entering COMMIT state.
>> Apr 25 13:15:12.323519 [TOTEM] entering RECOVERY state.
>> Apr 25 13:15:12.324943 [TOTEM] position [0] member 10.2.1.6:
>> Apr 25 13:15:12.325107 [TOTEM] previous ring seq 63748 rep 10.2.1.6
>> Apr 25 13:15:12.325329 [TOTEM] aru 73 high delivered 73 received  
>> flag 0
>> Apr 25 13:15:12.325412 [TOTEM] position [1] member 10.2.1.7:
>> Apr 25 13:15:12.325487 [TOTEM] previous ring seq 63756 rep 10.2.1.7
>> Apr 25 13:15:12.325557 [TOTEM] aru 2c high delivered 2b received  
>> flag 0
>> Apr 25 13:15:12.325636 [TOTEM] copying all old ring messages from  
>> 2d-2c.
>> Apr 25 13:15:12.325867 [TOTEM] Originated 0 messages in RECOVERY.
>> Apr 25 13:15:12.325937 [TOTEM] Originated for recovery:
>> Apr 25 13:15:12.326002 [TOTEM] Not Originated for recovery:
>> Apr 25 13:15:13.327013 [TOTEM] The token was lost in state 4 from
>> timer 83ce000
>> Apr 25 13:15:13.327242 [TOTEM] Restoring instance->my_aru 2c my high
>> seq received 2c
>> Apr 25 13:15:13.328109 [TOTEM] entering GATHER state.
>> Apr 25 13:15:13.529877 [TOTEM] entering GATHER state.
>> Apr 25 13:15:13.530717 [TOTEM] Creating commit token because I am the
>> rep.
>> Apr 25 13:15:13.531687 [TOTEM] Storing new sequence id for ring 63764
>> Apr 25 13:15:13.531887 [TOTEM] entering COMMIT state.
>> Apr 25 13:15:13.532459 [TOTEM] entering RECOVERY state.
>> Apr 25 13:15:13.533777 [TOTEM] position [0] member 10.2.1.7:
>> Apr 25 13:15:13.533927 [TOTEM] previous ring seq 63756 rep 10.2.1.7
>> Apr 25 13:15:13.534150 [TOTEM] aru 2c high delivered 2b received  
>> flag 0
>> Apr 25 13:15:13.534232 [TOTEM] copying all old ring messages from  
>> 2d-2c.
>> Apr 25 13:15:13.534309 [TOTEM] Originated 0 messages in RECOVERY.
>> Apr 25 13:15:13.534379 [TOTEM] Originated for recovery:
>> Apr 25 13:15:13.534444 [TOTEM] Not Originated for recovery:
>> Apr 25 13:15:13.535177 [TOTEM] Sending initial ORF token
>> Apr 25 13:15:13.543807 [CLM  ] CLM CONFIGURATION CHANGE
>> Apr 25 13:15:13.544203 [CLM  ] New Configuration:
>> Apr 25 13:15:13.544284 [CLM  ] 	10.2.1.7
>> Apr 25 13:15:13.544352 [CLM  ] Members Left:
>> Apr 25 13:15:13.544415 [CLM  ] Members Joined:
>> Apr 25 13:15:13.544645 [CLM  ] CLM CONFIGURATION CHANGE
>> Apr 25 13:15:13.544718 [CLM  ] New Configuration:
>> Apr 25 13:15:13.544790 [CLM  ] 	10.2.1.7
>> Apr 25 13:15:13.544856 [CLM  ] Members Left:
>> Apr 25 13:15:13.544921 [CLM  ] Members Joined:
>> Apr 25 13:15:13.545122 [SYNC ] This node is within the non-primary
>> component and will NOT provide any services.
>> Apr 25 13:15:13.594240 [TOTEM] entering OPERATIONAL state.
>> Apr 25 13:15:13.604969 [YKD  ] This processor is within the primary
>> component.
>> Apr 25 13:15:13.605348 [SYNC ] This node is within the primary
>> component and will provide service.
>> Apr 25 13:15:13.606757 [SYNC ] Synchronization barrier completed
>> Apr 25 13:15:13.606908 [SYNC ] Synchronization actions starting for
>> (openais cluster membership service B.01.01)
>> Apr 25 13:15:13.607207 [SYNC ] Synchronization actions done for
>> (openais cluster membership service B.01.01)
>> Apr 25 13:15:13.608359 [CLM  ] got nodejoin message 10.2.1.7
>> Apr 25 13:15:13.609707 [SYNC ] Synchronization barrier completed
>> Apr 25 13:15:13.609839 [SYNC ] Synchronization actions starting for
>> (openais checkpoint service B.01.01)
>> Apr 25 13:15:13.609962 [SYNC ] Synchronization actions done for
>> (openais checkpoint service B.01.01)
>> Apr 25 13:15:13.612200 [SYNC ] Synchronization barrier completed
>> Apr 25 13:15:13.612341 [SYNC ] Synchronization actions starting for
>> (openais event service B.01.01)
>> Apr 25 13:15:13.616743 [SYNC ] Synchronization actions done for
>> (openais event service B.01.01)
>> Apr 25 13:15:13.645939 [TOTEM] entering GATHER state.
>> Apr 25 13:15:13.648425 [TOTEM] Saving state aru 2b high seq  
>> received 2b
>> Apr 25 13:15:13.649434 [TOTEM] Storing new sequence id for ring 63768
>> Apr 25 13:15:13.649811 [TOTEM] entering COMMIT state.
>> Apr 25 13:15:13.654423 [TOTEM] entering RECOVERY state.
>> Apr 25 13:15:13.655888 [TOTEM] position [0] member 10.2.1.6:
>> Apr 25 13:15:13.656440 [TOTEM] previous ring seq 63748 rep 10.2.1.6
>> Apr 25 13:15:13.656544 [TOTEM] aru 73 high delivered 73 received  
>> flag 0
>> Apr 25 13:15:13.658976 [TOTEM] position [1] member 10.2.1.7:
>> Apr 25 13:15:13.659273 [TOTEM] previous ring seq 63764 rep 10.2.1.7
>> Apr 25 13:15:13.659355 [TOTEM] aru 2b high delivered 2b received  
>> flag 0
>> Apr 25 13:15:13.659438 [TOTEM] copying all old ring messages from  
>> 2c-2b.
>> Apr 25 13:15:13.659514 [TOTEM] Originated 0 messages in RECOVERY.
>> Apr 25 13:15:13.659714 [TOTEM] Originated for recovery:
>> Apr 25 13:15:13.659785 [TOTEM] Not Originated for recovery:
>> Apr 25 13:15:14.659438 [TOTEM] The token was lost in state 4 from
>> timer 83ce000
>> Apr 25 13:15:14.659700 [TOTEM] Restoring instance->my_aru 2b my high
>> seq received 2b
>> Apr 25 13:15:14.660615 [TOTEM] entering GATHER state.
>> ...
>>
>> _______________________________________________
>> Openais mailing list
>> Openais at lists.osdl.org
>> https://lists.osdl.org/mailman/listinfo/openais
>
>




More information about the Openais mailing list