[Openais] Re: defect 73 partial fix

Steven Dake sdake at mvista.com
Wed Oct 20 10:19:45 PDT 2004


Mark,

The update time bug..  What you describe where only one processor forms
a configuration still exists with this patch?  Could you file a bug on
updating system time?  I think right now this will definately cause a
configuration change (because the timeouts will timeout) but it should
reocver always.


On Tue, 2004-10-19 at 11:06, Mark Haverkamp wrote:
> On Mon, 2004-10-18 at 15:49 -0700, Steven Dake wrote:
> > Mark
> > 
> > Here is a partial fix for #73.  Basically I classify this bug as the
> > fact that openais behaves poorly if the executive is started and stopped
> > with the debugger.  In all cases we want this to work properly by
> > kicking the processor out of the membership and making correct conensus
> > decisions.
> > 
> > I find that the current gmi doesn't do this very well (yet).
> > 
> > Try out the partial fix and let me know if it improves things.
> 
> It may have helped.  It didn't help the gdb problem, but it seemed to
> help the problem where time advances on a node (i.e. call ntpdate to
> update the time).  It did seem to recover from this after a while.  It
> did act funny though.  The node that I updated the time on, thought that
> it had a token timeout and went through some config change cycles.  At
> one point it thought that it was the only node, but the others were
> still there and sending messages.  This confused cl017 because it was
> receiving messages from nodes that it didn't know about.  After a while,
> it came out of it and saw the other nodes again.  I've included the log
> from cl017.
> 
> 
> > 
> > During this process, I noticed that part of the membership algorithm is
> > not implemented properly (the failed processor conensus).  I'll work
> > that out next and see if that gets us better consensus behavior.
> > 
> > Thanks
> > -steve
> > 
> 
> 
> Oct 18 15:57:50 cl017 [NOTICE  ] [GMI  ] All processors unplugged, allowing messages to be transmitted.
> 
> Updated time here:
> 
> Oct 19 10:49:05 cl017 [WARNING ] [GMI  ] Token loss in OPERATIONAL.
> Oct 19 10:49:05 cl017 [NOTICE  ] [GMI  ] entering GATHER state.
> Oct 19 10:49:05 cl017 [NOTICE  ] [GMI  ] SENDING attempt join because this node is ring rep.
> Oct 19 10:49:05 cl017 [NOTICE  ] [GMI  ] Got attempt join from 192.168.1.8
> Oct 19 10:49:05 cl017 [NOTICE  ] [GMI  ] CONSENSUS reached!
> Oct 19 10:49:05 cl017 [NOTICE  ] [GMI  ] EVS recovery of messages complete, transitioning to operational.
> Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] CLM CONFIGURATION CHANGE
> Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] New Configuration:
> Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.8
> Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.17
> Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] Members Left:
> Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.18
> Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.19
> Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] Members Joined:
> Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] CLM CONFIGURATION CHANGE
> Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] New Configuration:
> Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.8
> Oct 19 10:49:05 cl017 [NOTICE  ] [GMI  ] All services unplugged, unplugging processor
> Oct 19 10:49:07 cl017 [NOTICE  ] [GMI  ] Got attempt join from 192.168.1.18
> Oct 19 10:49:07 cl017 [NOTICE  ] [GMI  ] rep is 192.168.1.8, not handling attempt join.
> Oct 19 10:49:07 cl017 [NOTICE  ] [GMI  ] Got attempt join from 192.168.1.8
> Oct 19 10:49:07 cl017 [NOTICE  ] [GMI  ] rep is 192.168.1.8, not handling attempt join.
> Oct 19 10:49:07 cl017 [NOTICE  ] [GMI  ] Got attempt join from 192.168.1.19
> Oct 19 10:49:07 cl017 [NOTICE  ] [GMI  ] rep is 192.168.1.8, not handling attempt join.
> Oct 19 10:49:07 cl017 [NOTICE  ] [GMI  ] EVS recovery of messages complete, transitioning to operational.
> Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] CLM CONFIGURATION CHANGE
> Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] New Configuration:
> Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.8
> Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.17
> Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.18
> Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.19
> Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] Members Left:
> Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] Members Joined:
> Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.18
> Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.19
> Oct 19 10:49:07 cl017 [WARNING ] [GMI  ] Token being retransmitted.
> Oct 19 10:49:08 cl017 [NOTICE  ] [GMI  ] Got attempt join from 192.168.1.19
> Oct 19 10:49:08 cl017 [NOTICE  ] [GMI  ] rep is 192.168.1.8, not handling attempt join.
> Oct 19 10:49:08 cl017 [WARNING ] [GMI  ] Token loss in OPERATIONAL.
> Oct 19 10:49:08 cl017 [NOTICE  ] [GMI  ] entering GATHER state.
> Oct 19 10:49:08 cl017 [NOTICE  ] [GMI  ] SENDING attempt join because this node is ring rep.
> Oct 19 10:49:08 cl017 [NOTICE  ] [GMI  ] Got attempt join from 192.168.1.18
> Oct 19 10:49:08 cl017 [NOTICE  ] [GMI  ] CONSENSUS reached!
> Oct 19 10:49:08 cl017 [NOTICE  ] [GMI  ] EVS recovery of messages complete, transitioning to operational.
> Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] CLM CONFIGURATION CHANGE
> Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] New Configuration:
> Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.17
> Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.18
> Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.19
> Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] Members Left:
> Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.8
> Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] Members Joined:
> Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] CLM CONFIGURATION CHANGE
> Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] New Configuration:
> Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.17
> Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] got nodejoin message 192.168.1.17
> Oct 19 10:49:08 cl017 [NOTICE  ] [GMI  ] All services unplugged, unplugging processor
> 
> These are evt messages saying that it received an event from a node 
> that isn't in the cluster. As you can see above, cl017 thinks that it is the 
> only node.
> 
> Oct 19 10:49:08 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:08 cl017 last message repeated 3 times
> Oct 19 10:49:08 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:08 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:08 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:08 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:08 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:08 cl017 last message repeated 2 times
> Oct 19 10:49:08 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:08 cl017 last message repeated 11 times
> Oct 19 10:49:08 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:09 cl017 last message repeated 106 times
> Oct 19 10:49:09 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:09 cl017 last message repeated 736 times
> Oct 19 10:49:09 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:09 cl017 last message repeated 10 times
> Oct 19 10:49:09 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:09 cl017 last message repeated 30 times
> Oct 19 10:49:09 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:09 cl017 last message repeated 11 times
> Oct 19 10:49:09 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:09 cl017 last message repeated 34 times
> Oct 19 10:49:09 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:11 cl017 last message repeated 279 times
> Oct 19 10:49:11 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:11 cl017 last message repeated 12 times
> Oct 19 10:49:11 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:11 cl017 last message repeated 11 times
> Oct 19 10:49:11 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:11 cl017 last message repeated 48 times
> Oct 19 10:49:11 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:11 cl017 last message repeated 11 times
> Oct 19 10:49:11 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:11 cl017 last message repeated 49 times
> Oct 19 10:49:11 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:11 cl017 last message repeated 11 times
> Oct 19 10:49:11 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:11 cl017 last message repeated 28 times
> Oct 19 10:49:11 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:11 cl017 last message repeated 106 times
> Oct 19 10:49:11 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:12 cl017 last message repeated 477 times
> Oct 19 10:49:12 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:12 cl017 last message repeated 11 times
> Oct 19 10:49:12 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:12 cl017 last message repeated 45 times
> Oct 19 10:49:12 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:12 cl017 last message repeated 11 times
> Oct 19 10:49:12 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:12 cl017 last message repeated 49 times
> Oct 19 10:49:12 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:12 cl017 last message repeated 11 times
> Oct 19 10:49:12 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:12 cl017 last message repeated 40 times
> Oct 19 10:49:12 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:12 cl017 last message repeated 181 times
> Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:13 cl017 last message repeated 239 times
> Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:13 cl017 last message repeated 11 times
> Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:13 cl017 last message repeated 37 times
> Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:13 cl017 last message repeated 11 times
> Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:13 cl017 last message repeated 47 times
> Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:13 cl017 last message repeated 11 times
> Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:13 cl017 last message repeated 45 times
> Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:13 cl017 last message repeated 11 times
> Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:13 cl017 last message repeated 41 times
> Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:13 cl017 last message repeated 162 times
> Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:14 cl017 last message repeated 224 times
> Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:14 cl017 last message repeated 2 times
> Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:14 cl017 last message repeated 8 times
> Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:14 cl017 last message repeated 2 times
> Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:14 cl017 last message repeated 11 times
> Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:14 cl017 last message repeated 4 times
> Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:14 cl017 last message repeated 20 times
> Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:14 cl017 last message repeated 14 times
> Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:14 cl017 last message repeated 34 times
> Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:14 cl017 last message repeated 11 times
> Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:14 cl017 last message repeated 40 times
> Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:14 cl017 last message repeated 11 times
> Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:14 cl017 last message repeated 36 times
> Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:14 cl017 last message repeated 120 times
> Oct 19 10:49:15 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:15 cl017 last message repeated 15 times
> Oct 19 10:49:15 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:15 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:15 cl017 last message repeated 11 times
> Oct 19 10:49:15 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:15 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:15 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:15 cl017 last message repeated 26 times
> Oct 19 10:49:15 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:15 cl017 last message repeated 11 times
> Oct 19 10:49:15 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:16 cl017 last message repeated 12 times
> Oct 19 10:49:16 cl017 [SECURITY] [GMI  ] Received message has invalid digest... ignoring.
> Oct 19 10:49:16 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:16 cl017 last message repeated 11 times
> Oct 19 10:49:16 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:16 cl017 last message repeated 11 times
> Oct 19 10:49:16 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:16 cl017 last message repeated 25 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 12 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 42 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 12 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 87 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 3 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 7 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 57 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 30 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 29 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 34 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 30 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 29 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 30 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 30 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:19 cl017 last message repeated 765 times
> Oct 19 10:49:19 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:19 cl017 last message repeated 294 times
> Oct 19 10:49:19 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:19 cl017 last message repeated 11 times
> Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:20 cl017 last message repeated 41 times
> Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:20 cl017 last message repeated 11 times
> Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:20 cl017 last message repeated 47 times
> Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:20 cl017 last message repeated 11 times
> Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:20 cl017 last message repeated 41 times
> Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:20 cl017 last message repeated 11 times
> Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:20 cl017 last message repeated 36 times
> Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:20 cl017 last message repeated 11 times
> Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:20 cl017 last message repeated 39 times
> Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:20 cl017 last message repeated 119 times
> Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:21 cl017 last message repeated 16 times
> Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:21 cl017 last message repeated 2 times
> Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:21 cl017 last message repeated 8 times
> Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:21 cl017 last message repeated 2 times
> Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:21 cl017 last message repeated 11 times
> Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:21 cl017 last message repeated 7 times
> Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:21 cl017 last message repeated 11 times
> Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:21 cl017 last message repeated 11 times
> Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:21 cl017 last message repeated 30 times
> Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:21 cl017 last message repeated 11 times
> Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:21 cl017 last message repeated 38 times
> Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:21 cl017 last message repeated 11 times
> Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:21 cl017 last message repeated 42 times
> Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:21 cl017 last message repeated 11 times
> Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:21 cl017 last message repeated 49 times
> Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:22 cl017 last message repeated 146 times
> Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:22 cl017 last message repeated 10 times
> Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:22 cl017 last message repeated 10 times
> Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:22 cl017 last message repeated 9 times
> Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:22 cl017 last message repeated 3 times
> Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:22 cl017 last message repeated 6 times
> Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:22 cl017 last message repeated 10 times
> Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
> Oct 19 10:49:23 cl017 last message repeated 16 times
> Oct 19 10:49:23 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
> Oct 19 10:49:23 cl017 last message repeated 11 times
> Oct 19 10:49:23 cl017 [NOTICE  ] [GMI  ] EVS recovery of messages complete, transitioning to operational.
> Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] CLM CONFIGURATION CHANGE
> Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] New Configuration:
> Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.8
> Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.17
> Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.18
> Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.19
> Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] Members Left:
> Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] Members Joined:
> Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.8
> Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] got nodejoin message 192.168.1.17
> Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] got nodejoin message 192.168.1.18
> Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] got nodejoin message 192.168.1.19
> Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] got nodejoin message 192.168.1.8
> Oct 19 10:49:23 cl017 [NOTICE  ] [GMI  ] All services unplugged, unplugging processor
> Oct 19 10:49:23 cl017 [NOTICE  ] [EVT  ] Channel Chan1, total 2, local 1
> Oct 19 10:49:23 cl017 [NOTICE  ] [GMI  ] All processors unplugged, allowing messages to be transmitted.
> 
> Back to normal again.....
> 
> 




More information about the Openais mailing list