[Openais] Re: defect 73 partial fix
Steven Dake
sdake at mvista.com
Wed Oct 20 10:19:45 PDT 2004
Mark,
The update time bug.. What you describe where only one processor forms
a configuration still exists with this patch? Could you file a bug on
updating system time? I think right now this will definately cause a
configuration change (because the timeouts will timeout) but it should
reocver always.
On Tue, 2004-10-19 at 11:06, Mark Haverkamp wrote:
> On Mon, 2004-10-18 at 15:49 -0700, Steven Dake wrote:
> > Mark
> >
> > Here is a partial fix for #73. Basically I classify this bug as the
> > fact that openais behaves poorly if the executive is started and stopped
> > with the debugger. In all cases we want this to work properly by
> > kicking the processor out of the membership and making correct conensus
> > decisions.
> >
> > I find that the current gmi doesn't do this very well (yet).
> >
> > Try out the partial fix and let me know if it improves things.
>
> It may have helped. It didn't help the gdb problem, but it seemed to
> help the problem where time advances on a node (i.e. call ntpdate to
> update the time). It did seem to recover from this after a while. It
> did act funny though. The node that I updated the time on, thought that
> it had a token timeout and went through some config change cycles. At
> one point it thought that it was the only node, but the others were
> still there and sending messages. This confused cl017 because it was
> receiving messages from nodes that it didn't know about. After a while,
> it came out of it and saw the other nodes again. I've included the log
> from cl017.
>
>
> >
> > During this process, I noticed that part of the membership algorithm is
> > not implemented properly (the failed processor conensus). I'll work
> > that out next and see if that gets us better consensus behavior.
> >
> > Thanks
> > -steve
> >
>
>
> Oct 18 15:57:50 cl017 [NOTICE ] [GMI ] All processors unplugged, allowing messages to be transmitted.
>
> Updated time here:
>
> Oct 19 10:49:05 cl017 [WARNING ] [GMI ] Token loss in OPERATIONAL.
> Oct 19 10:49:05 cl017 [NOTICE ] [GMI ] entering GATHER state.
> Oct 19 10:49:05 cl017 [NOTICE ] [GMI ] SENDING attempt join because this node is ring rep.
> Oct 19 10:49:05 cl017 [NOTICE ] [GMI ] Got attempt join from 192.168.1.8
> Oct 19 10:49:05 cl017 [NOTICE ] [GMI ] CONSENSUS reached!
> Oct 19 10:49:05 cl017 [NOTICE ] [GMI ] EVS recovery of messages complete, transitioning to operational.
> Oct 19 10:49:05 cl017 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
> Oct 19 10:49:05 cl017 [NOTICE ] [CLM ] New Configuration:
> Oct 19 10:49:05 cl017 [NOTICE ] [CLM ] ^I192.168.1.8
> Oct 19 10:49:05 cl017 [NOTICE ] [CLM ] ^I192.168.1.17
> Oct 19 10:49:05 cl017 [NOTICE ] [CLM ] Members Left:
> Oct 19 10:49:05 cl017 [NOTICE ] [CLM ] ^I192.168.1.18
> Oct 19 10:49:05 cl017 [NOTICE ] [CLM ] ^I192.168.1.19
> Oct 19 10:49:05 cl017 [NOTICE ] [CLM ] Members Joined:
> Oct 19 10:49:05 cl017 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
> Oct 19 10:49:05 cl017 [NOTICE ] [CLM ] New Configuration:
> Oct 19 10:49:05 cl017 [NOTICE ] [CLM ] ^I192.168.1.8
> Oct 19 10:49:05 cl017 [NOTICE ] [GMI ] All services unplugged, unplugging processor
> Oct 19 10:49:07 cl017 [NOTICE ] [GMI ] Got attempt join from 192.168.1.18
> Oct 19 10:49:07 cl017 [NOTICE ] [GMI ] rep is 192.168.1.8, not handling attempt join.
> Oct 19 10:49:07 cl017 [NOTICE ] [GMI ] Got attempt join from 192.168.1.8
> Oct 19 10:49:07 cl017 [NOTICE ] [GMI ] rep is 192.168.1.8, not handling attempt join.
> Oct 19 10:49:07 cl017 [NOTICE ] [GMI ] Got attempt join from 192.168.1.19
> Oct 19 10:49:07 cl017 [NOTICE ] [GMI ] rep is 192.168.1.8, not handling attempt join.
> Oct 19 10:49:07 cl017 [NOTICE ] [GMI ] EVS recovery of messages complete, transitioning to operational.
> Oct 19 10:49:07 cl017 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
> Oct 19 10:49:07 cl017 [NOTICE ] [CLM ] New Configuration:
> Oct 19 10:49:07 cl017 [NOTICE ] [CLM ] ^I192.168.1.8
> Oct 19 10:49:07 cl017 [NOTICE ] [CLM ] ^I192.168.1.17
> Oct 19 10:49:07 cl017 [NOTICE ] [CLM ] ^I192.168.1.18
> Oct 19 10:49:07 cl017 [NOTICE ] [CLM ] ^I192.168.1.19
> Oct 19 10:49:07 cl017 [NOTICE ] [CLM ] Members Left:
> Oct 19 10:49:07 cl017 [NOTICE ] [CLM ] Members Joined:
> Oct 19 10:49:07 cl017 [NOTICE ] [CLM ] ^I192.168.1.18
> Oct 19 10:49:07 cl017 [NOTICE ] [CLM ] ^I192.168.1.19
> Oct 19 10:49:07 cl017 [WARNING ] [GMI ] Token being retransmitted.
> Oct 19 10:49:08 cl017 [NOTICE ] [GMI ] Got attempt join from 192.168.1.19
> Oct 19 10:49:08 cl017 [NOTICE ] [GMI ] rep is 192.168.1.8, not handling attempt join.
> Oct 19 10:49:08 cl017 [WARNING ] [GMI ] Token loss in OPERATIONAL.
> Oct 19 10:49:08 cl017 [NOTICE ] [GMI ] entering GATHER state.
> Oct 19 10:49:08 cl017 [NOTICE ] [GMI ] SENDING attempt join because this node is ring rep.
> Oct 19 10:49:08 cl017 [NOTICE ] [GMI ] Got attempt join from 192.168.1.18
> Oct 19 10:49:08 cl017 [NOTICE ] [GMI ] CONSENSUS reached!
> Oct 19 10:49:08 cl017 [NOTICE ] [GMI ] EVS recovery of messages complete, transitioning to operational.
> Oct 19 10:49:08 cl017 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
> Oct 19 10:49:08 cl017 [NOTICE ] [CLM ] New Configuration:
> Oct 19 10:49:08 cl017 [NOTICE ] [CLM ] ^I192.168.1.17
> Oct 19 10:49:08 cl017 [NOTICE ] [CLM ] ^I192.168.1.18
> Oct 19 10:49:08 cl017 [NOTICE ] [CLM ] ^I192.168.1.19
> Oct 19 10:49:08 cl017 [NOTICE ] [CLM ] Members Left:
> Oct 19 10:49:08 cl017 [NOTICE ] [CLM ] ^I192.168.1.8
> Oct 19 10:49:08 cl017 [NOTICE ] [CLM ] Members Joined:
> Oct 19 10:49:08 cl017 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
> Oct 19 10:49:08 cl017 [NOTICE ] [CLM ] New Configuration:
> Oct 19 10:49:08 cl017 [NOTICE ] [CLM ] ^I192.168.1.17
> Oct 19 10:49:08 cl017 [NOTICE ] [CLM ] got nodejoin message 192.168.1.17
> Oct 19 10:49:08 cl017 [NOTICE ] [GMI ] All services unplugged, unplugging processor
>
> These are evt messages saying that it received an event from a node
> that isn't in the cluster. As you can see above, cl017 thinks that it is the
> only node.
>
> Oct 19 10:49:08 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:08 cl017 last message repeated 3 times
> Oct 19 10:49:08 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:08 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:08 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:08 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:08 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:08 cl017 last message repeated 2 times
> Oct 19 10:49:08 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:08 cl017 last message repeated 11 times
> Oct 19 10:49:08 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:09 cl017 last message repeated 106 times
> Oct 19 10:49:09 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:09 cl017 last message repeated 736 times
> Oct 19 10:49:09 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:09 cl017 last message repeated 10 times
> Oct 19 10:49:09 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:09 cl017 last message repeated 30 times
> Oct 19 10:49:09 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:09 cl017 last message repeated 11 times
> Oct 19 10:49:09 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:09 cl017 last message repeated 34 times
> Oct 19 10:49:09 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:11 cl017 last message repeated 279 times
> Oct 19 10:49:11 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:11 cl017 last message repeated 12 times
> Oct 19 10:49:11 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:11 cl017 last message repeated 11 times
> Oct 19 10:49:11 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:11 cl017 last message repeated 48 times
> Oct 19 10:49:11 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:11 cl017 last message repeated 11 times
> Oct 19 10:49:11 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:11 cl017 last message repeated 49 times
> Oct 19 10:49:11 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:11 cl017 last message repeated 11 times
> Oct 19 10:49:11 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:11 cl017 last message repeated 28 times
> Oct 19 10:49:11 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:11 cl017 last message repeated 106 times
> Oct 19 10:49:11 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:12 cl017 last message repeated 477 times
> Oct 19 10:49:12 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:12 cl017 last message repeated 11 times
> Oct 19 10:49:12 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:12 cl017 last message repeated 45 times
> Oct 19 10:49:12 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:12 cl017 last message repeated 11 times
> Oct 19 10:49:12 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:12 cl017 last message repeated 49 times
> Oct 19 10:49:12 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:12 cl017 last message repeated 11 times
> Oct 19 10:49:12 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:12 cl017 last message repeated 40 times
> Oct 19 10:49:12 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:12 cl017 last message repeated 181 times
> Oct 19 10:49:13 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:13 cl017 last message repeated 239 times
> Oct 19 10:49:13 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:13 cl017 last message repeated 11 times
> Oct 19 10:49:13 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:13 cl017 last message repeated 37 times
> Oct 19 10:49:13 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:13 cl017 last message repeated 11 times
> Oct 19 10:49:13 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:13 cl017 last message repeated 47 times
> Oct 19 10:49:13 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:13 cl017 last message repeated 11 times
> Oct 19 10:49:13 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:13 cl017 last message repeated 45 times
> Oct 19 10:49:13 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:13 cl017 last message repeated 11 times
> Oct 19 10:49:13 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:13 cl017 last message repeated 41 times
> Oct 19 10:49:13 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:13 cl017 last message repeated 162 times
> Oct 19 10:49:14 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:14 cl017 last message repeated 224 times
> Oct 19 10:49:14 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:14 cl017 last message repeated 2 times
> Oct 19 10:49:14 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:14 cl017 last message repeated 8 times
> Oct 19 10:49:14 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:14 cl017 last message repeated 2 times
> Oct 19 10:49:14 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:14 cl017 last message repeated 11 times
> Oct 19 10:49:14 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:14 cl017 last message repeated 4 times
> Oct 19 10:49:14 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:14 cl017 last message repeated 20 times
> Oct 19 10:49:14 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:14 cl017 last message repeated 14 times
> Oct 19 10:49:14 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:14 cl017 last message repeated 34 times
> Oct 19 10:49:14 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:14 cl017 last message repeated 11 times
> Oct 19 10:49:14 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:14 cl017 last message repeated 40 times
> Oct 19 10:49:14 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:14 cl017 last message repeated 11 times
> Oct 19 10:49:14 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:14 cl017 last message repeated 36 times
> Oct 19 10:49:14 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:14 cl017 last message repeated 120 times
> Oct 19 10:49:15 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:15 cl017 last message repeated 15 times
> Oct 19 10:49:15 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:15 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:15 cl017 last message repeated 11 times
> Oct 19 10:49:15 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:15 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:15 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:15 cl017 last message repeated 26 times
> Oct 19 10:49:15 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:15 cl017 last message repeated 11 times
> Oct 19 10:49:15 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:16 cl017 last message repeated 12 times
> Oct 19 10:49:16 cl017 [SECURITY] [GMI ] Received message has invalid digest... ignoring.
> Oct 19 10:49:16 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:16 cl017 last message repeated 11 times
> Oct 19 10:49:16 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:16 cl017 last message repeated 11 times
> Oct 19 10:49:16 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:16 cl017 last message repeated 25 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 12 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 42 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 12 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 87 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 3 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 7 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 57 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 30 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 29 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 34 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 30 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 29 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 30 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 last message repeated 30 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:17 cl017 last message repeated 11 times
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:17 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:19 cl017 last message repeated 765 times
> Oct 19 10:49:19 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:19 cl017 last message repeated 294 times
> Oct 19 10:49:19 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:19 cl017 last message repeated 11 times
> Oct 19 10:49:20 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:20 cl017 last message repeated 41 times
> Oct 19 10:49:20 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:20 cl017 last message repeated 11 times
> Oct 19 10:49:20 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:20 cl017 last message repeated 47 times
> Oct 19 10:49:20 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:20 cl017 last message repeated 11 times
> Oct 19 10:49:20 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:20 cl017 last message repeated 41 times
> Oct 19 10:49:20 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:20 cl017 last message repeated 11 times
> Oct 19 10:49:20 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:20 cl017 last message repeated 36 times
> Oct 19 10:49:20 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:20 cl017 last message repeated 11 times
> Oct 19 10:49:20 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:20 cl017 last message repeated 39 times
> Oct 19 10:49:20 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:20 cl017 last message repeated 119 times
> Oct 19 10:49:21 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:21 cl017 last message repeated 16 times
> Oct 19 10:49:21 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:21 cl017 last message repeated 2 times
> Oct 19 10:49:21 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:21 cl017 last message repeated 8 times
> Oct 19 10:49:21 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:21 cl017 last message repeated 2 times
> Oct 19 10:49:21 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:21 cl017 last message repeated 11 times
> Oct 19 10:49:21 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:21 cl017 last message repeated 7 times
> Oct 19 10:49:21 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:21 cl017 last message repeated 11 times
> Oct 19 10:49:21 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:21 cl017 last message repeated 11 times
> Oct 19 10:49:21 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:21 cl017 last message repeated 30 times
> Oct 19 10:49:21 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:21 cl017 last message repeated 11 times
> Oct 19 10:49:21 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:21 cl017 last message repeated 38 times
> Oct 19 10:49:21 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:21 cl017 last message repeated 11 times
> Oct 19 10:49:21 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:21 cl017 last message repeated 42 times
> Oct 19 10:49:21 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:21 cl017 last message repeated 11 times
> Oct 19 10:49:21 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:21 cl017 last message repeated 49 times
> Oct 19 10:49:21 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:22 cl017 last message repeated 146 times
> Oct 19 10:49:22 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:22 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:22 cl017 last message repeated 10 times
> Oct 19 10:49:22 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:22 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:22 cl017 last message repeated 10 times
> Oct 19 10:49:22 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:22 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:22 cl017 last message repeated 9 times
> Oct 19 10:49:22 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:22 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:22 cl017 last message repeated 3 times
> Oct 19 10:49:22 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:22 cl017 last message repeated 6 times
> Oct 19 10:49:22 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:22 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:22 cl017 last message repeated 10 times
> Oct 19 10:49:22 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:22 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.18
> Oct 19 10:49:23 cl017 last message repeated 16 times
> Oct 19 10:49:23 cl017 [NOTICE ] [EVT ] No cluster node data for 192.168.1.19
> Oct 19 10:49:23 cl017 last message repeated 11 times
> Oct 19 10:49:23 cl017 [NOTICE ] [GMI ] EVS recovery of messages complete, transitioning to operational.
> Oct 19 10:49:23 cl017 [NOTICE ] [CLM ] CLM CONFIGURATION CHANGE
> Oct 19 10:49:23 cl017 [NOTICE ] [CLM ] New Configuration:
> Oct 19 10:49:23 cl017 [NOTICE ] [CLM ] ^I192.168.1.8
> Oct 19 10:49:23 cl017 [NOTICE ] [CLM ] ^I192.168.1.17
> Oct 19 10:49:23 cl017 [NOTICE ] [CLM ] ^I192.168.1.18
> Oct 19 10:49:23 cl017 [NOTICE ] [CLM ] ^I192.168.1.19
> Oct 19 10:49:23 cl017 [NOTICE ] [CLM ] Members Left:
> Oct 19 10:49:23 cl017 [NOTICE ] [CLM ] Members Joined:
> Oct 19 10:49:23 cl017 [NOTICE ] [CLM ] ^I192.168.1.8
> Oct 19 10:49:23 cl017 [NOTICE ] [CLM ] got nodejoin message 192.168.1.17
> Oct 19 10:49:23 cl017 [NOTICE ] [CLM ] got nodejoin message 192.168.1.18
> Oct 19 10:49:23 cl017 [NOTICE ] [CLM ] got nodejoin message 192.168.1.19
> Oct 19 10:49:23 cl017 [NOTICE ] [CLM ] got nodejoin message 192.168.1.8
> Oct 19 10:49:23 cl017 [NOTICE ] [GMI ] All services unplugged, unplugging processor
> Oct 19 10:49:23 cl017 [NOTICE ] [EVT ] Channel Chan1, total 2, local 1
> Oct 19 10:49:23 cl017 [NOTICE ] [GMI ] All processors unplugged, allowing messages to be transmitted.
>
> Back to normal again.....
>
>
More information about the Openais
mailing list