[Openais] constant "CLM CONFIGURATION CHANGE" and "Retransmit List" in log
Joseph Salvaggio
jsalvaggio at norwoodma.gov
Mon Mar 8 06:58:03 PST 2010
We have a 2 node Red Hat cluster. Apache service and samba service, for which we keep the GFS2 SAN partition always mounted on both nodes. For the last few days been seeing "Retransmit List" a lot (over and over, every 1-2 minutes or so) in our "messages" log
What problem(s) is this possibly pointing to?
Thank you.
from /var/log/messages
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] The token was lost in the OPERATIONAL state.
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Receive multicast socket recv buffer size (320000 bytes).
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] entering GATHER state from 2.
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Creating commit token because I am the rep.
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Saving state aru c136 high seq received c136
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Storing new sequence id for ring 48b80
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] entering COMMIT state.
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] entering RECOVERY state.
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] position [0] member 192.168.100.7:
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] previous ring seq 297852 rep 192.168.100.7
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] aru c136 high delivered c136 received flag 1
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] position [1] member 192.168.100.10:
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] previous ring seq 297852 rep 192.168.100.7
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] aru c136 high delivered c136 received flag 1
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Did not need to originate any messages in recovery.
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Sending initial ORF token
Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] CLM CONFIGURATION CHANGE
Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] New Configuration:
Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] r(0) ip(192.168.100.7)
Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] r(0) ip(192.168.100.10)
Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] Members Left:
Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] Members Joined:
Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] CLM CONFIGURATION CHANGE
Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] New Configuration:
Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] r(0) ip(192.168.100.7)
Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] r(0) ip(192.168.100.10)
Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] Members Left:
Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] Members Joined:
Mar 8 08:03:42 kaf-gimel openais[3973]: [SYNC ] This node is within the primary component and will provide service.
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] entering OPERATIONAL state.
Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] got nodejoin message 192.168.100.7
Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] got nodejoin message 192.168.100.10
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 22 23 24 25 26 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 20 21 18 19 1a 1b 1c 1d 1e 1f 22 23 24 25 26
Mar 8 08:03:42 kaf-gimel gfs_controld[4004]: cpg_mcast_joined retry 100 MSG_PLOCK
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 1b 1c 1d 1e 1f 20 21 22 23 24 25 26
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 1f 20 21 22 23 24 25 26
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 20 21 22 23 24 25 26
Mar 8 08:03:42 kaf-gimel last message repeated 4 times
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 22 23 24 25 26
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 23 24 25 26
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 24 25 26
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 25 26
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 28 29
Mar 8 08:03:42 kaf-gimel last message repeated 34 times
Mar 8 08:03:42 kaf-gimel gfs_controld[4004]: cpg_mcast_joined retry 200 MSG_PLOCK
Mar 8 08:03:42 kaf-gimel openais[3973]: [CPG ] got joinlist message from node 1
Mar 8 08:03:42 kaf-gimel openais[3973]: [CPG ] got joinlist message from node 2
Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 32
Mar 8 08:40:46 kaf-gimel kernel: <=64 ID=6 ACK URGP=0
Mar 8 08:40:46 kaf-gimel kernel: <710.0.100.21 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=41685 DF PROTO=TCP SPT=25 DPT=58398 WINDOW=64 RES=0x00 ACK URGP=0
Mar 8 08:55:55 kaf-gimel kernel: BANDWIDTH_IN:IN=eth0 OUT= MAC=00:1b:21:46:fe:06:00:30:48:33:ad:b4:08:00 SRC=10.0.100.21 DST=10.0.100.23 LEN=52 TOS=0x00 PREC=0x00 TTL=64 IT55 WIS=0x00BANDWIDTH_IN:IN=eth0 OUT= MAC=00:1b:21:46:fe:06:00:30:48:33:ad:b4:08:00 SRC=10.0.100.21 DST=10.0.100.23 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=40663 DF PROTO=TCP SPT=25 DPT=39615 WINDOW=46336 RES=0x00 ACK URGP=0
Mar 8 08:55:55 kaf-gimel kernel: WIDTH_IN:IN=eth0OUT=b:48:33:ad:b4:08:00 SRC=10.0.100.21 DST=10.0.100.23 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=41095 DF PROTO=TCP SPT=25 DPT=39615 WINDOW=47784 RES=0x00 ACK URGP=0
Mar 8 09:00:48 kaf-gimel syslogd 1.4.1: restart.
Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] The token was lost in the OPERATIONAL state.
Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Receive multicast socket recv buffer size (320000 bytes).
Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] entering GATHER state from 2.
Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Creating commit token because I am the rep.
Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Saving state aru 142c1 high seq received 142c1
Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Storing new sequence id for ring 48b84
Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] entering COMMIT state.
Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] entering RECOVERY state.
Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] position [0] member 192.168.100.7:
Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] previous ring seq 297856 rep 192.168.100.7
Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] aru 142c1 high delivered 142c1 received flag 1
Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] position [1] member 192.168.100.10:
Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] previous ring seq 297856 rep 192.168.100.7
Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] aru 142c1 high delivered 142c1 received flag 1
Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Did not need to originate any messages in recovery.
Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Sending initial ORF token
Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] CLM CONFIGURATION CHANGE
Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] New Configuration:
Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] r(0) ip(192.168.100.7)
Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] r(0) ip(192.168.100.10)
Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] Members Left:
Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] Members Joined:
Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] CLM CONFIGURATION CHANGE
Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] New Configuration:
Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] r(0) ip(192.168.100.7)
Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] r(0) ip(192.168.100.10)
Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] Members Left:
Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] Members Joined:
Mar 8 09:03:12 kaf-gimel openais[3973]: [SYNC ] This node is within the primary component and will provide service.
Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] entering OPERATIONAL state.
Mar 8 09:03:13 kaf-gimel gfs_controld[4004]: cpg_mcast_joined retry 100 MSG_PLOCK
Mar 8 09:03:13 kaf-gimel openais[3973]: [CLM ] got nodejoin message 192.168.100.7
Mar 8 09:03:13 kaf-gimel openais[3973]: [CLM ] got nodejoin message 192.168.100.10
Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28
Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 26 27 28 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25
Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28
Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 1d 1e 1f 20 21 22 23 24 25 26 27 28
Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 20 21 22 23 24 25 26 27 28
Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 23 24 25 26 27 28
Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 25 26 27 28
Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 25 26 27 28
Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 27 28
Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 2a 2b
Mar 8 09:03:13 kaf-gimel last message repeated 34 times
Mar 8 09:03:13 kaf-gimel openais[3973]: [CPG ] got joinlist message from node 1
Mar 8 09:03:13 kaf-gimel openais[3973]: [CPG ] got joinlist message from node 2
Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 34
Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] The token was lost in the OPERATIONAL state.
Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Receive multicast socket recv buffer size (320000 bytes).
Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] entering GATHER state from 2.
Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Creating commit token because I am the rep.
Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Saving state aru 1c89 high seq received 1c89
Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Storing new sequence id for ring 48b88
Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] entering COMMIT state.
Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] entering RECOVERY state.
Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] position [0] member 192.168.100.7:
Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] previous ring seq 297860 rep 192.168.100.7
Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] aru 1c89 high delivered 1c89 received flag 1
Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] position [1] member 192.168.100.10:
Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] previous ring seq 297860 rep 192.168.100.7
Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] aru 1c89 high delivered 1c89 received flag 1
Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Did not need to originate any messages in recovery.
Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Sending initial ORF token
Mar 8 09:04:41 kaf-gimel openais[3973]: [CLM ] CLM CONFIGURATION CHANGE
Mar 8 09:04:41 kaf-gimel openais[3973]: [CLM ] New Configuration:
Mar 8 09:04:41 kaf-gimel openais[3973]: [CLM ] r(0) ip(192.168.100.7)
Mar 8 09:04:41 kaf-gimel openais[3973]: [CLM ] r(0) ip(192.168.100.10)
Mar 8 09:04:41 kaf-gimel gfs_controld[4004]: cpg_mcast_joined retry 100 MSG_PLOCK
Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] Members Left:
Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] Members Joined:
Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] CLM CONFIGURATION CHANGE
Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] New Configuration:
Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] r(0) ip(192.168.100.7)
Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] r(0) ip(192.168.100.10)
Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] Members Left:
Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] Members Joined:
Mar 8 09:04:42 kaf-gimel openais[3973]: [SYNC ] This node is within the primary component and will provide service.
Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] entering OPERATIONAL state.
Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] got nodejoin message 192.168.100.7
Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] got nodejoin message 192.168.100.10
Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27
Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 23 24 25 26 27 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22
Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 22 1a 1b 1c 1d 1e 1f 20 21 23 24 25 26 27
Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 1e 1f 20 21 22 23 24 25 26 27
Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 21 22 23 24 25 26 27
Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 21 22 23 24 25 26 27
Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 22 23 24 25 26 27
Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 25 26 27
Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 26 27
Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 29 2a
Mar 8 09:04:42 kaf-gimel last message repeated 34 times
Mar 8 09:04:42 kaf-gimel openais[3973]: [CPG ] got joinlist message from node 1
Mar 8 09:04:42 kaf-gimel openais[3973]: [CPG ] got joinlist message from node 2
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.linux-foundation.org/pipermail/openais/attachments/20100308/7bf9087b/attachment-0001.htm
More information about the Openais
mailing list