[Openais] constant "CLM CONFIGURATION CHANGE" and "Retransmit List" in log
Steven Dake
sdake at redhat.com
Mon Mar 8 08:26:04 PST 2010
On Mon, 2010-03-08 at 09:58 -0500, Joseph Salvaggio wrote:
> We have a 2 node Red Hat cluster. Apache service and samba service,
> for which we keep the GFS2 SAN partition always mounted on both nodes.
> For the last few days been seeing "Retransmit List" a lot (over and
> over, every 1-2 minutes or so) in our "messages" log
>
> What problem(s) is this possibly pointing to?
>
> Thank you.
>
>
> from /var/log/messages
>
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] The token was lost in
> the OPERATIONAL state.
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Receive multicast
> socket recv buffer size (320000 bytes).
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Transmit multicast
> socket send buffer size (262142 bytes).
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] entering GATHER state
> from 2.
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Creating commit token
> because I am the rep.
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Saving state aru c136
> high seq received c136
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Storing new sequence
> id for ring 48b80
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] entering COMMIT
> state.
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] entering RECOVERY
> state.
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] position [0] member
> 192.168.100.7:
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] previous ring seq
> 297852 rep 192.168.100.7
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] aru c136 high
> delivered c136 received flag 1
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] position [1] member
> 192.168.100.10:
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] previous ring seq
> 297852 rep 192.168.100.7
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] aru c136 high
> delivered c136 received flag 1
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Did not need to
> originate any messages in recovery.
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Sending initial ORF
> token
> Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] CLM CONFIGURATION
> CHANGE
> Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] New Configuration:
> Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] r(0)
> ip(192.168.100.7)
> Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] r(0)
> ip(192.168.100.10)
> Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] Members Left:
> Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] Members Joined:
> Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] CLM CONFIGURATION
> CHANGE
> Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] New Configuration:
> Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] r(0)
> ip(192.168.100.7)
> Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] r(0)
> ip(192.168.100.10)
> Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] Members Left:
> Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] Members Joined:
> Mar 8 08:03:42 kaf-gimel openais[3973]: [SYNC ] This node is within
> the primary component and will provide service.
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] entering OPERATIONAL
> state.
> Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] got nodejoin message
> 192.168.100.7
> Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] got nodejoin message
> 192.168.100.10
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 11
> 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 22
> 23 24 25 26 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 20
> 21 18 19 1a 1b 1c 1d 1e 1f 22 23 24 25 26
> Mar 8 08:03:42 kaf-gimel gfs_controld[4004]: cpg_mcast_joined retry
> 100 MSG_PLOCK
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 1b
> 1c 1d 1e 1f 20 21 22 23 24 25 26
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 1f
> 20 21 22 23 24 25 26
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 20
> 21 22 23 24 25 26
> Mar 8 08:03:42 kaf-gimel last message repeated 4 times
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 22
> 23 24 25 26
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 23
> 24 25 26
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 24
> 25 26
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 25
> 26
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 28
> 29
> Mar 8 08:03:42 kaf-gimel last message repeated 34 times
> Mar 8 08:03:42 kaf-gimel gfs_controld[4004]: cpg_mcast_joined retry
> 200 MSG_PLOCK
> Mar 8 08:03:42 kaf-gimel openais[3973]: [CPG ] got joinlist message
> from node 1
> Mar 8 08:03:42 kaf-gimel openais[3973]: [CPG ] got joinlist message
> from node 2
> Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 32
> Mar 8 08:40:46 kaf-gimel kernel: <=64 ID=6 ACK URGP=0
> Mar 8 08:40:46 kaf-gimel kernel: <710.0.100.21 LEN=52 TOS=0x00
> PREC=0x00 TTL=64 ID=41685 DF PROTO=TCP SPT=25 DPT=58398 WINDOW=64
> RES=0x00 ACK URGP=0
> Mar 8 08:55:55 kaf-gimel kernel: BANDWIDTH_IN:IN=eth0 OUT=
> MAC=00:1b:21:46:fe:06:00:30:48:33:ad:b4:08:00 SRC=10.0.100.21
> DST=10.0.100.23 LEN=52 TOS=0x00 PREC=0x00 TTL=64 IT55
> WIS=0x00BANDWIDTH_IN:IN=eth0 OUT=
> MAC=00:1b:21:46:fe:06:00:30:48:33:ad:b4:08:00 SRC=10.0.100.21
> DST=10.0.100.23 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=40663 DF PROTO=TCP
> SPT=25 DPT=39615 WINDOW=46336 RES=0x00 ACK URGP=0
> Mar 8 08:55:55 kaf-gimel kernel:
> WIDTH_IN:IN=eth0OUT=b:48:33:ad:b4:08:00 SRC=10.0.100.21
> DST=10.0.100.23 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=41095 DF PROTO=TCP
> SPT=25 DPT=39615 WINDOW=47784 RES=0x00 ACK URGP=0
> Mar 8 09:00:48 kaf-gimel syslogd 1.4.1: restart.
> Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] The token was lost in
> the OPERATIONAL state.
> Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Receive multicast
> socket recv buffer size (320000 bytes).
> Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Transmit multicast
> socket send buffer size (262142 bytes).
> Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] entering GATHER state
> from 2.
> Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Creating commit token
> because I am the rep.
> Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Saving state aru
> 142c1 high seq received 142c1
> Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Storing new sequence
> id for ring 48b84
> Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] entering COMMIT
> state.
> Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] entering RECOVERY
> state.
> Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] position [0] member
> 192.168.100.7:
> Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] previous ring seq
> 297856 rep 192.168.100.7
> Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] aru 142c1 high
> delivered 142c1 received flag 1
> Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] position [1] member
> 192.168.100.10:
> Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] previous ring seq
> 297856 rep 192.168.100.7
> Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] aru 142c1 high
> delivered 142c1 received flag 1
> Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Did not need to
> originate any messages in recovery.
> Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Sending initial ORF
> token
> Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] CLM CONFIGURATION
> CHANGE
> Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] New Configuration:
> Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] r(0)
> ip(192.168.100.7)
> Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] r(0)
> ip(192.168.100.10)
> Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] Members Left:
> Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] Members Joined:
> Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] CLM CONFIGURATION
> CHANGE
> Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] New Configuration:
> Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] r(0)
> ip(192.168.100.7)
> Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] r(0)
> ip(192.168.100.10)
> Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] Members Left:
> Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] Members Joined:
> Mar 8 09:03:12 kaf-gimel openais[3973]: [SYNC ] This node is within
> the primary component and will provide service.
> Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] entering OPERATIONAL
> state.
> Mar 8 09:03:13 kaf-gimel gfs_controld[4004]: cpg_mcast_joined retry
> 100 MSG_PLOCK
> Mar 8 09:03:13 kaf-gimel openais[3973]: [CLM ] got nodejoin message
> 192.168.100.7
> Mar 8 09:03:13 kaf-gimel openais[3973]: [CLM ] got nodejoin message
> 192.168.100.10
> Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 15
> 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28
> Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 26
> 27 28 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25
> Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 19
> 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28
> Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 1d
> 1e 1f 20 21 22 23 24 25 26 27 28
> Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 20
> 21 22 23 24 25 26 27 28
> Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 23
> 24 25 26 27 28
> Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 25
> 26 27 28
> Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 25
> 26 27 28
> Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 27
> 28
> Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 2a
> 2b
> Mar 8 09:03:13 kaf-gimel last message repeated 34 times
> Mar 8 09:03:13 kaf-gimel openais[3973]: [CPG ] got joinlist message
> from node 1
> Mar 8 09:03:13 kaf-gimel openais[3973]: [CPG ] got joinlist message
> from node 2
> Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 34
> Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] The token was lost in
> the OPERATIONAL state.
> Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Receive multicast
> socket recv buffer size (320000 bytes).
> Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Transmit multicast
> socket send buffer size (262142 bytes).
> Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] entering GATHER state
> from 2.
> Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Creating commit token
> because I am the rep.
> Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Saving state aru 1c89
> high seq received 1c89
> Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Storing new sequence
> id for ring 48b88
> Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] entering COMMIT
> state.
> Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] entering RECOVERY
> state.
> Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] position [0] member
> 192.168.100.7:
> Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] previous ring seq
> 297860 rep 192.168.100.7
> Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] aru 1c89 high
> delivered 1c89 received flag 1
> Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] position [1] member
> 192.168.100.10:
> Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] previous ring seq
> 297860 rep 192.168.100.7
> Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] aru 1c89 high
> delivered 1c89 received flag 1
> Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Did not need to
> originate any messages in recovery.
> Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Sending initial ORF
> token
> Mar 8 09:04:41 kaf-gimel openais[3973]: [CLM ] CLM CONFIGURATION
> CHANGE
> Mar 8 09:04:41 kaf-gimel openais[3973]: [CLM ] New Configuration:
> Mar 8 09:04:41 kaf-gimel openais[3973]: [CLM ] r(0)
> ip(192.168.100.7)
> Mar 8 09:04:41 kaf-gimel openais[3973]: [CLM ] r(0)
> ip(192.168.100.10)
> Mar 8 09:04:41 kaf-gimel gfs_controld[4004]: cpg_mcast_joined retry
> 100 MSG_PLOCK
> Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] Members Left:
> Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] Members Joined:
> Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] CLM CONFIGURATION
> CHANGE
> Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] New Configuration:
> Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] r(0)
> ip(192.168.100.7)
> Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] r(0)
> ip(192.168.100.10)
> Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] Members Left:
> Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] Members Joined:
> Mar 8 09:04:42 kaf-gimel openais[3973]: [SYNC ] This node is within
> the primary component and will provide service.
> Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] entering OPERATIONAL
> state.
> Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] got nodejoin message
> 192.168.100.7
> Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] got nodejoin message
> 192.168.100.10
> Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 12
> 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27
> Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 23
> 24 25 26 27 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22
> Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 22
> 1a 1b 1c 1d 1e 1f 20 21 23 24 25 26 27
> Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 1e
> 1f 20 21 22 23 24 25 26 27
> Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 21
> 22 23 24 25 26 27
> Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 21
> 22 23 24 25 26 27
> Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 22
> 23 24 25 26 27
> Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 25
> 26 27
> Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 26
> 27
> Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 29
> 2a
> Mar 8 09:04:42 kaf-gimel last message repeated 34 times
> Mar 8 09:04:42 kaf-gimel openais[3973]: [CPG ] got joinlist message
> from node 1
> Mar 8 09:04:42 kaf-gimel openais[3973]: [CPG ] got joinlist message
> from node 2
>
>
Are you using bonding?
Regards
-steve
> _______________________________________________
> Openais mailing list
> Openais at lists.linux-foundation.org
> https://lists.linux-foundation.org/mailman/listinfo/openais
More information about the Openais
mailing list