defect 1170 - assert in memb_state_recover_enter (was) Re: [Openais] looks like the synchronization code is still broke

Steven Dake sdake at redhat.com
Thu Apr 6 18:37:43 PDT 2006


Fabien,

I have seen a problem like this before.  Basically what happens is that
the recovery phase sends packets a little larger then the netmtu
configured size.  It does this because it sends the original mtu sized
packet plus a header telling the protocol to order the message as if it
were on the old ring (called encapsulation).  In a perfect world, the IP
layer will fragment these packets properly.  This fragmentation is
expensive, so we try to do it in the totem protocol to avoid further
fragmentation, but for the recovery phase no fragmentation can be done.

Some network drivers do not properly set their mtu values so the IP
layer doesn't fragment the packet.  Then other devices on the network
reject the packets as being too big for their incoming hardware buffer.
This FAILEDTORECEIVE error can also happen if your netmtu in the config
file is set to 1500, but for example your device has a MTU of 9000.  In
this condition, the packet will also not fragment in the UDP/IP layer of
the mtu9000 sender, and then the receiver will reject the packet because
it is larger then 1528 bytes.

Try setting the netmtu to 1400 on all processors and see if that fixes
the problem.  Also verify that no processor has an ifconfiged MTU
greater then 1500.

It could also be a bug in the protocol, so please report back if the
netmtu change solves your problem.  I'm not entirely satisfied with the
encapsulation mechanism.

Regards
-steve


On Thu, 2006-04-06 at 18:35 +0200, Fabien THOMAS wrote:
> just killing one node and sometimes one other node crash
> 
> i've replaced exit(1) with  assert(0) and here the conf file, the  
> core and the log file:
> 
> conf file:
> 
> totem {
>      version: 1
>      secauth: on
>      bindnetaddr: 10.2.0.0
>      mcastaddr: 226.94.1.3
>      mcastport: 5405
>      netmtu: 1500
>      threads: 0
>      heartbeat_failures_allowed: 3
>      max_network_delay: 50
> }
> 
> logging {
>      logoutput: file
>      logfile: /log/aisexec.log
>      debug: off
>      timestamp: on
> }
> 
> amf {
>      mode: disabled
> }
> 
> ...
> Apr 06 16:19:26 [NOTICE  ] [SYNC ] Synchronization barrier completed
> Apr 06 16:19:26 [NOTICE  ] [SYNC ] Synchronization actions starting  
> for (openais cluster membership service B.01.01)
> Apr 06 16:19:26 [NOTICE  ] [SYNC ] Synchronization actions done for  
> (openais cluster membership service B.01.01)
> Apr 06 16:19:26 [NOTICE  ] [CLM  ] got nodejoin message 10.2.1.7
> Apr 06 16:19:26 [NOTICE  ] [CLM  ] got nodejoin message 10.2.11.5
> Apr 06 16:19:26 [NOTICE  ] [SYNC ] Synchronization barrier completed
> Apr 06 16:19:26 [NOTICE  ] [SYNC ] Synchronization actions starting  
> for (openais checkpoint service B.01.01)
> Apr 06 16:19:26 [NOTICE  ] [SYNC ] Synchronization actions done for  
> (openais checkpoint service B.01.01)
> Apr 06 16:19:26 [NOTICE  ] [SYNC ] Synchronization barrier completed
> Apr 06 16:19:26 [NOTICE  ] [SYNC ] Synchronization actions starting  
> for (openais event service B.01.01)
> Apr 06 16:19:26 [NOTICE  ] [SYNC ] Synchronization actions done for  
> (openais event service B.01.01)
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] entering GATHER state.
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] entering GATHER state.
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] entering GATHER state.
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Creating commit token because I am  
> the rep.
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Saving state aru 5d high seq  
> received 5d
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Storing new sequence id for ring  
> 85592
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] entering COMMIT state.
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] entering RECOVERY state.
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] position [0] member 10.2.1.7:
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] previous ring seq 85588 rep 10.2.1.7
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] aru 5d high delivered 5d received  
> flag 0
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] position [1] member 10.2.11.5:
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] previous ring seq 85588 rep 10.2.1.7
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] aru 5d high delivered 5d received  
> flag 0
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] copying all old ring messages from  
> 5e-5d.
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Originated 0 messages in RECOVERY.
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Originated for recovery:
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Not Originated for recovery:
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Sending initial ORF token
> Apr 06 16:19:26 [NOTICE  ] [CLM  ] CLM CONFIGURATION CHANGE
> Apr 06 16:19:26 [NOTICE  ] [CLM  ] New Configuration:
> Apr 06 16:19:26 [NOTICE  ] [CLM  ]      10.2.1.7
> Apr 06 16:19:26 [NOTICE  ] [CLM  ]      10.2.11.5
> Apr 06 16:19:26 [NOTICE  ] [CLM  ] Members Left:
> Apr 06 16:19:26 [NOTICE  ] [CLM  ] Members Joined:
> Apr 06 16:19:26 [NOTICE  ] [CLM  ] CLM CONFIGURATION CHANGE
> Apr 06 16:19:26 [NOTICE  ] [CLM  ] New Configuration:
> Apr 06 16:19:26 [NOTICE  ] [CLM  ]      10.2.1.7
> Apr 06 16:19:26 [NOTICE  ] [CLM  ]      10.2.11.5
> Apr 06 16:19:26 [NOTICE  ] [CLM  ] Members Left:
> Apr 06 16:19:26 [NOTICE  ] [CLM  ] Members Joined:
> Apr 06 16:19:26 [NOTICE  ] [SYNC ] This node is within the non- 
> primary component and will NOT provide any services.
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] entering OPERATIONAL state.
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] entering GATHER state.
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Saving state aru 43 high seq  
> received 43
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Storing new sequence id for ring  
> 85596
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] entering COMMIT state.
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] entering RECOVERY state.
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] position [0] member 10.2.1.6:
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] previous ring seq 85592 rep 10.2.1.6
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] aru 11 high delivered 0 received  
> flag 0
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] position [1] member 10.2.1.7:
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] previous ring seq 85592 rep 10.2.1.7
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] aru 43 high delivered 4 received  
> flag 0
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] position [2] member 10.2.11.5:
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] previous ring seq 85592 rep 10.2.1.7
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] aru 45 high delivered 11 received  
> flag 0
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] position [3] member 10.2.20.254:
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] previous ring seq 85592 rep 10.2.1.6
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] aru 11 high delivered 0 received  
> flag 0
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] copying all old ring messages from  
> 44-43.
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Originated 0 messages in RECOVERY.
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Originated for recovery:
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Not Originated for recovery:
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:26 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:27 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:27 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:27 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:27 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:27 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:27 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:27 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:27 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:27 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:27 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:27 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:27 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:27 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:28 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:28 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:28 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:28 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:28 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:28 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:29 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:29 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:29 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:29 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:29 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:30 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:30 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:30 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:30 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:30 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:31 [NOTICE  ] [TOTEM] Retransmit List: 1
> Apr 06 16:19:31 [ERROR   ] [TOTEM] FAILED TO RECEIVE
> 
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and  
> you are
> welcome to change it and/or distribute copies of it under certain  
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for  
> details.
> This GDB was configured as "i386-marcel-freebsd"...
> Core was generated by `aisexec'.
> Program terminated with signal 6, Aborted.
> Reading symbols from /usr/lib/libpthread.so.2...done.
> Loaded symbols for /usr/lib/libpthread.so.2
> Reading symbols from /lib/libc.so.6...done.
> Loaded symbols for /lib/libc.so.6
> Reading symbols from /libexec/ld-elf.so.1...done.
> Loaded symbols for /libexec/ld-elf.so.1
> #0  0x28187723 in kill () from /lib/libc.so.6
> [New LWP 100103]
> (gdb) bt
> #0  0x28187723 in kill () from /lib/libc.so.6
> #1  0x280b61da in raise () from /usr/lib/libpthread.so.2
> #2  0x281863d4 in abort () from /lib/libc.so.6
> #3  0x28164358 in __assert () from /lib/libc.so.6
> #4  0x0805459f in message_handler_orf_token (instance=0x83c6000,
>      system_from=0x3fbfeb90, msg=0x83e2650, msg_len=1108,
>      endian_conversion_needed=0) at totemsrp.c:2947
> #5  0x08056dcd in main_deliver_fn (context=0x83c6000,  
> system_from=0x3fbfeb90,
>      msg=0x83e2650, msg_len=1108) at totemsrp.c:3721
> #6  0x0804e193 in active_token_recv (instance=0x83b4700, interface_no=0,
>      context=0x83c6000, system_from=0x3fbfeb90, msg=0x83e2650,  
> msg_len=1108,
>      token_seqid=36) at totemrrp.c:477
> #7  0x0804e296 in rrp_deliver_fn (context=0x83b5670,  
> system_from=0x3fbfeb90,
>      msg=0x83e2650, msg_len=1108) at totemrrp.c:537
> #8  0x0804c3b6 in net_deliver_fn (handle=0, fd=8, revents=1,  
> data=0x83e2000,
>      prio=0x83b4a14) at totemnet.c:687
> #9  0x0804ab76 in poll_run (handle=0) at aispoll.c:424
> #10 0x0805fdb7 in main (argc=1, argv=0x3fbfee88) at main.c:1317
> (gdb) quit
> F200XA105910400601>gdb aisexec aisexec-2.core
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and  
> you are
> welcome to change it and/or distribute copies of it under certain  
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for  
> details.
> This GDB was configured as "i386-marcel-freebsd"...
> Core was generated by `aisexec'.
> Program terminated with signal 6, Aborted.
> Reading symbols from /usr/lib/libpthread.so.2...done.
> Loaded symbols for /usr/lib/libpthread.so.2
> Reading symbols from /lib/libc.so.6...done.
> Loaded symbols for /lib/libc.so.6
> Reading symbols from /libexec/ld-elf.so.1...done.
> Loaded symbols for /libexec/ld-elf.so.1
> #0  0x28187723 in kill () from /lib/libc.so.6
> [New LWP 100048]
> (gdb) bt
> #0  0x28187723 in kill () from /lib/libc.so.6
> #1  0x280b61da in raise () from /usr/lib/libpthread.so.2
> #2  0x281863d4 in abort () from /lib/libc.so.6
> #3  0x28164358 in __assert () from /lib/libc.so.6
> #4  0x0805459f in message_handler_orf_token (instance=0x83c6000,
>      system_from=0x3fbfeb90, msg=0x83e2650, msg_len=122,
>      endian_conversion_needed=0) at totemsrp.c:2947
> #5  0x08056dcd in main_deliver_fn (context=0x83c6000,  
> system_from=0x3fbfeb90,
>      msg=0x83e2650, msg_len=122) at totemsrp.c:3721
> #6  0x0804e193 in active_token_recv (instance=0x83b4700, interface_no=0,
>      context=0x83c6000, system_from=0x3fbfeb90, msg=0x83e2650,  
> msg_len=122,
>      token_seqid=2) at totemrrp.c:477
> #7  0x0804e296 in rrp_deliver_fn (context=0x83b5670,  
> system_from=0x3fbfeb90,
>      msg=0x83e2650, msg_len=122) at totemrrp.c:537
> #8  0x0804c3b6 in net_deliver_fn (handle=0, fd=8, revents=1,  
> data=0x83e2000,
>      prio=0x84b7694) at totemnet.c:687
> #9  0x0804ab76 in poll_run (handle=0) at aispoll.c:424
> #10 0x0805fdb7 in main (argc=1, argv=0x3fbfee88) at main.c:1317
> (gdb) frame 4
> #4  0x0805459f in message_handler_orf_token (instance=0x83c6000,
>      system_from=0x3fbfeb90, msg=0x83e2650, msg_len=122,
>      endian_conversion_needed=0) at totemsrp.c:2947
> 2947    totemsrp.c: No such file or directory.
>          in totemsrp.c
> (gdb) print *instance
> $1 = {first_run = 1, fcc_remcast_last = 0, fcc_mcast_last = 0,
>    fcc_mcast_current = 0, fcc_remcast_current = 1, consensus_list =  
> {{addr = {
>          nodeid = 117506570, family = 2,
>          addr = "\n\002\001\a??k?\022(?r>\b\220?"}, set = 1}, {addr = {
>          nodeid = 4262724106, family = 2,
>          addr = "\n\002\024???k?\022(?r>\b\220?"}, set = 1}, {addr = {
>          nodeid = 84607498, family = 2,
>          addr = "\n\002\v\005??k?\022(?r>\b\220?"}, set = 1}, {addr = {
>          nodeid = 100729354, family = 2,
>          addr = "\n\002\001\006??C?\005\bL1;\b?_"}, set = 1}, {addr = {
>          nodeid = 0, family = 0, addr = '\0' <repeats 15 times>},
>        set = 0} <repeats 28 times>}, consensus_list_entries = 4,
>    my_proc_list = {{nodeid = 117506570, family = 2,
>        addr = "\n\002\001\a", '\0' <repeats 11 times>}, {nodeid =  
> 84607498,
>        family = 2, addr = "\n\002\v\005", '\0' <repeats 11 times>}, {
>        nodeid = 100729354, family = 2,
>        addr = "\n\002\001\006", '\0' <repeats 11 times>}, {nodeid =  
> 4262724106,
>        family = 2, addr = "\n\002\024?", '\0' <repeats 11 times>},  
> {nodeid = 0,
>        family = 0, addr = '\0' <repeats 15 times>} <repeats 28 times>},
>    my_failed_list = {{nodeid = 100729354, family = 2,
>        addr = "\n\002\001\006??k?\022(0\216>\b\220?"}, {nodeid =  
> 4262724106,
>        family = 2, addr = "\n\002\024???C?\005\bL1;\b?_"}, {nodeid =  
> 84607498,
>        family = 2, addr = "\n\002\v\005", '\0' <repeats 11 times>}, {
>        nodeid = 0, family = 0,
> ---Type <return> to continue, or q <return> to quit---
>        addr = '\0' <repeats 15 times>} <repeats 29 times>},  
> my_new_memb_list = {
>      {nodeid = 100729354, family = 2,
>        addr = "\n\002\001\006", '\0' <repeats 11 times>}, {nodeid =  
> 117506570,
>        family = 2, addr = "\n\002\001\a", '\0' <repeats 11 times>}, {
>        nodeid = 84607498, family = 2,
>        addr = "\n\002\v\005??C?\005\bL1;\b\2004"}, {nodeid = 4262724106,
>        family = 2, addr = "\n\002\024?", '\0' <repeats 11 times>},  
> {nodeid = 0,
>        family = 0, addr = '\0' <repeats 15 times>} <repeats 28 times>},
>    my_trans_memb_list = {{nodeid = 117506570, family = 2,
>        addr = "\n\002\001\a", '\0' <repeats 11 times>}, {nodeid =  
> 84607498,
>        family = 2, addr = "\n\002\v\005??C?\005\bL1;\b\2004"},  
> {nodeid = 0,
>        family = 0, addr = '\0' <repeats 15 times>} <repeats 30 times>},
>    my_memb_list = {{nodeid = 117506570, family = 2,
>        addr = "\n\002\001\a", '\0' <repeats 11 times>}, {nodeid =  
> 84607498,
>        family = 2, addr = "\n\002\v\005", '\0' <repeats 11 times>}, {
>        nodeid = 0, family = 0,
>        addr = '\0' <repeats 15 times>} <repeats 30 times>},
>    my_deliver_memb_list = {{nodeid = 117506570, family = 2,
>        addr = "\n\002\001\a", '\0' <repeats 11 times>}, {nodeid =  
> 84607498,
>        family = 2, addr = "\n\002\v\005??C?\005\bL1;\b\2004"},  
> {nodeid = 0,
>        family = 0, addr = '\0' <repeats 15 times>} <repeats 30 times>},
>    my_nodeid_lookup_list = {{nodeid = 117506570, family = 2,
>        addr = "\n\002\001\a", '\0' <repeats 11 times>}, {nodeid =  
> 84607498,
>        family = 2, addr = "\n\002\v\005", '\0' <repeats 11 times>}, {
> ---Type <return> to continue, or q <return> to quit---
>        nodeid = 100729354, family = 2,
>        addr = "\n\002\001\006", '\0' <repeats 11 times>}, {nodeid =  
> 4262724106,
>        family = 2, addr = "\n\002\024?", '\0' <repeats 11 times>},  
> {nodeid = 0,
>        family = 0, addr = '\0' <repeats 15 times>} <repeats 28 times>},
>    my_proc_list_entries = 4, my_failed_list_entries = 0,
>    my_new_memb_entries = 4, my_trans_memb_entries = 2,  
> my_memb_entries = 2,
>    my_deliver_memb_entries = 2, my_nodeid_lookup_entries = 4,  
> my_ring_id = {
>      rep = {nodeid = 100729354, family = 2,
>        addr = "\n\002\001\006", '\0' <repeats 11 times>}, seq = 85596},
>    my_old_ring_id = {rep = {nodeid = 117506570, family = 2,
>        addr = "\n\002\001\a", '\0' <repeats 11 times>}, seq = 85592},
>    my_aru_count = 51, my_merge_detect_timeout_outstanding = 1,  
> my_last_aru = 0,
>    my_seq_unchanged = 51, my_received_flg = 0, my_high_seq_received = 2,
>    my_install_seq = 0, my_rotation_counter = 0, my_set_retrans_flg = 0,
>    my_retrans_flg_count = 0, my_high_ring_delivered = 17,
>    heartbeat_timeout = 764, new_message_queue = {head = 16, tail = 14,
>      used = 1, usedhw = 37, size = 181, items = 0x83e7000,  
> size_per_item = 48,
>      iterator = 0}, retrans_message_queue = {head = 0, tail = 499,  
> used = 0,
>      usedhw = 0, size = 500, items = 0x83ce000, size_per_item = 48,
>      iterator = 0}, regular_sort_queue = {head = 0, size = 256,
>      items = 0x83d4000, items_inuse = 0x83c0c00, size_per_item = 44,
>      head_seqid = 0, item_count = 256, pos_max = 67},  
> recovery_sort_queue = {
>      head = 0, size = 256, items = 0x83d7000, items_inuse = 0x83c4000,
>      size_per_item = 44, head_seqid = 0, item_count = 256, pos_max = 2},
> ---Type <return> to continue, or q <return> to quit---
>    my_aru = 2, my_high_delivered = 4,  
> token_callback_received_listhead = {
>      next = 0x83b3440, prev = 0x83b3440},  
> token_callback_sent_listhead = {
>      next = 0x83c77f0, prev = 0x83c77f0}, orf_token_retransmit =  
> 0x83ca000 "",
>    orf_token_retransmit_size = 88, my_token_seq = 204,
>    timer_orf_token_timeout = 0x8749500,
>    timer_orf_token_retransmit_timeout = 0x87495c0,
>    timer_orf_token_hold_retransmit_timeout = 0x0,
>    timer_merge_detect_timeout = 0x87495e0,
>    memb_timer_state_gather_join_timeout = 0x0,
>    memb_timer_state_gather_consensus_timeout = 0x0,
>    memb_timer_state_commit_timeout = 0x0, timer_heartbeat_timeout =  
> 0x87490c0,
>    totemsrp_log_level_security = 65538, totemsrp_log_level_error =  
> 131074,
>    totemsrp_log_level_warning = 196610, totemsrp_log_level_notice =  
> 262146,
>    totemsrp_log_level_debug = 327682,
>    totemsrp_log_printf = 0x805ff68 <internal_log_printf>,
>    memb_state = MEMB_STATE_RECOVERY, my_id = {nodeid = 117506570,  
> family = 2,
>      addr = "\n\002\001\a", '\0' <repeats 11 times>}, next_memb = {
>      nodeid = 84607498, family = 2, addr = "\n\002\v\005??C?\005\bL1; 
> \b\2004"},
>    iov_buffer = '\0' <repeats 8999 times>, totemsrp_iov_recv =  
> {iov_base = 0x0,
>      iov_len = 0}, totemsrp_poll_handle = 0, totemsrp_recv = 0,
>    mcast_address = {nodeid = 0, family = 2,
>      addr = "?^\001\003", '\0' <repeats 11 times>},
>    totemsrp_deliver_fn = 0x8056ee4 <totemmrp_deliver_fn>,
>    totemsrp_confchg_fn = 0x8056f18 <totemmrp_confchg_fn>,  
> global_seqno = 1463,
> ---Type <return> to continue, or q <return> to quit---
>    my_token_held = 1, token_ring_id_seq = 85596, last_released = 0,
>    set_aru = 4294967295, old_ring_state_saved = 1, old_ring_state_aru  
> = 67,
>    old_ring_state_high_seq_received = 67, ring_saved = 1, my_last_seq  
> = 2,
>    tv_old = {tv_sec = 0, tv_usec = 0}, totemrrp_handle = 0,
>    totem_config = 0x3fbfed14, use_heartbeat = 1, my_trc = 1, my_pbl = 0}
> (gdb) print *token
> $2 = {header = {type = 0 '\0', encapsulated = 0 '\0', endian_detector  
> = 65314,
>      nodeid = 4262724106}, seq = 2, token_seq = 208, aru = 0,  
> aru_addr = {
>      nodeid = 100729354, family = 2,
>      addr = "\n\002\001\006", '\0' <repeats 11 times>}, ring_id =  
> {rep = {
>        nodeid = 100729354, family = 2,
>        addr = "\n\002\001\006", '\0' <repeats 11 times>}, seq = 85596},
>    backlog = 0, fcc = 1, retrans_flg = 1, rtr_list_entries = 0,
>    rtr_list = 0x3fbfe518}
> (gdb)
> 
> core saved under aisexec.ml2
> 
> 
> 
> 




More information about the Openais mailing list