[Openais] [Corosync] Corosync does not retransmit the lost mcast message

[Steven Dake]

On Fri, 2010-03-19 at 16:45 -0600, hj lee wrote:
> Hi Steve,
> 
> I added and changed some log messages, so my log won't match with the
> source tree. Any way I think I found the problem. This issue seems to
> be happening easily where a multicast messages are infrequently sent.
> The problem is the rtr field is filled based on my_high_seq_received!
> It should be set based on token->seq value.
> 

I did notice this inconsistency and was thinking along these lines too,
but I wanted to see your log to see if some other events were occuring
related to oldring_state_save()/restore().  Another possibility is some
sort of misfeeding to or from the regular/recovery queue. (ie do you
have more the this retransmission bug)

> Let's assume very simple case, just one mcast message(seq 77) was lost
> in node2.
> 
> In node1:
> all messages are received up to 77.
> token seq = 77
> my_aru = 77
> my_high_seq_received = 77
> 
> in node2:
> message 77 was lost.
> my_aru = 76
> token seq = 77
> my_high_seq_received = 76
> 
> 
> Once node2 gets into this state, it does not set the rtr filed for the
> lost message 77. Then my_aru_count keeps increasing and the corosync
> enters "FAILED TO RECEIVE" and gather. The totem spec. says clearly if
> token seq is greater than my_aru, it means this processor lost some
> messages, it should set rtr field to request the retransmission.
> 
> The related code is in orf_token_rtr() at totemsrp.c.
> 
> range = instance->my_high_seq_received - instance->my_aru;
> 
> Above line should be changed to 
> 
> range = orf_token->seq - instance->my_aru;
> 

Ya good catch.  More totem experts = win for the community ;)

> What was the reason of introducing my_high_seq_received? The original
> spec does not have this variable.
> 

It does indirectly.  EVS requires that we continue to search through the
list of messages in the sort queue when we lose a message during the
entry to the operational state.  As an example:
n1, n2, n3
n1 sends a seq = 53
n2 sends b seq = 54
n3 sends c seq = 55
n1 receives a, c
n3 receives a, c
n2 fails

In this case we should deliver A, then a transitional configuration,
then skip any messages missing that could not be recovered (because they
are from failed processors specifically message b). Then we should
deliver C.  To deliver C, we need to know the last entry in the sort
queue, and my_high_seq_received is how we know when we have reached that
point (ie how to calculate the range of the queue).


Good catch
-steve

> Thanks
> hj
> 
> 
> On Fri, Mar 19, 2010 at 9:59 AM, Steven Dake <sdake at redhat.com> wrote:
>         can you please attach the logs from the last configuration
>         change until
>         the failure?
>         
>         It would really help me understand the condition so i can
>         generate a
>         reproducer.
>         
>         Thanks
>         -steve
>