[Openais] proposal for better end to end flow control

Steven Dake sdake at redhat.com
Wed Mar 31 15:55:29 PDT 2010


On Thu, 2010-04-01 at 11:23 +1300, Tim Beale wrote:
> Hi Steve,
> 
> End-to-end flow control is something I'd really love to see. It sounds like
> your proposal won't fix all the problems we're seeing with flow control though.
> 
> A problem we've seen is kind of permanent congestion - the receiver gets a
> burst of several hundred CPG messages queued up and never really recovers. The
> sender continues sending enough CPG messages that the receiver never clears out
> its queue, but doesn't run out of memory either. The receiver's queue could
> hover in this state indefinitely. On our setup, a healthcheck mechanism detects
> the receiver has locked up (some operations are blocking due to flow control
> congestion) and eventually restarts the process.
> (As an interim workaround for this on our setup, I fudged the token backlog
> calculation to gradually force the sender to backoff, so the sender's totem
> message queue fills up and it starts getting TRY_AGAIN errors).
> 
> I was wondering whether end-to-end flow control at the CPG group level is a
> possible/feasible option that'd solve both this case and the oom one? E.g. in
> the CPG library code it sends an internal message to notify the rest of the CPG
> group whenever the flow control status for an application changes?
> 

Tim,

Also more directly addressing this last point, I have tried that in the
past and it didn't operate as I would expect, but alot of things have
changed since I tried (openais 0.80.2ish... which used a different
non-shared memory ipc system).  Likely worth some more investigation.

In the case of memory overusage on one node, lib user gets try again on
all nodes
if that hasn't happened, but cpg flow control is active, lib user gets
try again on all nodes

Double flow control features (one based on protecting against oom, one
to protect the individual cpg group).

Regards
-steve

> Regards,
> Tim
> _______________________________________________
> Openais mailing list
> Openais at lists.linux-foundation.org
> https://lists.linux-foundation.org/mailman/listinfo/openais



More information about the Openais mailing list