[Gluster-devel] Priority based ping packet for 3.10

[Raghavendra Talur]

On Tue, Jan 24, 2017 at 10:39 AM, Vijay Bellur <vbellur at redhat.com> wrote:
>
>
> On Thu, Jan 19, 2017 at 8:06 AM, Jeff Darcy <jdarcy at redhat.com> wrote:
>>
>> > The more relevant question would be with TCP_KEEPALIVE and
>> > TCP_USER_TIMEOUT
>> > on sockets, do we really need ping-pong framework in Clients? We might
>> > need
>> > that in transport/rdma setups, but my question is concentrating on
>> > transport/rdma. In other words would like to hear why do we need
>> > heart-beat
>> > mechanism in the first place. One scenario might be a healthy socket
>> > level
>> > connection but an unhealthy brick/client (like a deadlocked one).
>>
>> This is an important case to consider.  On the one hand, I think it
>> answers
>> your question about TCP_KEEPALIVE.  What we really care about is whether a
>> brick's request queue is moving.  In other words, what's the time since
>> the
>> last reply from that brick, and does that time exceed some threshold?  On
>> a
>> busy system, we don't even need ping packets to know that.  We can just
>> use
>> responses on other requests to set/reset that timer.  We only need to send
>> ping packets when our *outbound* queue has remained empty for some
>> fraction
>> of our timeout.
>>
>> However, it's important that our measurements be *end to end* and not just
>> at the transport level.  This is particularly true with multiplexing,
>> where multiple bricks will share and contend on various resources.  We
>> should ping *through* client and server, with separate translators above
>> and below each.  This would give us a true end-to-end ping *for that
>> brick*, and also keep the code nicely modular.
>
>
> +1 to this. Having ping, pong xlators immediately above and below protocol
> translators would also address the problem of epoll threads getting blocked
> in gluster's xlator stacks in busy systems.

IMO, as Jeff mentioned for ping timeout that it is required only when
there are no operations happening, similarly, we can consider
frame-timeout as a mechanism of *end to end* keep alive. What extra
feature(s) do we get by adding a ping-pong xlator couple?

>
> Having said that, I do see value in Rafi's patch that prompted this thread.
> Would it not help to prioritize ping - pong traffic in all parts of the
> gluster stack including the send queue on the client?
>
> Regards,
> Vijay