<div dir="ltr">The more relevant question would be with TCP_KEEPALIVE and TCP_USER_TIMEOUT on sockets, do we really need ping-pong framework in Clients? We might need that in transport/rdma setups, but my question is concentrating on transport/rdma. In other words would like to hear why do we need heart-beat mechanism in the first place. One scenario might be a healthy socket level connection but an unhealthy brick/client (like a deadlocked one). Are there enough such realistic scenarios which make ping-pong/heartbeat necessary? What other ways brick/client can go bad?<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 19, 2017 at 3:36 PM, Raghavendra G <span dir="ltr"><<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Thu, Jan 19, 2017 at 1:50 PM, Mohammed Rafi K C <span dir="ltr"><<a href="mailto:rkavunga@redhat.com" target="_blank">rkavunga@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
The patch for priority based ping packets [1] are ready to review. As<br>
Shyam mentioned in the comment on patch set 12, it doesn't solve the<br>
problem with network conjunction nor the disk latency. Also it won't<br>
priorities the reply of ping packets at the server end (We don't have a<br>
straight way to identify prognum in the reply).<br>
<br>
<br>
So my question , Is it worth of taking the patch or do we need to think<br>
through a more generic solutions.<br></blockquote><div><br></div></span><div>Though ping requests can take more time to reach server due to heavy traffic, realistically speaking common reasons for ping-timer expiry have been either<br><br>1. client not been able to read ping response [2]<br></div><div>2. server not able to read ping request.<br><br></div><div>Speaking about 2 above, Me, Kritika and Pranith were just discussing today morning about an issue where they had hit ping timer expiry in replicated setups when disk usage was high. The reason for this as Pranith pointed out was,<br></div><div>1. posix has some fops (like posix_xattrop, posix_fxattrop) which do syscalls after holding a lock on inode (inode->lock).<br></div><div>2. During high disk usage scenarios, syscall latencies were high (sometimes >= ping-timeout value)<br></div><div>3. Before being handed over to a new thread at io-threads xlator, a fop gets executed in one of the threads that reads incoming messages from socket. This execution path includes some translators like protocol/server, index, quota-enforcer, marker. And these translators might access inode-ctx which involves locking inode (inode->lock). Due to this locking latency of syscall gets transferred to poller thread. Since poller thread is waiting on inode->lock, it won't be able to read ping requests from network in-time resulting in ping-timer expiry.<br></div><div><br>I think Kritika is working on a patch to eliminate locking on inode in 1 above. We also need to reduce the actual fop execution in poller thread. IOW, we need to hand over the fop execution to io-threads/syncop-threads as early as we can. [3] helps in this scenario as it adds back the socket for polling immediately after reading the entire msg but before execution of fop begins. So, even though fop execution is happening in poller thread, msgs from same connection can be read in other poller threads parallely (and we can scale up the number of epoll-threads when load is high).<br><br></div><div>Also, note that there is no way we can send entire ping request as "URGENT" data over network. So, prioritization in [1] is only the queue of messages waiting to be written to network. So, Though I suggested [1], the more I think of it, it seems less irrelevant.</div><div><br>[2] <a href="http://review.gluster.org/12402" target="_blank">http://review.gluster.org/<wbr>12402</a><br>[3] <a href="http://review.gluster.org/15036" target="_blank">http://review.gluster.org/<wbr>15036</a><br><br></div><span class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
<br>
Note : We could make this patch more generic so that any packets can be<br>
marked as priority to add into the head instead of just Ping packets.<br>
<br>
[1] : <a href="http://review.gluster.org/#/c/11935/" rel="noreferrer" target="_blank">http://review.gluster.org/#/c/<wbr>11935/</a><br>
<br>
Regards<br>
<br>
Rafi KC<br>
<br>
______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-devel</a><br>
</blockquote></span></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="m_-1651803091369338098gmail_signature">Raghavendra G<br></div>
</font></span></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Raghavendra G<br></div>
</div>