[Bugs] [Bug 1408104] New: Fix potential socket_poller thread deadlock and resource leak

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1408104

            Bug ID: 1408104
           Summary: Fix potential socket_poller thread deadlock and
                    resource leak
           Product: Red Hat Gluster Storage
           Version: 3.2
         Component: rpc
          Assignee: rtalur at redhat.com
          Reporter: amukherj at redhat.com
        QA Contact: rhinduja at redhat.com
                CC: bugs at gluster.org, kaushal at redhat.com,
                    rhs-bugs at redhat.com
        Depends On: 1408101



+++ This bug was initially created as a clone of Bug #1408101 +++

The fix for bug #1404181 [1], has a potential deadlock and resource leak of the
socket_poller thread.

A disconnect caused by a PARENT_DOWN event during a fuse graph switch, can lead
to the socket_poller thread being deadlocked. The deadlock doesn't affect the
fuse client as no new fops are sent on the old graph.

In addition to the above, the race in gfapi solved by [1] can also occur in
other codepaths, and need to be solved.

Quoting Raghavendra G's comment from the review,
"""
- The race addressed by this patch (race b/w socket_disconnect cleaning up
resources in priv and socket_poller using the same and resulting in undefined
behaviour - crash/corruption etc) can potentially happen irrespective of the
codepaths socket_disconnect is invoked from (like glusterd, client_portmap_cbk,
handling of PARENT_DOWN, changelog etc). Note the usage of word "potential"
here and I am not saying that this race happens in existing code. However, I
would like this issue gets fixed for these potential cases too.
- If there are fops in progress at the time of graph switch, sending
PARENT_DOWN event on the currently active (soon to be old) graph is deferred
till all the fops are complete (though new graph becomes active and new I/O is
redirected to that graph). So, PARENT_DOWN event can be sent after processing
last response (to fop). This means PARENT_DOWN can be sent in thread executing
socket_poller itself. Since PARENT_DOWN triggers a disconnect and disconnect
waits for socket_poller to complete, we've a deadlock. Specifically the
deadlock is: socket_poller -> notify-msg-received -> fuse processes fop
response -> fuse sends PARENT_DOWN -> rpc-clnt calls rpc_clnt_disable ->
socket_disconnect -> wait till socket_poller to complete before returning from
socket_disconnect. Luckily we've have a socket_poller thread for each transport
and threads that deadlock are the threads belonging to transports from older
graphs on which no I/O happening. So, at worst this will be a case of resource
leakage (threads/sockets etc) of old graph.

"""


[1] https://review.gluster.org/16141


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1408101
[Bug 1408101] Fix potential socket_poller thread deadlock and resource leak
-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=5jyR0FHBaS&a=cc_unsubscribe