<div dir="ltr">After having tried several things, it seems that it will be complex to solve these races. All attempts to fix them have caused failures in other connections. Since I've other work to do and it doesn't seem to be causing serious failures in production, for now I'll leave this. I'll retake this when I've more time.<div><br></div><div>Xavi</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 29, 2018 at 11:07 PM, Xavi Hernandez <span dir="ltr"><<a href="mailto:jahernan@redhat.com" target="_blank">jahernan@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi all,<div><br></div><div>I've identified a race in RPC layer that caused some spurious disconnections and CHILD_DOWN notifications.</div><div><br></div><div>The problem happens when protocol/client reconfigures a connection to move from glusterd to glusterfsd. This is done by calling rpc_clnt_reconfig() followed by rpc_transport_disconnect().</div><div><br></div><div>This seems fine because client_rpc_notify() will call rpc_clnt_cleanup_and_start() when the disconnect notification is received. However There's a problem.</div><div><br></div><div>Suppose that the disconnection notification has been executed and we are just about to call rpc_clnt_cleanup_and_start(). If at this point the reconnection timer is fired, rpc_clnt_reconnect() will be processed. This will cause the socket to be reconnected and a connection notification will be processed. Then a handshake request will be sent to the server.</div><div><br></div><div>However, when rpc_clnt_cleanup_and_start() continues, all sent XID's are deleted. When we receive the answer from the handshake, we are unable to map the XID, making the request to fail. So the handshake fails and the client is considered down, sending a CHILD_DOWN notification to upper xlators.</div><div><br></div><div>This causes, in some tests, to start processing things while a brick is down unexpectedly, causing spurious failures on the test.</div><div><br></div><div>To solve the problem I've forced the rpc_clnt_reconfig() function to disable the RPC connection using similar code to rcp_clnt_disable(). This prevents the background rpc_clnt_reconnect() timer to be executed, avoiding the problem.</div><div><br></div><div>This seems to work fine for many tests, but it seems to be causing some issue in gfapi based tests. I'm still investigating this.</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>Xavi<br></div></font></span></div>
</blockquote></div><br></div>