[Gluster-devel] rpc problems when using syncops in callbacks

[Krishnan Parthasarathi]

Hi Fog,

RPC callbacks are executed in the epoll thread[1]. Calling 
synctask_yield causes that epoll thread to be blocked until, the 
corresponding "wake" is called. Most likely, the code calling the "wake" 
is tied to a network-event, which wouldn't be 'noticed' until the epoll 
thread is unblocked. This is a classic deadlock.

You could attach gdb to the hung process and check if synctask_yield was 
called on the epoll thread. For further analysis, you might want to 
paste the output of "thread apply all bt full" from gdb, attached to the 
hung process.

[1] - epoll thread - is a short name for the thread executing epoll (), 
'listening' for network events.

HTH,
krish

On 04/26/2013 03:10 PM, fog - wrote:
> Hello everyone,
>
> I am trying to use syncops in a custom translator to keep my code at 
> least borderline readable, but I am having limited success.
>
> Problem Symptoms:
> Using a syncop in a regular fop is fine. However, in a callback it 
> causes a 'freeze' (synctask_yield called by the SYNCOP macro doesn't 
> return).
>
> What seems to be the Problem:
> Looking at the traces, there is no corresponding trace from 
> rpc_clnt_reply_init on the client to the trace from 
> rpcsvc_submit_generic on the server. In other words, the rpc reply 
> gets sent but isn't correctly received. Obviously this is not really a 
> networking problem but something else... I'd guess it's a deadlock 
> somewhere on the client?
> From the point of the syncop call onwards the client doesn't 'get' any 
> rpc replies any more (the next GlusterFS Handshake sent by the client, 
> which is received by the server and replied to, leads to a 
> disconnection accordingly).
>
> Again: This problem is only occurring when calling a syncop from a 
> callback function inside my translator, if I call the same syncop in a 
> fop call it completes fine.
>
> I hope you can make sense out of the above problem description.
> Thanks for your time ~
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> https://lists.nongnu.org/mailman/listinfo/gluster-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20130429/04fbd132/attachment-0001.html>