[Gluster-devel] syncop_getxattr stuck

Mon Dec 19 16:50:36 UTC 2016

Raghavendra,
                               Thanks for explaining this. I moved the call from callback to FOP and that seems to have resolved the issue.

Thanks and Regards,
Ram

-----Original Message-----
From: Raghavendra Gowdappa [mailto:rgowdapp at redhat.com] 
Sent: Sunday, December 18, 2016 11:39 PM
To: Ankireddypalle Reddy
Cc: Gluster Devel (gluster-devel at gluster.org)
Subject: Re: [Gluster-devel] syncop_getxattr stuck

A rule of thumb while using syncops is that one shouldn't use them in a thread which cannot be blocked (these are mostly threads polling for events on sockets). In other words, syncops are synchronous calls and the thread calling syncop gets blocked till the syncop is complete. This can easily lead to deadlocks as in the situation you've encountered here.

1. syncop_getxattr is invoked in the poller thread which reads responses from a socket (lets say s1 connected to brick b1).
2. If syncop_getxattr (called in 1) needs a response from b1 for the call to complete, we've a deadlock as
   2a. the thread calling syncop_getxattr needs to go back polling for incoming messages on socket s1 for syncop_getxattr to complete
   2b. syncop_getxattr can't complete unless we read response from socket s1.

One way you can remove the deadlock is to use a STACK_WIND (getxattr) instead of syncop_getxattr here.

regards,
Raghavendra

----- Original Message -----
> From: "Ankireddypalle Reddy" <areddy at commvault.com>
> To: "Gluster Devel (gluster-devel at gluster.org)" 
> <gluster-devel at gluster.org>
> Sent: Friday, December 16, 2016 1:44:32 AM
> Subject: [Gluster-devel] syncop_getxattr stuck
> 
> 
> 
> 
> 	
> 
> Attachment (1):
> 	
> 
> 1
> 
> trusted-cachevol.tcp-fuse.vol [Download] (2.48 KB)
> 
> 
> Hi,
> 
> I am working on a sample xlator. The xlator is stuck in a 
> syncop_getxattr call. Here’s the stack trace. Attached is my gluster volume config file.
> Please advise what could have possibly gone wrong.
> 
> 
> 
> #0 0x00007fd42bb236d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> 
> (gdb) where
> 
> #0 0x00007fd42bb236d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> 
> #1 0x00007fd42cd1aca3 in syncop_getxattr (subvol=0x7fd4180107d0, 
> loc=0x7fd41000752c, dict=0x7fd420447620, key=0x7fd41f293d08 
> "trusted.glusterfs.archive-size", xdata_in=0x0, xdata_out=0x0) at
> syncop.c:1675
> 
> #2 0x00007fd41f28ddb8 in get_arch_file_uint64_xattr 
> (this=0x7fd4180107d0, loc=0x7fd41000752c, name=0x7fd41f293d08 
> "trusted.glusterfs.archive-size", val=0x7fd4204476e0, 
> op_errno=0x7fd420447704) at archive.c:225
> 
> #3 0x00007fd41f28dea8 in get_arch_file_xattrs (this=0x7fd4180107d0, 
> loc=0x7fd41000752c, xattrs=0x7fd4204476e0, op_errno=0x7fd420447704) at
> archive.c:263
> 
> #4 0x00007fd41f2919ba in archive_stat_cbk (frame=0x7fd410006fec, 
> cookie=0x7fd41000752c, this=0x7fd4180107d0, op_ret=0, op_errno=117, 
> buf=0x7fd410006074, xdata=0x0) at archive.c:903
> 
> #5 0x00007fd41f519619 in dht_attr_cbk (frame=0x7fd4100075dc, 
> cookie=0x7fd41000770c, this=0x7fd41800f0c0, op_ret=0, op_errno=0, 
> stbuf=0x7fd420447940, xdata=0x0) at dht-inode-read.c:291
> 
> #6 0x00007fd41f75ada8 in afr_stat_cbk (frame=0x7fd41000770c, 
> cookie=0x0, this=0x7fd41800d610, op_ret=0, op_errno=0, 
> buf=0x7fd420447940, xdata=0x0) at
> afr-inode-read.c:211
> 
> #7 0x00007fd41fa044c2 in client3_3_stat_cbk (req=0x7fd41000437c, 
> iov=0x7fd4100043bc, count=1, myframe=0x7fd4100040ec) at
> client-rpc-fops.c:507
> 
> #8 0x00007fd42ca923b6 in rpc_clnt_handle_reply (clnt=0x7fd418054ef0,
> pollin=0x7fd418056840) at rpc-clnt.c:790
> 
> #9 0x00007fd42ca92909 in rpc_clnt_notify (trans=0x7fd418055400, 
> mydata=0x7fd418054f48, event=RPC_TRANSPORT_MSG_RECEIVED,
> data=0x7fd418056840) at rpc-clnt.c:970
> 
> #10 0x00007fd42ca8eb22 in rpc_transport_notify (this=0x7fd418055400, 
> event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7fd418056840) at
> rpc-transport.c:537
> 
> #11 0x00007fd421efcd0e in socket_event_poll_in (this=0x7fd418055400) 
> at
> socket.c:2265
> 
> #12 0x00007fd421efd26c in socket_event_handler (fd=16, idx=3, 
> data=0x7fd418055400, poll_in=1, poll_out=0, poll_err=0) at 
> socket.c:2395
> 
> #13 0x00007fd42cd3aadc in event_dispatch_epoll_handler 
> (event_pool=0xb96810,
> event=0x7fd420447ea0) at event-epoll.c:571
> 
> #14 0x00007fd42cd3aef9 in event_dispatch_epoll_worker (data=0xbece80) 
> at
> event-epoll.c:674
> 
> #15 0x00007fd42bb1fdc5 in start_thread () from /lib64/libpthread.so.0
> 
> #16 0x00007fd42b46321d in clone () from /lib64/libc.so.6
> 
> 
> 
> Thanks and Regards,
> 
> Ram
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material 
> for the sole use of the intended recipient. Any unauthorized review, 
> use or distribution by others is strictly prohibited. If you have 
> received the message by mistake, please advise the sender by reply 
> email and delete the message. Thank you."
> **********************************************************************
> 
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************