<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=koi8-r">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:DengXian;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:DengXian;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:DengXian;}
/* Page Definitions */
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="#0563C1" vlink="#954F72" style="text-justify-trim:punctuation">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hi glusterfs experts,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Good day!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">In my test env, sometimes glusterd stuck issue happened, and it is not responding to any gluster commands, when I checked this issue I find that glusterd thread 9 and thread 8 is dealing with the same socket, I thought
following patch should be able to solve this issue, however after I merged this patch this issue still exist. When I looked into this code, it seems socket_event_poll_in called event_handled before rpc_transport_pollin_destroy, I think this gives the chance
for another poll for the exactly the same socket. And caused this glusterd stuck issue, also, I find there is no LOCK_DESTROY(&iobref->lock)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">In iobref_destroy, I think it is better to add destroy lock.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Following is the gdb info when this issue happened, I would like to know your opinion on this issue, thanks!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" align="left" style="text-align:left;text-autospace:none"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537<o:p></o:p></span></p>
<p class="MsoNormal" align="left" style="text-align:left;text-autospace:none"><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">* socket: fix issue on concurrent handle of a socket<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New""><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:8.0pt;font-family:"Courier New"">GDB INFO:</span></b><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in iobref_unref, I think
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#0 0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#1 0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#2 0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#3 0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#4 0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#5 0x00007f9ee4fbf932 in socket_event_handler (<b><span style="color:red">fd=36</span>,<span style="color:red"> idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0</span></b>) at socket.c:2471<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#6 0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#7 0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#8 0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#9 0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#0 0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#1 0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#2 0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#3 0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0, gen=4, idx=27) at socket.c:1201<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#4 0x00007f9ee4fbf99c in socket_event_handler (<b><span style="color:red">fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0</span></b>) at socket.c:2480<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#5 0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#6 0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at event-epoll.c:659<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#7 0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#8 0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) thread 9<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#0 0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) bt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#0 0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#1 0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#2 0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#3 0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#4 0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#5 0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#6 0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#7 0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#8 0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#9 0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) frame 2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#2 0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">944 iobuf.c: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) print *iobref<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222, __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> __elision = 0, __list = {__prev = 0x4000, __next = 0x7f9ed00063b000}},
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> __size = "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@\000\000\000\000\000\000\000\260c\000ï\177", __align = 953482739714}}, ref = -256, iobrefs = 0xffffffffffffffff,
alloced = -1, used = -1}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) quit<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">A debugging session is active.<o:p></o:p></span></p>
</div>
</body>
</html>