<div dir="ltr"><div dir="ltr">Hello,<div><br></div><div>I'm unable to figure out the issue by just looking at the backtrace. You might be hitting <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1650115">https://bugzilla.redhat.com/show_bug.cgi?id=1650115</a>. If you come across the same problem in future, please capture pstack output and share it with us.</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 8, 2019 at 2:31 PM Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com">cynthia.zhou@nokia-sbell.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="ZH-CN">
<div class="gmail-m_6082912093597218601WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">The env is not there anymore, but I have collected the thread stack trace of glusterd with command “thread apply all bt”<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Using host libthread_db library "/lib64/libthread_db.so.1".<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">0x00007f9ee9fcfa3d in __pthread_timedjoin_ex () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Missing separate debuginfos, use: dnf debuginfo-install rcp-pack-glusterfs-1.12.0_1_gc999db1-RCP2.wf29.x86_64<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">(gdb) thread apply all bt<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#0 0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#1 0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#2 0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#3 0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#4 0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#5 0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#6 0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#7 0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#8 0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#9 0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#0 0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#1 0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#2 0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#3 0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0, gen=4, idx=27) at socket.c:1201<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#4 0x00007f9ee4fbf99c in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2480<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#5 0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#6 0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at event-epoll.c:659<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#7 0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#8 0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Thread 7 (Thread 0x7f9ee49b3700 (LWP 1931)):<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#0 0x00007f9ee9fd45bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#1 0x00007f9ee5e651b9 in hooks_worker (args=0x1813000) at glusterd-hooks.c:529<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#2 0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#3 0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Thread 6 (Thread 0x7f9ee692e700 (LWP 1762)):<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#0 0x00007f9ee9fd497a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#1 0x00007f9eeb25d904 in syncenv_task (proc=0x1808e00) at syncop.c:603<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#2 0x00007f9eeb25db9f in syncenv_processor (thdata=0x1808e00) at syncop.c:695<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#3 0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#4 0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Thread 5 (Thread 0x7f9ee712f700 (LWP 1761)):<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">---Type <return> to continue, or q <return> to quit---<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#0 0x00007f9ee9fd497a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#1 0x00007f9eeb25d904 in syncenv_task (proc=0x1808a40) at syncop.c:603<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#2 0x00007f9eeb25db9f in syncenv_processor (thdata=0x1808a40) at syncop.c:695<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#3 0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#4 0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Thread 4 (Thread 0x7f9ee7930700 (LWP 1760)):<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#0 0x00007f9ee98725d0 in nanosleep () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#1 0x00007f9ee98724aa in sleep () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#2 0x00007f9eeb247fdf in pool_sweeper (arg=0x0) at mem-pool.c:481<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#3 0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#4 0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Thread 3 (Thread 0x7f9ee8131700 (LWP 1759)):<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#0 0x00007f9ee97e3d7c in sigtimedwait () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#1 0x00007f9ee9fd8bac in sigwait () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#2 0x0000000000409ed7 in glusterfs_sigwaiter ()<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#3 0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#4 0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Thread 2 (Thread 0x7f9ee8932700 (LWP 1758)):<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#0 0x00007f9ee9fd83b0 in nanosleep () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#1 0x00007f9eeb224545 in gf_timer_proc (data=0x1808580) at timer.c:164<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#2 0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#3 0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Thread 1 (Thread 0x7f9eeb707780 (LWP 1757)):<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#0 0x00007f9ee9fcfa3d in __pthread_timedjoin_ex () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#1 0x00007f9eeb282b09 in event_dispatch_epoll (event_pool=0x17feb00) at event-epoll.c:746<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#2 0x00007f9eeb246786 in event_dispatch (event_pool=0x17feb00) at event.c:124<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">#3 0x000000000040ab95 in main ()<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">(gdb)
<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">(gdb)
<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">(gdb) q!<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">A syntax error in expression, near `'.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">(gdb) quit<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif"> Sanju Rakonde <<a href="mailto:srakonde@redhat.com" target="_blank">srakonde@redhat.com</a>>
<br>
<b>Sent:</b> Monday, April 08, 2019 4:58 PM<br>
<b>To:</b> Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>><br>
<b>Cc:</b> Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>>; <a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-devel] glusterd stuck for glusterfs with version 3.12.15<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Can you please capture output of "pstack $(pidof glusterd)" and send it to us? We need to capture this information when glusterd is struck.<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">On Mon, Apr 8, 2019 at 8:05 AM Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>> wrote:<u></u><u></u></span></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi glusterfs experts,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Good day!<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">In my test env, sometimes glusterd stuck issue happened, and it is not responding to any gluster commands, when I checked this issue I find that glusterd thread
9 and thread 8 is dealing with the same socket, I thought following patch should be able to solve this issue, however after I merged this patch this issue still exist. When I looked into this code, it seems socket_event_poll_in called event_handled before
rpc_transport_pollin_destroy, I think this gives the chance for another poll for the exactly the same socket. And caused this glusterd stuck issue, also, I find there is no LOCK_DESTROY(&iobref->lock)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">In iobref_destroy, I think it is better to add destroy lock.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Following is the gdb info when this issue happened, I would like to know your opinion on this issue, thanks!<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8pt;font-family:"Courier New"">SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8pt;font-family:"Courier New""> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:8pt;font-family:"Courier New"">* socket: fix issue on concurrent handle of a socket</span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:8pt;font-family:"Courier New""> </span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:8pt;font-family:"Courier New""> </span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:8pt;font-family:"Courier New""> </span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:8pt;font-family:"Courier New"">GDB INFO:</span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in iobref_unref, I think
<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#0 0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#1 0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#2 0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#3 0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#4 0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#5 0x00007f9ee4fbf932 in socket_event_handler (<b><span style="color:red">fd=36</span>,<span style="color:red"> idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1,
poll_out=0, poll_err=0</span></b>) at socket.c:2471<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#6 0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#7 0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#8 0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#9 0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#0 0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#1 0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#2 0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#3 0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0, gen=4, idx=27) at socket.c:1201<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#4 0x00007f9ee4fbf99c in socket_event_handler (<b><span style="color:red">fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0</span></b>)
at socket.c:2480<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#5 0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#6 0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at event-epoll.c:659<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#7 0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#8 0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) thread 9<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">[Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#0 0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) bt<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#0 0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#1 0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#2 0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#3 0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#4 0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#5 0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#6 0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#7 0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#8 0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#9 0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) frame 2<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">#2 0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">944 iobuf.c: No such file or directory.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) print *iobref<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">$1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222, __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,
<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> __elision = 0, __list = {__prev = 0x4000, __next = 0x7f9ed00063b000}},
<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> __size = "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@\000\000\000\000\000\000\000\260c\000</span>О<span lang="EN-US">\177",
__align = 953482739714}}, ref = -256, iobrefs = 0xffffffffffffffff, alloced = -1, used = -1}<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) quit<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">A debugging session is active.<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US">_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a><u></u><u></u></span></p>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US"><br clear="all">
<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US">-- <u></u><u></u></span></p>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thanks,<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US">Sanju<u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>Thanks,<br></div>Sanju<br></div></div>