<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
font-size:10.5pt;
font-family:DengXian;
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
text-align:left;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;}
span.EmailStyle19
{mso-style-type:personal;
font-family:DengXian;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:DengXian;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="ZH-CN" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:windowtext"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" align="left" style="text-align:left"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext">
Ravishankar N <ravishankar@redhat.com> <br>
<b>Sent:</b> Thursday, September 27, 2018 6:04 PM<br>
<b>To:</b> Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou@nokia-sbell.com><br>
<b>Subject:</b> Re: query about one glustershd coredump issue<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" align="left" style="text-align:left"><span lang="EN-US"><o:p> </o:p></span></p>
<p><span lang="EN-US">Hi,</span><span lang="EN-US" style="font-size:11.0pt"><o:p></o:p></span></p>
<p><span lang="EN-US">I think it is better to send it on gluster-users mailing list to get more attention.
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US">Regards,<br>
Ravi<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On 09/27/2018 01:10 PM, Zhou, Cynthia (NSB - CN/Hangzhou) wrote:<o:p></o:p></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span lang="EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">In my test env(glusterfs 3.12.3 3brick config) when restart sn-0 sn-1 and sn-2 at the same time, occasionally, I met glustershd on sn-0 coredump.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Could you help to shed some light on this issue? Thanks!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">My gdb result of the coredump file<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[root@sn-0:/root]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"># gdb /usr/sbin/glusterfs core.glusterfs.0.c5f0c5547fbd4e5aa8f350b748e5675e.1812.1537967075000000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">GNU gdb (GDB) Fedora 8.1-14.wf29<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Copyright (C) 2018 Free Software Foundation, Inc.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">License GPLv3+: GNU GPL version 3 or later
<a href="http://gnu.org/licenses/gpl.html"><http://gnu.org/licenses/gpl.html></a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">This is free software: you are free to change and redistribute it.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">There is NO WARRANTY, to the extent permitted by law. Type "show copying"<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">and "show warranty" for details.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">This GDB was configured as "x86_64-redhat-linux-gnu".<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Type "show configuration" for configuration details.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">For bug reporting instructions, please see:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><a href="http://www.gnu.org/software/gdb/bugs/"><http://www.gnu.org/software/gdb/bugs/></a>.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Find the GDB manual and other documentation resources online at:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><a href="http://www.gnu.org/software/gdb/documentation/"><http://www.gnu.org/software/gdb/documentation/></a>.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">For help, type "help".<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Type "apropos word" to search for commands related to "word"...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Reading symbols from /usr/sbin/glusterfs...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">warning: core file may not match specified executable file.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1818]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1812]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1813]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1817]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1966]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1968]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1970]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1974]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1976]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1814]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1815]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1816]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1828]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Thread debugging using libthread_db enabled]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Using host libthread_db library "/lib64/libthread_db.so.1".<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Core was generated by `/usr/sbin/glusterfs -s sn-0.local -</span><b><span lang="EN-US" style="color:red">-volfile-id gluster/glustershd</span></b><span lang="EN-US"> -p /var/run/g'.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Program terminated with signal SIGSEGV, Segmentation fault.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#0 0x00007f1b5e5d7d24 in client3_3_lookup_cbk (req=0x7f1b44002300, iov=0x7f1b44002340, count=1,
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> myframe=0x7f1b4401c850) at client-rpc-fops.c:2802<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2802 client-rpc-fops.c: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Current thread is 1 (Thread 0x7f1b5f00c700 (LWP 1818))]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Missing separate debuginfos, use: dnf debuginfo-install rcp-pack-glusterfs-1.2.0-RCP2.wf29.x86_64<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) <b>bt</b><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#0 0x00007f1b5e5d7d24 in</span><b><span lang="EN-US" style="color:red"> client3_3_lookup_cbk</span></b><span lang="EN-US"> (req=0x7f1b44002300, iov=0x7f1b44002340, count=1,
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> myframe=0x7f1b4401c850) at client-rpc-fops.c:2802<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#1 0x00007f1b64553d47 in rpc_clnt_handle_reply (clnt=0x7f1b5808bbb0, pollin=0x7f1b580c6620)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> at rpc-clnt.c:778<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#2 0x00007f1b645542e5 in rpc_clnt_notify (trans=0x7f1b5808bde0, mydata=0x7f1b5808bbe0,
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f1b580c6620) at rpc-clnt.c:971<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#3 0x00007f1b64550319 in rpc_transport_notify (this=0x7f1b5808bde0, event=RPC_TRANSPORT_MSG_RECEIVED,
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> data=0x7f1b580c6620) at rpc-transport.c:538<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#4 0x00007f1b5f49734d in socket_event_poll_in (this=0x7f1b5808bde0, notify_handled=_gf_true)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> at socket.c:2315<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#5 0x00007f1b5f497992 in socket_event_handler (fd=25, idx=15, gen=7, data=0x7f1b5808bde0, poll_in=1,
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> poll_out=0, poll_err=0) at socket.c:2471<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#6 0x00007f1b647fe5ac in event_dispatch_epoll_handler (event_pool=0x230cb00, event=0x7f1b5f00be84)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> at event-epoll.c:583<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#7 0x00007f1b647fe883 in event_dispatch_epoll_worker (data=0x23543d0) at event-epoll.c:659<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#8 0x00007f1b6354a5da in start_thread () from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#9 0x00007f1b62e20cbf in clone () from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb)<b> info thread</b><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Id Target Id Frame <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">* 1 Thread 0x7f1b5f00c700 (LWP 1818) 0x00007f1b5e5d7d24 in
</span><b><span lang="EN-US" style="color:red">client3_3_lookup_cbk</span></b><span lang="EN-US"> (req=0x7f1b44002300,
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> iov=0x7f1b44002340, count=1, myframe=0x7f1b4401c850) at client-rpc-fops.c:2802<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 2 Thread 0x7f1b64c83780 (LWP 1812) 0x00007f1b6354ba3d in __pthread_timedjoin_ex ()<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 3 Thread 0x7f1b61eae700 (LWP 1813) 0x00007f1b63554300 in nanosleep () from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 4 Thread 0x7f1b5feaa700 (LWP 1817) 0x00007f1b635508ca in
<a href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2">pthread_cond_timedwait@@GLIBC_2.3.2</a> ()<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 5 Thread 0x7f1b5ca2b700 (LWP 1966) 0x00007f1b62dee4b0 in nanosleep () from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 6 Thread 0x7f1b4f7fe700 (LWP 1968) 0x00007f1b6355050c in
<a href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 7 Thread 0x7f1b4e7fc700 (LWP 1970) 0x00007f1b6355050c in
<a href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> ()<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 8 Thread 0x7f1b4d7fa700 (LWP 1974) 0x00007f1b62dee4b0 in nanosleep () from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 9 Thread 0x7f1b33fff700 (LWP 1976) 0x00007f1b62dee4b0 in nanosleep () from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 10 Thread 0x7f1b616ad700 (LWP 1814) 0x00007f1b62d5fcbc in sigtimedwait () from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 11 Thread 0x7f1b60eac700 (LWP 1815) 0x00007f1b62dee4b0 in nanosleep () from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 12 Thread 0x7f1b606ab700 (LWP 1816) 0x00007f1b635508ca in
<a href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2">pthread_cond_timedwait@@GLIBC_2.3.2</a> ()<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 13 Thread 0x7f1b5d6d5700 (LWP 1828) 0x00007f1b62e20fe7 in epoll_wait () from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) print local->loc<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Cannot access memory at address 0x0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb)</span><span lang="EN-US" style="color:red">
<b>print local</b></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:red">$1 = (clnt_local_t *) 0x0</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) print frame<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$2 = (call_frame_t *) 0x7f1b4401c850<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) print frame->local<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$3 = (void *) 0x0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) p *(call_frame_t *)myframe<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$6 = {root = 0x100000000, parent = 0x100000005, frames = {next = 0x7f1b4401c8a8, prev = 0x7f1b44010190},
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> </span><b><span lang="EN-US" style="color:red">local = 0x0</span></b><span lang="EN-US">, this = 0x0, ret = 0x0, ref_count = 0, lock = {spinlock = 0, mutex = {__data = {__lock = 0,
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> __prev = 0x7f1b44010190, __next = 0x0}},
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> __size = '\000' <repeats 24 times>, "\220\001\001D\033\177\000\000\000\000\000\000\000\000\000",
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> __align = 0}}, cookie = 0x7f1b4401ccf0, complete = _gf_false, op = GF_FOP_NULL, begin = {<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> tv_sec = 139755081730912, tv_usec = 139755081785872}, end = {tv_sec = 448811404,
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> tv_usec = 21474836481}, wind_from = 0x0, wind_to = 0x0, unwind_from = 0x0, unwind_to = 0x0}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Following is the stack info<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Sep 26 16:04:40.034577 sn-0 systemd-coredump[2612]: Process 1812 (glusterfs) of user 0 dumped core.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack trace of thread 1818:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #0 0x00007f1b5e5d7d24 client3_3_lookup_cbk (client.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #1 0x00007f1b64553d47 rpc_clnt_handle_reply (libgfrpc.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #2 0x00007f1b645542e5 rpc_clnt_notify (libgfrpc.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #3 0x00007f1b64550319 rpc_transport_notify (libgfrpc.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #4 0x00007f1b5f49734d socket_event_poll_in (socket.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #5 0x00007f1b5f497992 socket_event_handler (socket.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #6 0x00007f1b647fe5ac event_dispatch_epoll_handler (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #7 0x00007f1b647fe883 event_dispatch_epoll_worker (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #8 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #9 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack trace of thread 1812:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #0 0x00007f1b6354ba3d __GI___pthread_timedjoin_ex (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #1 0x00007f1b647feae1 event_dispatch_epoll (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #2 0x00007f1b647c2703 event_dispatch (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #3 0x000000000040ab95 main (glusterfsd)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #4 0x00007f1b62d4baf7 __libc_start_main (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #5 0x000000000040543a _start (glusterfsd)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack trace of thread 1813:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #0 0x00007f1b63554300 __nanosleep (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #1 0x00007f1b647a04e5 gf_timer_proc (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #2 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #3 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack trace of thread 1817:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #0 0x00007f1b635508ca
<a href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2">pthread_cond_timedwait@@GLIBC_2.3.2</a> (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #1 0x00007f1b647d98e3 syncenv_task (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #2 0x00007f1b647d9b7e syncenv_processor (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #3 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #4 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack trace of thread 1966:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #0 0x00007f1b62dee4b0 __nanosleep (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #1 0x00007f1b62dee38a sleep (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #2 0x00007f1b5e36970c afr_shd_index_healer (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #3 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #4 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack trace of thread 1968:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #0 0x00007f1b6355050c
<a href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #1 0x00007f1b647da993 __syncbarrier_wait (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #2 0x00007f1b647da9e4 syncbarrier_wait (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #3 0x00007f1b5e357cde afr_selfheal_unlocked_discover_on (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #4 0x00007f1b5e357d80 afr_selfheal_unlocked_discover (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #5 0x00007f1b5e363bf8 __afr_selfheal_entry_prepare (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #6 0x00007f1b5e3641c0 afr_selfheal_entry_dirent (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #7 0x00007f1b5e36488a afr_selfheal_entry_do_subvol (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #8 0x00007f1b5e365077 afr_selfheal_entry_do (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #9 0x00007f1b5e3656b6 __afr_selfheal_entry (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #10 0x00007f1b5e365bba afr_selfheal_entry (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #11 0x00007f1b5e35d250 afr_selfheal_do (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #12 0x00007f1b5e35d346 afr_selfheal (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #13 0x00007f1b5e368c5c afr_shd_selfheal (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #14 0x00007f1b5e36900b afr_shd_index_heal (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #15 0x00007f1b647ffed3 syncop_mt_dir_scan (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #16 0x00007f1b5e3692b5 afr_shd_index_sweep (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #17 0x00007f1b5e369394 afr_shd_index_sweep_all (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #18 0x00007f1b5e369697 afr_shd_index_healer (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #19 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #20 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack trace of thread 1970:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #0 0x00007f1b6355050c
<a href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #1 0x00007f1b647da993 __syncbarrier_wait (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #2 0x00007f1b647da9e4 syncbarrier_wait (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #3 0x00007f1b5e357742 afr_selfheal_unlocked_lookup_on (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #4 0x00007f1b5e364204 afr_selfheal_entry_dirent (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #5 0x00007f1b5e36488a afr_selfheal_entry_do_subvol (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #6 0x00007f1b5e365077 afr_selfheal_entry_do (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #7 0x00007f1b5e3656b6 __afr_selfheal_entry (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #8 0x00007f1b5e365bba afr_selfheal_entry (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #9 0x00007f1b5e35d250 afr_selfheal_do (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #10 0x00007f1b5e35d346 afr_selfheal (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #11 0x00007f1b5e368c5c afr_shd_selfheal (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #12 0x00007f1b5e36900b afr_shd_index_heal (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #13 0x00007f1b647ffed3 syncop_mt_dir_scan (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #14 0x00007f1b5e3692b5 afr_shd_index_sweep (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #15 0x00007f1b5e369394 afr_shd_index_sweep_all (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #16 0x00007f1b5e369697 afr_shd_index_healer (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #17 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #18 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack trace of thread 1974:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #0 0x00007f1b62dee4b0 __nanosleep (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #1 0x00007f1b62dee38a sleep (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #2 0x00007f1b5e36970c afr_shd_index_healer (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #3 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #4 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack trace of thread 1976:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #0 0x00007f1b62dee4b0 __nanosleep (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #1 0x00007f1b62dee38a sleep (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #2 0x00007f1b5e36970c afr_shd_index_healer (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #3 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #4 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack trace of thread 1814:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #0 0x00007f1b62d5fcbc __sigtimedwait (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #1 0x00007f1b63554afc sigwait (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #2 0x0000000000409ed7 glusterfs_sigwaiter (glusterfsd)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #3 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #4 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack trace of thread 1815:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #0 0x00007f1b62dee4b0 __nanosleep (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #1 0x00007f1b62dee38a sleep (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #2 0x00007f1b647c3f5c pool_sweeper (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #3 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #4 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack trace of thread 1816:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #0 0x00007f1b635508ca
<a href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2">pthread_cond_timedwait@@GLIBC_2.3.2</a> (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #1 0x00007f1b647d98e3 syncenv_task (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #2 0x00007f1b647d9b7e syncenv_processor (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #3 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #4 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack trace of thread 1828:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #0 0x00007f1b62e20fe7 epoll_wait (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #1 0x00007f1b647fe855 event_dispatch_epoll_worker (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #2 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #3 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
</blockquote>
<p class="MsoNormal" align="left" style="text-align:left"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
</div>
</body>
</html>