<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi,</p>
<p>- Is this stock <span lang="EN-US">glusterfs-3.12.3? Or do you
have any patches applied on top of it?</span></p>
<p><span lang="EN-US">- If it is stock, could you create a BZ and
attach the core file and the /var/log/glusterfs/ logs from 3
nodes at the time of crash?<br>
</span></p>
<span lang="EN-US">Thanks,</span><br>
<span lang="EN-US">Ravi<br>
</span><br>
<br>
<div class="moz-cite-prefix">On 10/16/2018 08:45 AM, Zhou, Cynthia
(NSB - CN/Hangzhou) wrote:<br>
</div>
<blockquote type="cite"
cite="mid:DB7PR07MB39774D1FACD913EC7BDB91A3C8FE0@DB7PR07MB3977.eurprd07.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Cambria;
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:"\@等线";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:DengXian;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:DengXian;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:DengXian;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hi, <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">This issue happened
twice recently, when glustershd do heal, it generate
coredump occassinally,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I do some debug and find
that sometimes afr_selfheal_unlocked_discover_on do lookup
and saved the frame in function rpc_clnt_submit, when reply
comes, it find the saved frame , but the address is
different from the saved frame address, I think this is
wrong, but I can not find a clue how this happened?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[root@mn-0:/home/robot]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Thread debugging using
libthread_db enabled]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Using host libthread_db
library "/lib64/libthread_db.so.1".<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Core was generated by
`/usr/sbin/glusterfs -s sn-0.local --volfile-id
gluster/glustershd -p /var/run/g'.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Program terminated with
signal SIGSEGV, Segmentation fault.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#0 0x00007fb1a6fd9d24
in client3_3_lookup_cbk (req=0x7fb188010fb0,
iov=0x7fb188010ff0, count=1, myframe=<b><span
style="color:#C00000">0x7fb188215740</span></b>) at
client-rpc-fops.c:2802<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2802
client-rpc-fops.c: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Current thread is 1
(Thread 0x7fb1a7a0e700 (LWP 8151))]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Missing separate
debuginfos, use: dnf debuginfo-install
rcp-pack-glusterfs-1.2.0-RCP2.wf29.x86_64<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) bt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#0 0x00007fb1a6fd9d24
in client3_3_lookup_cbk (req=0x7fb188010fb0,
iov=0x7fb188010ff0, count=1, myframe=0x7fb188215740) at
client-rpc-fops.c:2802<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#1 0x00007fb1acf55d47
in rpc_clnt_handle_reply (clnt=0x7fb1a008fff0,
pollin=0x7fb1a0843910) at rpc-clnt.c:778<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#2 0x00007fb1acf562e5
in rpc_clnt_notify (trans=0x7fb1a00901c0,
mydata=0x7fb1a0090020, event=RPC_TRANSPORT_MSG_RECEIVED,
data=0x7fb1a0843910) at rpc-clnt.c:971<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#3 0x00007fb1acf52319
in rpc_transport_notify (this=0x7fb1a00901c0,
event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7fb1a0843910) at
rpc-transport.c:538<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#4 0x00007fb1a7e9934d
in socket_event_poll_in (this=0x7fb1a00901c0,
notify_handled=_gf_true) at socket.c:2315<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#5 0x00007fb1a7e99992
in socket_event_handler (fd=20, idx=14, gen=103,
data=0x7fb1a00901c0, poll_in=1, poll_out=0, poll_err=0) at
socket.c:2471<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#6 0x00007fb1ad2005ac
in event_dispatch_epoll_handler (event_pool=0x175fb00,
event=0x7fb1a7a0de84) at event-epoll.c:583<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#7 0x00007fb1ad200883
in event_dispatch_epoll_worker (data=0x17a73d0) at
event-epoll.c:659<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#8 0x00007fb1abf4c5da
in start_thread () from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#9 0x00007fb1ab822cbf
in clone () from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) info thread<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Id Target Id
Frame <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">* 1 Thread
0x7fb1a7a0e700 (LWP 8151) 0x00007fb1a6fd9d24 in
client3_3_lookup_cbk (req=0x7fb188010fb0,
iov=0x7fb188010ff0, count=1, myframe=0x7fb188215740) at
client-rpc-fops.c:2802<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 2 Thread
0x7fb1aa0af700 (LWP 8147) 0x00007fb1ab761cbc in sigtimedwait
() from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 3 Thread
0x7fb1a98ae700 (LWP 8148) 0x00007fb1ab7f04b0 in nanosleep ()
from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 4 Thread
0x7fb1957fa700 (LWP 8266) 0x00007fb1abf528ca in
<a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2">pthread_cond_timedwait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 5 Thread
0x7fb1a88ac700 (LWP 8150) 0x00007fb1abf528ca in
<a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2">pthread_cond_timedwait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 6 Thread
0x7fb17f7fe700 (LWP 8269) 0x00007fb1abf5250c in
<a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 7 Thread
0x7fb1aa8b0700 (LWP 8146) 0x00007fb1abf56300 in nanosleep ()
from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 8 Thread
0x7fb1ad685780 (LWP 8145) 0x00007fb1abf4da3d in
__pthread_timedjoin_ex () from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 9 Thread
0x7fb1a542d700 (LWP 8251) 0x00007fb1ab7f04b0 in nanosleep ()
from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 10 Thread
0x7fb1a4c2c700 (LWP 8260) 0x00007fb1abf528ca in
<a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2">pthread_cond_timedwait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 11 Thread
0x7fb196ffd700 (LWP 8263) 0x00007fb1abf528ca in
<a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2">pthread_cond_timedwait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 12 Thread
0x7fb1a60d7700 (LWP 8247) 0x00007fb1ab822fe7 in epoll_wait
() from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 13 Thread
0x7fb1a90ad700 (LWP 8149) 0x00007fb1abf528ca in
<a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2">pthread_cond_timedwait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) print
(call_frame_t*)myframe<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$1 = (call_frame_t *)
0x7fb188215740<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) print
*(call_frame_t*)myframe<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$2 = {root =
0x7fb1a0085090, parent = 0xcd4642c4a3efd678, frames = {next
= 0x151e2a92a5ae1bb, prev = 0x0},
<b><span style="color:#C00000">local = 0x0, this = 0x0, ret
= 0x0</span></b>, ref_count = 0, lock = {spinlock = 0,
mutex = {__data = {<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> __lock = 0,
__count = 0, __owner = 0, __nusers = 4, __kind = 0, __spins
= 0, __elision = 0, __list = {__prev = 0x7fb188215798,
__next = 0x7fb188215798}},
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> __size = '\000'
<repeats 12 times>, "\004", '\000' <repeats 11
times>,
"\230W!\210\261\177\000\000\230W!\210\261\177\000", __align
= 0}}, cookie = 0x7fb1882157a8, complete = (unknown:
2283886504),
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> op = 32689, begin =
{tv_sec = 140400469825464, tv_usec = 140400469825464}, end =
{tv_sec = 140400878737576, tv_usec = 140400132101048},
wind_from = 0x7fb18801cdc0 "", wind_to = 0x0, unwind_from =
0x0,
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> unwind_to = 0x0}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) thread 6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Switching to thread 6
(Thread 0x7fb17f7fe700 (LWP 8269))]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#0 0x00007fb1abf5250c
in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) bt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#0 0x00007fb1abf5250c
in <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#1 0x00007fb1ad1dc993
in __syncbarrier_wait (barrier=0x7fb188014790, waitfor=3) at
syncop.c:1138<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#2 0x00007fb1ad1dc9e4
in syncbarrier_wait (barrier=0x7fb188014790, waitfor=3) at
syncop.c:1155<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#3 0x00007fb1a6d59cde
in afr_selfheal_unlocked_discover_on (<b><span
style="color:#C00000">frame=0x7fb1882162d0</span></b>,
inode=0x7fb188215740, gfid=0x7fb17f7fdb00
"x\326\357\243\304BF</span><span
style="font-family:"Cambria",serif" lang="EN-US">ͻ</span><span
lang="EN-US">\341Z*\251\342Q\001\060\333\177\177\261\177",
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> replies=0x7fb17f7fcf40,
discover_on=0x7fb1a0084cb0 "\001\001\001", <incomplete
sequence \360\255\272>) at afr-self-heal-common.c:1809<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#4 0x00007fb1a6d59d80
in afr_selfheal_unlocked_discover (<b><span
style="color:#C00000">frame=0x7fb1882162d0</span></b>,
inode=0x7fb188215740, gfid=0x7fb17f7fdb00
"x\326\357\243\304BF</span><span
style="font-family:"Cambria",serif" lang="EN-US">ͻ</span><span
lang="EN-US">\341Z*\251\342Q\001\060\333\177\177\261\177",
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> replies=0x7fb17f7fcf40)
at afr-self-heal-common.c:1828<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#5 0x00007fb1a6d5e51f
in afr_selfheal_unlocked_inspect (frame=0x7fb1882162d0,
this=0x7fb1a001db40, gfid=0x7fb17f7fdb00
"x\326\357\243\304BF</span><span
style="font-family:"Cambria",serif" lang="EN-US">ͻ</span><span
lang="EN-US">\341Z*\251\342Q\001\060\333\177\177\261\177",
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> link_inode=0x7fb17f7fd9c8,
data_selfheal=0x7fb17f7fd9c4,
metadata_selfheal=0x7fb17f7fd9c0,
entry_selfheal=0x7fb17f7fd9bc) at
afr-self-heal-common.c:2241<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#6 0x00007fb1a6d5f19b
in afr_selfheal_do (frame=0x7fb1882162d0,
this=0x7fb1a001db40, gfid=0x7fb17f7fdb00
"x\326\357\243\304BF</span><span
style="font-family:"Cambria",serif" lang="EN-US">ͻ</span><span
lang="EN-US">\341Z*\251\342Q\001\060\333\177\177\261\177")
at afr-self-heal-common.c:2483<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#7 0x00007fb1a6d5f346
in afr_selfheal (this=0x7fb1a001db40, gfid=0x7fb17f7fdb00
"x\326\357\243\304BF</span><span
style="font-family:"Cambria",serif" lang="EN-US">ͻ</span><span
lang="EN-US">\341Z*\251\342Q\001\060\333\177\177\261\177")
at afr-self-heal-common.c:2543<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#8 0x00007fb1a6d6ac5c
in afr_shd_selfheal (healer=0x7fb1a0085640, child=0,
gfid=0x7fb17f7fdb00 "x\326\357\243\304BF</span><span
style="font-family:"Cambria",serif" lang="EN-US">ͻ</span><span
lang="EN-US">\341Z*\251\342Q\001\060\333\177\177\261\177")
at afr-self-heald.c:343<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#9 0x00007fb1a6d6b00b
in afr_shd_index_heal (subvol=0x7fb1a00171e0,
entry=0x7fb1a0714180, parent=0x7fb17f7fddc0,
data=0x7fb1a0085640) at afr-self-heald.c:440<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#10 0x00007fb1ad201ed3
in syncop_mt_dir_scan (frame=0x7fb1a07a0e90,
subvol=0x7fb1a00171e0, loc=0x7fb17f7fddc0, pid=-6,
data=0x7fb1a0085640, fn=0x7fb1a6d6aebc
<afr_shd_index_heal>, xdata=0x7fb1a07b4ed0,
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> max_jobs=1,
max_qlen=1024) at syncop-utils.c:407<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#11 0x00007fb1a6d6b2b5
in afr_shd_index_sweep (healer=0x7fb1a0085640,
vgfid=0x7fb1a6d93610 "glusterfs.xattrop_index_gfid") at
afr-self-heald.c:494<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#12 0x00007fb1a6d6b394
in afr_shd_index_sweep_all (healer=0x7fb1a0085640) at
afr-self-heald.c:517<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#13 0x00007fb1a6d6b697
in afr_shd_index_healer (data=0x7fb1a0085640) at
afr-self-heald.c:597<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#14 0x00007fb1abf4c5da
in start_thread () from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#15 0x00007fb1ab822cbf
in clone () from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="text-align:left" align="left"><b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif"
lang="EN-US">From:</span></b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif"
lang="EN-US"> Zhou, Cynthia (NSB - CN/Hangzhou)
<br>
<b>Sent:</b> Thursday, October 11, 2018 3:36 PM<br>
<b>To:</b> Ravishankar N <a class="moz-txt-link-rfc2396E" href="mailto:ravishankar@redhat.com"><ravishankar@redhat.com></a><br>
<b>Cc:</b> gluster-users
<a class="moz-txt-link-rfc2396E" href="mailto:gluster-users@gluster.org"><gluster-users@gluster.org></a><br>
<b>Subject:</b> glustershd coredump generated while
reboot all 3 sn nodes<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="text-align:left" align="left"><span
lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I find that when restart
sn node sometimes, the glustershd will exit and generate
coredump. It has happened twice in my env, I would like to
know your opinion on this issue, thanks!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">The glusterfs version I
use is glusterfs3.12.3<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[root@sn-1:/root]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"># gluster v info log<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Volume Name: log<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Type: Replicate<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Volume ID:
87bcbaf8-5fa4-4060-9149-23f832befe92<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Status: Started<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Snapshot Count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Number of Bricks: 1 x 3
= 3<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Transport-type: tcp<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Bricks:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Brick1:
sn-0.local:/mnt/bricks/log/brick<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Brick2:
sn-1.local:/mnt/bricks/log/brick<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Brick3:
sn-2.local:/mnt/bricks/log/brick<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Options Reconfigured:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">server.allow-insecure:
on<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">cluster.quorum-type:
auto<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">network.ping-timeout: 42<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">cluster.consistent-metadata:
on<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">cluster.favorite-child-policy:
mtime<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">cluster.quorum-reads: no<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">cluster.server-quorum-type:
none<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">transport.address-family:
inet<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">nfs.disable: on<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">performance.client-io-threads:
off<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">cluster.server-quorum-ratio:
51%<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[root@sn-1:/root]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">///////////////////////////////////////////////glustershd
coredump////////////////////////////////////////////////////////////////<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"># lz4 -d
core.glusterfs.0.c5f0c5547fbd4e5aa8f350b748e5675e.1812.1537967075000000.lz4<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Decoding file
core.glusterfs.0.c5f0c5547fbd4e5aa8f350b748e5675e.1812.1537967075000000
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">core.glusterfs.0.c5f :
decoded 263188480 bytes
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[root@sn-0:/mnt/export]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"># gdb
/usr/sbin/glusterfs
core.glusterfs.0.c5f0c5547fbd4e5aa8f350b748e5675e.1812.1537967075000000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">GNU gdb (GDB) Fedora
8.1-14.wf29<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Copyright (C) 2018 Free
Software Foundation, Inc.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">License GPLv3+: GNU GPL
version 3 or later <<a
href="http://gnu.org/licenses/gpl.html"
moz-do-not-send="true">http://gnu.org/licenses/gpl.html</a>><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">This is free software:
you are free to change and redistribute it.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">There is NO WARRANTY, to
the extent permitted by law. Type "show copying"<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">and "show warranty" for
details.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">This GDB was configured
as "x86_64-redhat-linux-gnu".<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Type "show
configuration" for configuration details.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">For bug reporting
instructions, please see:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><<a
href="http://www.gnu.org/software/gdb/bugs/"
moz-do-not-send="true">http://www.gnu.org/software/gdb/bugs/</a>>.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Find the GDB manual and
other documentation resources online at:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><<a
href="http://www.gnu.org/software/gdb/documentation/"
moz-do-not-send="true">http://www.gnu.org/software/gdb/documentation/</a>>.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">For help, type "help".<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Type "apropos word" to
search for commands related to "word"...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Reading symbols from
/usr/sbin/glusterfs...(no debugging symbols found)...done.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">warning: core file may
not match specified executable file.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1818]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1812]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1813]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1817]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1966]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1968]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1970]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1974]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1976]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1814]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1815]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1816]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[New LWP 1828]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Thread debugging using
libthread_db enabled]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Using host libthread_db
library "/lib64/libthread_db.so.1".<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Core was generated by
`/usr/sbin/glusterfs -s sn-0.local --volfile-id
gluster/glustershd -p /var/run/g'.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Program terminated with
signal SIGSEGV, Segmentation fault.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#0 0x00007f1b5e5d7d24
in client3_3_lookup_cbk (req=0x7f1b44002300,
iov=0x7f1b44002340, count=1, myframe=0x7f1b4401c850) at
client-rpc-fops.c:2802<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">2802
client-rpc-fops.c: No such file or directory.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Current thread is 1
(Thread 0x7f1b5f00c700 (LWP 1818))]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Missing separate
debuginfos, use: dnf debuginfo-install
rcp-pack-glusterfs-1.2.0_1_g54e6196-RCP2.wf29.x86_64<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) bt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#0 0x00007f1b5e5d7d24
in client3_3_lookup_cbk (req=0x7f1b44002300,
iov=0x7f1b44002340, count=1, myframe=0x7f1b4401c850) at
client-rpc-fops.c:2802<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#1 0x00007f1b64553d47
in rpc_clnt_handle_reply (clnt=0x7f1b5808bbb0,
pollin=0x7f1b580c6620) at rpc-clnt.c:778<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#2 0x00007f1b645542e5
in rpc_clnt_notify (trans=0x7f1b5808bde0,
mydata=0x7f1b5808bbe0, event=RPC_TRANSPORT_MSG_RECEIVED,
data=0x7f1b580c6620) at rpc-clnt.c:971<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#3 0x00007f1b64550319
in rpc_transport_notify (this=0x7f1b5808bde0,
event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f1b580c6620) at
rpc-transport.c:538<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#4 0x00007f1b5f49734d
in socket_event_poll_in (this=0x7f1b5808bde0,
notify_handled=_gf_true) at socket.c:2315<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#5 0x00007f1b5f497992
in socket_event_handler (fd=25, idx=15, gen=7,
data=0x7f1b5808bde0, poll_in=1, poll_out=0, poll_err=0) at
socket.c:2471<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#6 0x00007f1b647fe5ac
in event_dispatch_epoll_handler (event_pool=0x230cb00,
event=0x7f1b5f00be84) at event-epoll.c:583<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#7 0x00007f1b647fe883
in event_dispatch_epoll_worker (data=0x23543d0) at
event-epoll.c:659<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#8 0x00007f1b6354a5da
in start_thread () from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#9 0x00007f1b62e20cbf
in clone () from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="color:red" lang="EN-US">(gdb)
print *(call_frame_t*)myframe<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="color:red" lang="EN-US">$1
= {root = 0x100000000, parent = 0x100000005, frames =
{next = 0x7f1b4401c8a8, prev = 0x7f1b44010190},
</span><span style="color:#1F4E79" lang="EN-US">local = 0x0</span><span
style="color:red" lang="EN-US">, this = 0x0, ret = 0x0,
ref_count = 0, lock = {spinlock = 0, mutex = {__data = {<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="color:red" lang="EN-US">
__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind
= 0, __spins = 0, __elision = 0, __list = {__prev =
0x7f1b44010190, __next = 0x0}},
<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="color:red" lang="EN-US"> __size
= '\000' <repeats 24 times>,
"\220\001\001D\033\177\000\000\000\000\000\000\000\000\000",
__align = 0}}, cookie = 0x7f1b4401ccf0, complete =
_gf_false, op = GF_FOP_NULL, begin = {<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="color:red" lang="EN-US">
tv_sec = 139755081730912, tv_usec = 139755081785872}, end
= {tv_sec = 448811404, tv_usec = 21474836481}, wind_from =
0x0, wind_to = 0x0, unwind_from = 0x0, unwind_to = 0x0}<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) info thread<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Id Target Id
Frame <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">* 1 Thread
0x7f1b5f00c700 (LWP 1818) 0x00007f1b5e5d7d24 in
client3_3_lookup_cbk (req=0x7f1b44002300,
iov=0x7f1b44002340, count=1, myframe=0x7f1b4401c850) at
client-rpc-fops.c:2802<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 2 Thread
0x7f1b64c83780 (LWP 1812) 0x00007f1b6354ba3d in
__pthread_timedjoin_ex () from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 3 Thread
0x7f1b61eae700 (LWP 1813) 0x00007f1b63554300 in nanosleep ()
from /lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 4 Thread
0x7f1b5feaa700 (LWP 1817) 0x00007f1b635508ca in
<a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2">pthread_cond_timedwait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 5 Thread
0x7f1b5ca2b700 (LWP 1966) 0x00007f1b62dee4b0 in nanosleep ()
from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 6 Thread
0x7f1b4f7fe700 (LWP 1968) 0x00007f1b6355050c in
<a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 7 Thread
0x7f1b4e7fc700 (LWP 1970) 0x00007f1b6355050c in
<a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 8 Thread
0x7f1b4d7fa700 (LWP 1974) 0x00007f1b62dee4b0 in nanosleep ()
from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 9 Thread
0x7f1b33fff700 (LWP 1976) 0x00007f1b62dee4b0 in nanosleep ()
from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 10 Thread
0x7f1b616ad700 (LWP 1814) 0x00007f1b62d5fcbc in sigtimedwait
() from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 11 Thread
0x7f1b60eac700 (LWP 1815) 0x00007f1b62dee4b0 in nanosleep ()
from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 12 Thread
0x7f1b606ab700 (LWP 1816) 0x00007f1b635508ca in
<a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2">pthread_cond_timedwait@@GLIBC_2.3.2</a> () from
/lib64/libpthread.so.0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> 13 Thread
0x7f1b5d6d5700 (LWP 1828) 0x00007f1b62e20fe7 in epoll_wait
() from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) quit<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">The source code is like
this, so from gdb it coredump because frame->local is
<b><span style="color:red">NULL</span></b>!!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><img
style="width:10.3833in;height:3.85in" id="Picture_x0020_1"
src="cid:part4.202E5D53.070B190F@redhat.com" class=""
height="370" border="0" width="997"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">From sn-0 journal log,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Sep 26 16:04:40.034577
sn-0 systemd-coredump[2612]: Process 1812 (glusterfs) of
user 0 dumped core.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack
trace of thread 1818:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#0 0x00007f1b5e5d7d24 client3_3_lookup_cbk (client.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#1 0x00007f1b64553d47 rpc_clnt_handle_reply (libgfrpc.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#2 0x00007f1b645542e5 rpc_clnt_notify (libgfrpc.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#3 0x00007f1b64550319 rpc_transport_notify (libgfrpc.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#4 0x00007f1b5f49734d socket_event_poll_in (socket.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#5 0x00007f1b5f497992
socket_event_handler (socket.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#6 0x00007f1b647fe5ac event_dispatch_epoll_handler
(libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#7 0x00007f1b647fe883 event_dispatch_epoll_worker
(libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#8 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#9 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack
trace of thread 1812:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#0 0x00007f1b6354ba3d __GI___pthread_timedjoin_ex
(libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#1 0x00007f1b647feae1 event_dispatch_epoll
(libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#2 0x00007f1b647c2703
event_dispatch (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#3 0x000000000040ab95 main (glusterfsd)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#4 0x00007f1b62d4baf7 __libc_start_main (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#5 0x000000000040543a _start (glusterfsd)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack
trace of thread 1813:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#0 0x00007f1b63554300 __nanosleep (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#1 0x00007f1b647a04e5 gf_timer_proc (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#2 0x00007f1b6354a5da start_thread
(libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#3 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack
trace of thread 1817:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#0 0x00007f1b635508ca <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2">pthread_cond_timedwait@@GLIBC_2.3.2</a>
(libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#1 0x00007f1b647d98e3 syncenv_task (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#2 0x00007f1b647d9b7e syncenv_processor (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#3 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#4 0x00007f1b62e20cbf
__clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack
trace of thread 1966:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#0 0x00007f1b62dee4b0 __nanosleep (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#1 0x00007f1b62dee38a sleep (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#2 0x00007f1b5e36970c afr_shd_index_healer (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#3 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#4 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack
trace of thread 1968:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#0 0x00007f1b6355050c <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a>
(libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#1 0x00007f1b647da993 __syncbarrier_wait
(libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#2 0x00007f1b647da9e4 syncbarrier_wait (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#3 0x00007f1b5e357cde
afr_selfheal_unlocked_discover_on (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#4 0x00007f1b5e357d80 afr_selfheal_unlocked_discover
(replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#5 0x00007f1b5e363bf8
__afr_selfheal_entry_prepare (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#6 0x00007f1b5e3641c0 afr_selfheal_entry_dirent
(replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#7 0x00007f1b5e36488a afr_selfheal_entry_do_subvol
(replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#8 0x00007f1b5e365077 afr_selfheal_entry_do (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#9 0x00007f1b5e3656b6 __afr_selfheal_entry (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#10 0x00007f1b5e365bba afr_selfheal_entry (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#11 0x00007f1b5e35d250 afr_selfheal_do (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#12 0x00007f1b5e35d346
afr_selfheal (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#13 0x00007f1b5e368c5c afr_shd_selfheal (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#14 0x00007f1b5e36900b afr_shd_index_heal (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#15 0x00007f1b647ffed3 syncop_mt_dir_scan
(libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#16 0x00007f1b5e3692b5 afr_shd_index_sweep (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#17 0x00007f1b5e369394 afr_shd_index_sweep_all
(replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#18 0x00007f1b5e369697 afr_shd_index_healer (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#19 0x00007f1b6354a5da
start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#20 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack
trace of thread 1970:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#0 0x00007f1b6355050c <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_wait@@GLIBC_2.3.2">pthread_cond_wait@@GLIBC_2.3.2</a>
(libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#1 0x00007f1b647da993 __syncbarrier_wait
(libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#2 0x00007f1b647da9e4 syncbarrier_wait (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#3 0x00007f1b5e357742 afr_selfheal_unlocked_lookup_on
(replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#4 0x00007f1b5e364204 afr_selfheal_entry_dirent
(replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#5 0x00007f1b5e36488a afr_selfheal_entry_do_subvol
(replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#6 0x00007f1b5e365077 afr_selfheal_entry_do (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#7 0x00007f1b5e3656b6 __afr_selfheal_entry (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#8 0x00007f1b5e365bba afr_selfheal_entry
(replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#9 0x00007f1b5e35d250 afr_selfheal_do (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#10 0x00007f1b5e35d346 afr_selfheal (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#11 0x00007f1b5e368c5c afr_shd_selfheal (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#12 0x00007f1b5e36900b afr_shd_index_heal (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#13
0x00007f1b647ffed3 syncop_mt_dir_scan (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#14 0x00007f1b5e3692b5 afr_shd_index_sweep (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#15 0x00007f1b5e369394 afr_shd_index_sweep_all
(replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#16 0x00007f1b5e369697 afr_shd_index_healer (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#17 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#18 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack
trace of thread 1974:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#0 0x00007f1b62dee4b0 __nanosleep (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#1 0x00007f1b62dee38a sleep (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#2 0x00007f1b5e36970c afr_shd_index_healer (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#3 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#4 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack
trace of thread 1976:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#0 0x00007f1b62dee4b0 __nanosleep (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#1 0x00007f1b62dee38a sleep
(libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#2 0x00007f1b5e36970c afr_shd_index_healer (replicate.so)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#3 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#4 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack
trace of thread 1814:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#0 0x00007f1b62d5fcbc __sigtimedwait (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#1 0x00007f1b63554afc sigwait (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#2 0x0000000000409ed7 glusterfs_sigwaiter (glusterfsd)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#3 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#4 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack
trace of thread 1815:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#0 0x00007f1b62dee4b0 __nanosleep (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#1 0x00007f1b62dee38a sleep
(libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#2 0x00007f1b647c3f5c pool_sweeper (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#3 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#4 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack
trace of thread 1816:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> #0
0x00007f1b635508ca <a class="moz-txt-link-abbreviated" href="mailto:pthread_cond_timedwait@@GLIBC_2.3.2">pthread_cond_timedwait@@GLIBC_2.3.2</a>
(libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#1 0x00007f1b647d98e3 syncenv_task (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#2 0x00007f1b647d9b7e
syncenv_processor (libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#3 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#4 0x00007f1b62e20cbf __clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Stack
trace of thread 1828:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#0 0x00007f1b62e20fe7 epoll_wait (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#1 0x00007f1b647fe855 event_dispatch_epoll_worker
(libglusterfs.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#2 0x00007f1b6354a5da start_thread (libpthread.so.0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">
#3 0x00007f1b62e20cbf
__clone (libc.so.6)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</blockquote>
<br>
</body>
</html>