<div dir="ltr"><div dir="ltr">I upgraded the node that was crashing to 5.5 yesterday. Today, it got another crash. This is a 1x4 replicate cluster, you can find the config mentioned in my previous reports, and Amar should have it as well. Here's the log:<div><br></div><div><div>==> mnt-<SNIP>_data1.log <==</div><div>The message "I [MSGID: 108031] [afr-common.c:2543:afr_local_discovery_cbk] 0-<SNIP>_data1-replicate-0: selecting local read_child <SNIP>_data1-client-3" repeated 4 times between [2019-03-19 14:40:50.741147] and [2019-03-19 14:40:56.874832]</div><div>pending frames:</div><div>frame : type(1) op(LOOKUP)</div><div>frame : type(1) op(LOOKUP)</div><div>frame : type(1) op(READ)</div><div>frame : type(1) op(READ)</div><div>frame : type(1) op(READ)</div><div>frame : type(1) op(READ)</div><div>frame : type(0) op(0)</div><div>patchset: git://<a href="http://git.gluster.org/glusterfs.git" target="_blank">git.gluster.org/glusterfs.git</a></div><div>signal received: 6</div><div>time of crash:</div><div>2019-03-19 14:40:57</div><div>configuration details:</div><div>argp 1</div><div>backtrace 1</div><div>dlfcn 1</div><div>libpthread 1</div><div>llistxattr 1</div><div>setfsid 1</div><div>spinlock 1</div><div>epoll.h 1</div><div>xattr.h 1</div><div>st_atim.tv_nsec 1</div><div>package-string: glusterfs 5.5</div><div>/usr/lib64/libglusterfs.so.0(+0x2764c)[0x7ff841f8364c]</div><div>/usr/lib64/libglusterfs.so.0(gf_print_trace+0x306)[0x7ff841f8dd26]</div><div>/lib64/libc.so.6(+0x36160)[0x7ff84114a160]</div><div>/lib64/libc.so.6(gsignal+0x110)[0x7ff84114a0e0]</div><div>/lib64/libc.so.6(abort+0x151)[0x7ff84114b6c1]</div><div>/lib64/libc.so.6(+0x2e6fa)[0x7ff8411426fa]</div><div>/lib64/libc.so.6(+0x2e772)[0x7ff841142772]</div><div>/lib64/libpthread.so.0(pthread_mutex_lock+0x228)[0x7ff8414d80b8]</div><div>/usr/lib64/glusterfs/5.5/xlator/cluster/replicate.so(+0x5de3d)[0x7ff839fbae3d]</div><div>/usr/lib64/glusterfs/5.5/xlator/cluster/replicate.so(+0x70d51)[0x7ff839fcdd51]</div><div>/usr/lib64/glusterfs/5.5/xlator/protocol/client.so(+0x58e1f)[0x7ff83a252e1f]</div><div>/usr/lib64/libgfrpc.so.0(+0xe820)[0x7ff841d4e820]</div><div>/usr/lib64/libgfrpc.so.0(+0xeb6f)[0x7ff841d4eb6f]</div><div>/usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7ff841d4b063]</div><div>/usr/lib64/glusterfs/5.5/rpc-transport/socket.so(+0xa0ce)[0x7ff83b9690ce]</div><div>/usr/lib64/libglusterfs.so.0(+0x85519)[0x7ff841fe1519]</div><div>/lib64/libpthread.so.0(+0x7559)[0x7ff8414d5559]</div><div>/lib64/libc.so.6(clone+0x3f)[0x7ff84120c81f]</div><div>---------</div><div><div dir="ltr" class="m_6115359015614636207gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><br>Sincerely,<br>Artem<br><br>--<br>Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8px" target="_blank">APK Mirror</a><span style="font-size:12.8px">, Illogical Robot LLC</span></div><div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a> | <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br></div></div></div></div></div></div></div></div></div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 18, 2019 at 9:46 PM Amar Tumballi Suryanarayan <<a href="mailto:atumball@redhat.com" target="_blank">atumball@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Due to this issue, along with few other logging issues, we did make a glusterfs-5.5 release, which has the fix for particular crash.<div dir="auto"><br></div><div dir="auto">Regards,</div><div dir="auto">Amar</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 19 Mar, 2019, 1:04 AM , <<a href="mailto:brandon@thinkhuge.net" target="_blank">brandon@thinkhuge.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div class="gmail-m_6115359015614636207gmail-m_4165333850108574022m_-6922874925156956448WordSection1"><p class="MsoNormal">Hello Ville-Pekka and list,<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">I believe we are experiencing similar gluster fuse client crashes on 5.3 as mentioned here. This morning I made a post in regards.<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><a href="https://lists.gluster.org/pipermail/gluster-users/2019-March/036036.html" rel="noreferrer" target="_blank">https://lists.gluster.org/pipermail/gluster-users/2019-March/036036.html</a><u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Has this "performance.write-behind: off" setting continued to be all you needed to workaround the issue?<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Thanks,<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Brandon<u></u><u></u></p></div></div>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" rel="noreferrer" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div>