<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 10, 2018 at 8:59 PM, Nithya Balachandran <span dir="ltr"><<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="gmail-">On 10 January 2018 at 20:36, Jeff Darcy <span dir="ltr"><<a href="mailto:jeff@pl.atyp.us" target="_blank">jeff@pl.atyp.us</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u>
<div><div style="font-family:Arial">FYI, the crash that Kaleb mentioned looks like this, so I think Nithya was right that it's a bug in shutdown. Better than memory corruption, which is what I tend to think of when I hear about random crashes.<br></div></div></blockquote><div><br></div></span><div>I was actually thinking of a different crash. Someone from the Glusterd team should take a look and see if this is a known crash. </div></div></div></div></blockquote><div><br></div><div>This is not known to me atleast. Even though the crash is from glusterd binary, we need some epoll and rpc experts to debug this. And yes, from the logs it does look like this happened when cleanup_and_exit () was triggered in glusterd.</div><div><br></div><div>t a a bt says:</div><div><br></div><div>(gdb) t a a bt<br><br>Thread 8 (LWP 529):<br>#0 0x00007ff603d2773c in _int_free () from ./lib64/libc.so.6<br>#1 0x00007ff6040ade1d in CRYPTO_free () from ./usr/lib64/libcrypto.so.10<br>#2 0x00007ff6041293e9 in lh_free () from ./usr/lib64/libcrypto.so.10<br>#3 0x00007ff60412bca0 in ?? () from ./usr/lib64/libcrypto.so.10<br>#4 0x00007ff5f81dc129 in fini_openssl_mt ()<br> at /home/jenkins/root/workspace/centos6-regression/rpc/rpc-transport/socket/src/socket.c:4081<br>#5 0x00007ff5f81cf9ff in __do_global_dtors_aux () from ./build/install/lib/glusterfs/4.0dev1/rpc-transport/socket.so<br>#6 0x0000000000000000 in ?? ()<br><br>Thread 7 (LWP 530):<br>#0 0x00007ff603d5bc4d in nanosleep () from ./lib64/libc.so.6<br>#1 0x00007ff603d5bac0 in sleep () from ./lib64/libc.so.6<br>#2 0x00007ff605185ab8 in pool_sweeper (arg=0x0)<br> at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/mem-pool.c:470<br>#3 0x00007ff60442faa1 in start_thread () from ./lib64/libpthread.so.0<br>#4 0x00007ff603d97bcd in clone () from ./lib64/libc.so.6<br><br>Thread 6 (LWP 560):<br>#0 0x00007ff60443368c in pthread_cond_wait@@GLIBC_2.3.2 () from ./lib64/libpthread.so.0<br>#1 0x00007ff5f9c3a3b6 in hooks_worker (args=0x1d0f470)<br> at /home/jenkins/root/workspace/centos6-regression/xlators/mgmt/glusterd/src/glusterd-hooks.c:528<br>#2 0x00007ff60442faa1 in start_thread () from ./lib64/libpthread.so.0<br>#3 0x00007ff603d97bcd in clone () from ./lib64/libc.so.6<br><br>Thread 5 (LWP 532):<br>#0 0x00007ff604433a5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from ./lib64/libpthread.so.0<br>#1 0x00007ff60519d075 in syncenv_task (proc=0x1d07070)<br> at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/syncop.c:603<br>#2 0x00007ff60519d317 in syncenv_processor (thdata=0x1d07070)<br> at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/syncop.c:695<br>#3 0x00007ff60442faa1 in start_thread () from ./lib64/libpthread.so.0<br>#4 0x00007ff603d97bcd in clone () from ./lib64/libc.so.6<br><br>Thread 4 (LWP 531):<br>#0 0x00007ff604433a5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from ./lib64/libpthread.so.0<br>#1 0x00007ff60519d075 in syncenv_task (proc=0x1d06cb0)<br> at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/syncop.c:603<br>#2 0x00007ff60519d317 in syncenv_processor (thdata=0x1d06cb0)<br> at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/syncop.c:695<br>#3 0x00007ff60442faa1 in start_thread () from ./lib64/libpthread.so.0<br>#4 0x00007ff603d97bcd in clone () from ./lib64/libc.so.6<br><br>Thread 3 (LWP 528):<br>#0 0x00007ff60443700d in nanosleep () from ./lib64/libpthread.so.0<br>#1 0x00007ff60515da5b in gf_timer_proc (data=0x1d05b60)<br> at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/timer.c:201<br>#2 0x00007ff60442faa1 in start_thread () from ./lib64/libpthread.so.0<br>#3 0x00007ff603d97bcd in clone () from ./lib64/libc.so.6<br><br>Thread 2 (LWP 527):<br>#0 0x00007ff6044302fd in pthread_join () from ./lib64/libpthread.so.0<br>#1 0x00007ff6051c5641 in event_dispatch_epoll (event_pool=0x1cfdbc0)<br> at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/event-epoll.c:742<br>---Type <return> to continue, or q <return> to quit---<br>#2 0x00007ff6051841e6 in event_dispatch (event_pool=0x1cfdbc0)<br> at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/event.c:124<br>#3 0x000000000040b1da in main (argc=11, argv=0x7fffbe19b1d8)<br> at /home/jenkins/root/workspace/centos6-regression/glusterfsd/src/glusterfsd.c:2672<br><br>Thread 1 (LWP 562):<br>#0 0x00007ff603ce1495 in raise () from ./lib64/libc.so.6<br>#1 0x00007ff603ce2c75 in abort () from ./lib64/libc.so.6<br>#2 0x00007ff603cda60e in __assert_fail_base () from ./lib64/libc.so.6<br>#3 0x00007ff603cda6d0 in __assert_fail () from ./lib64/libc.so.6<br>#4 0x00007ff6051c4a58 in event_unregister_epoll_common (event_pool=0x1cfdbc0, fd=7, idx=-1, do_close=1)<br> at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/event-epoll.c:409<br>#5 0x00007ff6051c4c02 in event_unregister_close_epoll (event_pool=0x1cfdbc0, fd=7, idx_hint=-1)<br> at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/event-epoll.c:453<br>#6 0x00007ff605184096 in event_unregister_close (event_pool=0x1cfdbc0, fd=7, idx=-1)<br> at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/event.c:95<br>#7 0x00007ff5f81d22a5 in __socket_reset (this=0x7ff5e80128f0)<br> at /home/jenkins/root/workspace/centos6-regression/rpc/rpc-transport/socket/src/socket.c:971<br>#8 0x00007ff5f81dd863 in fini (this=0x7ff5e80128f0)<br> at /home/jenkins/root/workspace/centos6-regression/rpc/rpc-transport/socket/src/socket.c:4561<br>#9 0x00007ff604f12c2a in rpc_transport_destroy (this=0x7ff5e80128f0)<br> at /home/jenkins/root/workspace/centos6-regression/rpc/rpc-lib/src/rpc-transport.c:470<br>#10 0x00007ff604f12e40 in rpc_transport_unref (this=0x7ff5e80128f0)<br> at /home/jenkins/root/workspace/centos6-regression/rpc/rpc-lib/src/rpc-transport.c:520<br>#11 0x00007ff5f81d9132 in socket_server_event_handler (fd=11, idx=0, gen=1, data=0x1d1ed90, poll_in=1, poll_out=0, <br> poll_err=0) at /home/jenkins/root/workspace/centos6-regression/rpc/rpc-transport/socket/src/socket.c:2973<br>#12 0x00007ff6051c50d7 in event_dispatch_epoll_handler (event_pool=0x1cfdbc0, event=0x7ff5f6035e70)<br> at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/event-epoll.c:579<br>#13 0x00007ff6051c53d9 in event_dispatch_epoll_worker (data=0x1d5ace0)<br> at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/event-epoll.c:655<br>#14 0x00007ff60442faa1 in start_thread () from ./lib64/libpthread.so.0<br>#15 0x00007ff603d97bcd in clone () from ./lib64/libc.so.6<br></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-h5"><div><div style="font-family:Arial"></div>
<div style="font-family:Arial"><br></div>
<div style="font-family:Arial"><a href="https://build.gluster.org/job/centos6-regression/8304/consoleFull" target="_blank">https://build.gluster.org/job/<wbr>centos6-regression/8304/consol<wbr>eFull</a><br></div>
<div style="font-family:Arial"><br></div>
<div style="font-family:Arial">#0 0x00007ff603ce1495 in raise () from /lib64/libc.so.6<br></div>
<div style="font-family:Arial">#1 0x00007ff603ce2c75 in abort () from /lib64/libc.so.6<br></div>
<div style="font-family:Arial">#2 0x00007ff603cda60e in __assert_fail_base () from /lib64/libc.so.6<br></div>
<div style="font-family:Arial">#3 0x00007ff603cda6d0 in __assert_fail () from /lib64/libc.so.6<br></div>
<div style="font-family:Arial">#4 0x00007ff6051c4a58 in event_unregister_epoll_common (event_pool=0x1cfdbc0, fd=7, idx=-1,<br></div>
<div style="font-family:Arial"> do_close=1)<br></div>
<div style="font-family:Arial"> at /home/jenkins/root/workspace/c<wbr>entos6-regression/libglusterfs<wbr>/src/event-epoll.c:409<br></div>
<div style="font-family:Arial">#5 0x00007ff6051c4c02 in event_unregister_close_epoll (event_pool=0x1cfdbc0, fd=7, idx_hint=-1)<br></div>
<div style="font-family:Arial"> at /home/jenkins/root/workspace/c<wbr>entos6-regression/libglusterfs<wbr>/src/event-epoll.c:453<br></div>
<div style="font-family:Arial">#6 0x00007ff605184096 in event_unregister_close (event_pool=0x1cfdbc0, fd=7, idx=-1)<br></div>
<div style="font-family:Arial"> at /home/jenkins/root/workspace/c<wbr>entos6-regression/libglusterfs<wbr>/src/event.c:95<br></div>
<div style="font-family:Arial">#7 0x00007ff5f81d22a5 in __socket_reset (this=0x7ff5e80128f0)<br></div>
<div style="font-family:Arial"> at /home/jenkins/root/workspace/c<wbr>entos6-regression/rpc/rpc-tran<wbr>sport/socket/src/socket.c:971<br></div>
<div style="font-family:Arial">#8 0x00007ff5f81dd863 in fini (this=0x7ff5e80128f0)<br></div>
<div style="font-family:Arial"> at /home/jenkins/root/workspace/c<wbr>entos6-regression/rpc/rpc-tran<wbr>sport/socket/src/socket.c:4561<br></div>
<div style="font-family:Arial">#9 0x00007ff604f12c2a in rpc_transport_destroy (this=0x7ff5e80128f0)<br></div>
<div style="font-family:Arial"> at /home/jenkins/root/workspace/c<wbr>entos6-regression/rpc/rpc-lib/<wbr>src/rpc-transport.c:470<br></div>
<div style="font-family:Arial">#10 0x00007ff604f12e40 in rpc_transport_unref (this=0x7ff5e80128f0)<br></div>
<div style="font-family:Arial"> at /home/jenkins/root/workspace/c<wbr>entos6-regression/rpc/rpc-lib/<wbr>src/rpc-transport.c:520<br></div>
<div style="font-family:Arial">#11 0x00007ff5f81d9132 in socket_server_event_handler (fd=11, idx=0, gen=1, data=0x1d1ed90,<br></div>
<div style="font-family:Arial"> poll_in=1, poll_out=0, poll_err=0)<br></div>
<div style="font-family:Arial"> at /home/jenkins/root/workspace/c<wbr>entos6-regression/rpc/rpc-tran<wbr>sport/socket/src/socket.c:2973<br></div>
<div style="font-family:Arial">#12 0x00007ff6051c50d7 in event_dispatch_epoll_handler (event_pool=0x1cfdbc0, event=0x7ff5f6035e70)<br></div>
<div style="font-family:Arial"> at /home/jenkins/root/workspace/c<wbr>entos6-regression/libglusterfs<wbr>/src/event-epoll.c:579<br></div>
<div style="font-family:Arial">#13 0x00007ff6051c53d9 in event_dispatch_epoll_worker (data=0x1d5ace0)<br></div>
<div style="font-family:Arial"> at /home/jenkins/root/workspace/c<wbr>entos6-regression/libglusterfs<wbr>/src/event-epoll.c:655<br></div>
<div style="font-family:Arial">#14 0x00007ff60442faa1 in start_thread () from /lib64/libpthread.so.0<br></div>
<div style="font-family:Arial">#15 0x00007ff603d97bcd in clone () from /lib64/libc.so.6<br></div>
</div>
<br></div></div>______________________________<wbr>_________________<br>
maintainers mailing list<br>
<a href="mailto:maintainers@gluster.org" target="_blank">maintainers@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/maintainers" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/maintainers</a><br>
<br></blockquote></div><br></div></div>
<br>______________________________<wbr>_________________<br>
maintainers mailing list<br>
<a href="mailto:maintainers@gluster.org">maintainers@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/maintainers" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/maintainers</a><br>
<br></blockquote></div><br></div></div>