[Gluster-Maintainers] [Gluster-devel] Maintainers' meeting Agenda (10th Jan, 2018)

Atin Mukherjee amukherj at redhat.com
Thu Jan 11 10:59:40 UTC 2018 

On Wed, Jan 10, 2018 at 8:59 PM, Nithya Balachandran <nbalacha at redhat.com>
wrote:

>
>
> On 10 January 2018 at 20:36, Jeff Darcy <jeff at pl.atyp.us> wrote:
>
>> FYI, the crash that Kaleb mentioned looks like this, so I think Nithya
>> was right that it's a bug in shutdown. Better than memory corruption, which
>> is what I tend to think of when I hear about random crashes.
>>
>
> I was actually thinking of a different crash. Someone from the Glusterd
> team should take a look and see if this is a known crash.
>

This is not known to me atleast. Even though the crash is from glusterd
binary, we need some epoll and rpc experts to debug this. And yes, from the
logs it does look like this happened when cleanup_and_exit () was triggered
in glusterd.

t a a bt says:

(gdb) t a a bt

Thread 8 (LWP 529):
#0  0x00007ff603d2773c in _int_free () from ./lib64/libc.so.6
#1  0x00007ff6040ade1d in CRYPTO_free () from ./usr/lib64/libcrypto.so.10
#2  0x00007ff6041293e9 in lh_free () from ./usr/lib64/libcrypto.so.10
#3  0x00007ff60412bca0 in ?? () from ./usr/lib64/libcrypto.so.10
#4  0x00007ff5f81dc129 in fini_openssl_mt ()
    at
/home/jenkins/root/workspace/centos6-regression/rpc/rpc-transport/socket/src/socket.c:4081
#5  0x00007ff5f81cf9ff in __do_global_dtors_aux () from
./build/install/lib/glusterfs/4.0dev1/rpc-transport/socket.so
#6  0x0000000000000000 in ?? ()

Thread 7 (LWP 530):
#0  0x00007ff603d5bc4d in nanosleep () from ./lib64/libc.so.6
#1  0x00007ff603d5bac0 in sleep () from ./lib64/libc.so.6
#2  0x00007ff605185ab8 in pool_sweeper (arg=0x0)
    at
/home/jenkins/root/workspace/centos6-regression/libglusterfs/src/mem-pool.c:470
#3  0x00007ff60442faa1 in start_thread () from ./lib64/libpthread.so.0
#4  0x00007ff603d97bcd in clone () from ./lib64/libc.so.6

Thread 6 (LWP 560):
#0  0x00007ff60443368c in pthread_cond_wait@@GLIBC_2.3.2 () from
./lib64/libpthread.so.0
#1  0x00007ff5f9c3a3b6 in hooks_worker (args=0x1d0f470)
    at
/home/jenkins/root/workspace/centos6-regression/xlators/mgmt/glusterd/src/glusterd-hooks.c:528
#2  0x00007ff60442faa1 in start_thread () from ./lib64/libpthread.so.0
#3  0x00007ff603d97bcd in clone () from ./lib64/libc.so.6

Thread 5 (LWP 532):
#0  0x00007ff604433a5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
./lib64/libpthread.so.0
#1  0x00007ff60519d075 in syncenv_task (proc=0x1d07070)
    at
/home/jenkins/root/workspace/centos6-regression/libglusterfs/src/syncop.c:603
#2  0x00007ff60519d317 in syncenv_processor (thdata=0x1d07070)
    at
/home/jenkins/root/workspace/centos6-regression/libglusterfs/src/syncop.c:695
#3  0x00007ff60442faa1 in start_thread () from ./lib64/libpthread.so.0
#4  0x00007ff603d97bcd in clone () from ./lib64/libc.so.6

Thread 4 (LWP 531):
#0  0x00007ff604433a5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
./lib64/libpthread.so.0
#1  0x00007ff60519d075 in syncenv_task (proc=0x1d06cb0)
    at
/home/jenkins/root/workspace/centos6-regression/libglusterfs/src/syncop.c:603
#2  0x00007ff60519d317 in syncenv_processor (thdata=0x1d06cb0)
    at
/home/jenkins/root/workspace/centos6-regression/libglusterfs/src/syncop.c:695
#3  0x00007ff60442faa1 in start_thread () from ./lib64/libpthread.so.0
#4  0x00007ff603d97bcd in clone () from ./lib64/libc.so.6

Thread 3 (LWP 528):
#0  0x00007ff60443700d in nanosleep () from ./lib64/libpthread.so.0
#1  0x00007ff60515da5b in gf_timer_proc (data=0x1d05b60)
    at
/home/jenkins/root/workspace/centos6-regression/libglusterfs/src/timer.c:201
#2  0x00007ff60442faa1 in start_thread () from ./lib64/libpthread.so.0
#3  0x00007ff603d97bcd in clone () from ./lib64/libc.so.6

Thread 2 (LWP 527):
#0  0x00007ff6044302fd in pthread_join () from ./lib64/libpthread.so.0
#1  0x00007ff6051c5641 in event_dispatch_epoll (event_pool=0x1cfdbc0)
    at
/home/jenkins/root/workspace/centos6-regression/libglusterfs/src/event-epoll.c:742
---Type <return> to continue, or q <return> to quit---
#2  0x00007ff6051841e6 in event_dispatch (event_pool=0x1cfdbc0)
    at
/home/jenkins/root/workspace/centos6-regression/libglusterfs/src/event.c:124
#3  0x000000000040b1da in main (argc=11, argv=0x7fffbe19b1d8)
    at
/home/jenkins/root/workspace/centos6-regression/glusterfsd/src/glusterfsd.c:2672

Thread 1 (LWP 562):
#0  0x00007ff603ce1495 in raise () from ./lib64/libc.so.6
#1  0x00007ff603ce2c75 in abort () from ./lib64/libc.so.6
#2  0x00007ff603cda60e in __assert_fail_base () from ./lib64/libc.so.6
#3  0x00007ff603cda6d0 in __assert_fail () from ./lib64/libc.so.6
#4  0x00007ff6051c4a58 in event_unregister_epoll_common
(event_pool=0x1cfdbc0, fd=7, idx=-1, do_close=1)
    at
/home/jenkins/root/workspace/centos6-regression/libglusterfs/src/event-epoll.c:409
#5  0x00007ff6051c4c02 in event_unregister_close_epoll
(event_pool=0x1cfdbc0, fd=7, idx_hint=-1)
    at
/home/jenkins/root/workspace/centos6-regression/libglusterfs/src/event-epoll.c:453
#6  0x00007ff605184096 in event_unregister_close (event_pool=0x1cfdbc0,
fd=7, idx=-1)
    at
/home/jenkins/root/workspace/centos6-regression/libglusterfs/src/event.c:95
#7  0x00007ff5f81d22a5 in __socket_reset (this=0x7ff5e80128f0)
    at
/home/jenkins/root/workspace/centos6-regression/rpc/rpc-transport/socket/src/socket.c:971
#8  0x00007ff5f81dd863 in fini (this=0x7ff5e80128f0)
    at
/home/jenkins/root/workspace/centos6-regression/rpc/rpc-transport/socket/src/socket.c:4561
#9  0x00007ff604f12c2a in rpc_transport_destroy (this=0x7ff5e80128f0)
    at
/home/jenkins/root/workspace/centos6-regression/rpc/rpc-lib/src/rpc-transport.c:470
#10 0x00007ff604f12e40 in rpc_transport_unref (this=0x7ff5e80128f0)
    at
/home/jenkins/root/workspace/centos6-regression/rpc/rpc-lib/src/rpc-transport.c:520
#11 0x00007ff5f81d9132 in socket_server_event_handler (fd=11, idx=0, gen=1,
data=0x1d1ed90, poll_in=1, poll_out=0,
    poll_err=0) at
/home/jenkins/root/workspace/centos6-regression/rpc/rpc-transport/socket/src/socket.c:2973
#12 0x00007ff6051c50d7 in event_dispatch_epoll_handler
(event_pool=0x1cfdbc0, event=0x7ff5f6035e70)
    at
/home/jenkins/root/workspace/centos6-regression/libglusterfs/src/event-epoll.c:579
#13 0x00007ff6051c53d9 in event_dispatch_epoll_worker (data=0x1d5ace0)
    at
/home/jenkins/root/workspace/centos6-regression/libglusterfs/src/event-epoll.c:655
#14 0x00007ff60442faa1 in start_thread () from ./lib64/libpthread.so.0
#15 0x00007ff603d97bcd in clone () from ./lib64/libc.so.6



>> https://build.gluster.org/job/centos6-regression/8304/consoleFull
>>
>> #0  0x00007ff603ce1495 in raise () from /lib64/libc.so.6
>> #1  0x00007ff603ce2c75 in abort () from /lib64/libc.so.6
>> #2  0x00007ff603cda60e in __assert_fail_base () from /lib64/libc.so.6
>> #3  0x00007ff603cda6d0 in __assert_fail () from /lib64/libc.so.6
>> #4  0x00007ff6051c4a58 in event_unregister_epoll_common
>> (event_pool=0x1cfdbc0, fd=7, idx=-1,
>>     do_close=1)
>>     at /home/jenkins/root/workspace/centos6-regression/libglusterfs
>> /src/event-epoll.c:409
>> #5  0x00007ff6051c4c02 in event_unregister_close_epoll
>> (event_pool=0x1cfdbc0, fd=7, idx_hint=-1)
>>     at /home/jenkins/root/workspace/centos6-regression/libglusterfs
>> /src/event-epoll.c:453
>> #6  0x00007ff605184096 in event_unregister_close (event_pool=0x1cfdbc0,
>> fd=7, idx=-1)
>>     at /home/jenkins/root/workspace/centos6-regression/libglusterfs
>> /src/event.c:95
>> #7  0x00007ff5f81d22a5 in __socket_reset (this=0x7ff5e80128f0)
>>     at /home/jenkins/root/workspace/centos6-regression/rpc/rpc-tran
>> sport/socket/src/socket.c:971
>> #8  0x00007ff5f81dd863 in fini (this=0x7ff5e80128f0)
>>     at /home/jenkins/root/workspace/centos6-regression/rpc/rpc-tran
>> sport/socket/src/socket.c:4561
>> #9  0x00007ff604f12c2a in rpc_transport_destroy (this=0x7ff5e80128f0)
>>     at /home/jenkins/root/workspace/centos6-regression/rpc/rpc-lib/
>> src/rpc-transport.c:470
>> #10 0x00007ff604f12e40 in rpc_transport_unref (this=0x7ff5e80128f0)
>>     at /home/jenkins/root/workspace/centos6-regression/rpc/rpc-lib/
>> src/rpc-transport.c:520
>> #11 0x00007ff5f81d9132 in socket_server_event_handler (fd=11, idx=0,
>> gen=1, data=0x1d1ed90,
>>     poll_in=1, poll_out=0, poll_err=0)
>>     at /home/jenkins/root/workspace/centos6-regression/rpc/rpc-tran
>> sport/socket/src/socket.c:2973
>> #12 0x00007ff6051c50d7 in event_dispatch_epoll_handler
>> (event_pool=0x1cfdbc0, event=0x7ff5f6035e70)
>>     at /home/jenkins/root/workspace/centos6-regression/libglusterfs
>> /src/event-epoll.c:579
>> #13 0x00007ff6051c53d9 in event_dispatch_epoll_worker (data=0x1d5ace0)
>>     at /home/jenkins/root/workspace/centos6-regression/libglusterfs
>> /src/event-epoll.c:655
>> #14 0x00007ff60442faa1 in start_thread () from /lib64/libpthread.so.0
>> #15 0x00007ff603d97bcd in clone () from /lib64/libc.so.6
>>
>> _______________________________________________
>> maintainers mailing list
>> maintainers at gluster.org
>> http://lists.gluster.org/mailman/listinfo/maintainers
>>
>>
>
> _______________________________________________
> maintainers mailing list
> maintainers at gluster.org
> http://lists.gluster.org/mailman/listinfo/maintainers
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/maintainers/attachments/20180111/027755e0/attachment.html>

More information about the maintainers mailing list