[Gluster-users] Gluster server crashes with signal 11 after probing peers.
Atin Mukherjee
amukherj at redhat.com
Fri Apr 1 04:07:59 UTC 2016
On 03/31/2016 11:18 PM, Ernie Dunbar wrote:
> Oops. I replied to Mohammed and not the whole list. Here's the backtrace
> and the full backtrace too:
>
> root at nfs3:/home/ernied# gdb /usr/sbin/glusterd /core
> GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
> Copyright (C) 2014 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from /usr/sbin/glusterd...(no debugging symbols
> found)...done.
>
> warning: core file may not match specified executable file.
> [New LWP 1519]
> [New LWP 1520]
> [New LWP 1516]
> [New LWP 1780]
> [New LWP 1518]
> [New LWP 1517]
> [New LWP 1781]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Core was generated by `/usr/sbin/glusterd -p /var/run/glusterd.pid'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 pthread_spin_lock () at ../nptl/sysdeps/x86_64/pthread_spin_lock.S:24
> 24 ../nptl/sysdeps/x86_64/pthread_spin_lock.S: No such file or
> directory.
>
> (gdb) bt
>
> #0 pthread_spin_lock () at ../nptl/sysdeps/x86_64/pthread_spin_lock.S:24
> #1 0x00007fb81dee520d in __gf_free () from
> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0
> #2 0x00007fb81deaa625 in data_destroy () from
> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0
> #3 0x00007fb81dead1cd in dict_get_str () from
> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0
> #4 0x00007fb8193f52f9 in glusterd_xfer_cli_probe_resp ()
> from /usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so
> #5 0x00007fb8193f6017 in __glusterd_handle_cli_probe ()
> from /usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so
> #6 0x00007fb8193ee9a0 in glusterd_big_locked_handler ()
> from /usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so
> #7 0x00007fb81def38d2 in synctask_wrap () from
> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0
> #8 0x00007fb81d2c38b0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> (gdb) bt full
>
> #0 pthread_spin_lock () at ../nptl/sysdeps/x86_64/pthread_spin_lock.S:24
> No locals.
> #1 0x00007fb81dee520d in __gf_free () from
> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0
> No symbol table info available.
> #2 0x00007fb81deaa625 in data_destroy () from
> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0
> No symbol table info available.
> #3 0x00007fb81dead1cd in dict_get_str () from
> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0
> No symbol table info available.
> #4 0x00007fb8193f52f9 in glusterd_xfer_cli_probe_resp ()
> from /usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so
> No symbol table info available.
> #5 0x00007fb8193f6017 in __glusterd_handle_cli_probe ()
> from /usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so
> No symbol table info available.
> #6 0x00007fb8193ee9a0 in glusterd_big_locked_handler ()
> from /usr/lib/x86_64-linux-gnu/glusterfs/3.7.9/xlator/mgmt/glusterd.so
> No symbol table info available.
> #7 0x00007fb81def38d2 in synctask_wrap () from
> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0
> No symbol table info available.
> #8 0x00007fb81d2c38b0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #9 0x0000000000000000 in ?? ()
> No symbol table info available.
This looks like similar to BZ 1310677 and unfortunately the fix is
missed in 3.7.9 and would be available in 3.7.10.
~Atin
>
>
> On 2016-03-30 23:15, Mohammed Rafi K C wrote:
>> Hi Ernie,
>>
>> Can you please paste the back trace from the core file.
>>
>> Regards
>> Rafi KC
>>
>> On 03/31/2016 02:31 AM, Ernie Dunbar wrote:
>>> Hi everyone.
>>>
>>> I'm trying to add a new Gluster node to our cluster, and when trying
>>> to probing the first node in the cluster, the new node crashes with
>>> the following report (logs start when the daemon starts):
>>>
>>> ---------
>>> [2016-03-30 20:32:05.191659] I [MSGID: 100030]
>>> [glusterfsd.c:2332:main] 0-/usr/sbin/glusterd: Started running
>>> /usr/sbin/glusterd version 3.7.9 (args: /usr/sbin/glusterd -p
>>> /var/run/glusterd.pid)
>>> [2016-03-30 20:32:05.195695] I [MSGID: 106478] [glusterd.c:1337:init]
>>> 0-management: Maximum allowed open file descriptors set to 65536
>>> [2016-03-30 20:32:05.195752] I [MSGID: 106479] [glusterd.c:1386:init]
>>> 0-management: Using /var/lib/glusterd as working directory
>>> [2016-03-30 20:32:05.200609] W [MSGID: 103071]
>>> [rdma.c:4594:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event
>>> channel creation failed [No such device]
>>> [2016-03-30 20:32:05.200648] W [MSGID: 103055] [rdma.c:4901:init]
>>> 0-rdma.management: Failed to initialize IB Device
>>> [2016-03-30 20:32:05.200662] W
>>> [rpc-transport.c:359:rpc_transport_load] 0-rpc-transport: 'rdma'
>>> initialization failed
>>> [2016-03-30 20:32:05.200723] W [rpcsvc.c:1597:rpcsvc_transport_create]
>>> 0-rpc-service: cannot create listener, initing the transport failed
>>> [2016-03-30 20:32:05.200743] E [MSGID: 106243] [glusterd.c:1610:init]
>>> 0-management: creation of 1 listeners failed, continuing with
>>> succeeded transport
>>> [2016-03-30 20:32:07.135310] I [MSGID: 106513]
>>> [glusterd-store.c:2062:glusterd_restore_op_version] 0-glusterd:
>>> retrieved op-version: 30501
>>> [2016-03-30 20:32:07.135775] I [MSGID: 106498]
>>> [glusterd-handler.c:3640:glusterd_friend_add_from_peerinfo]
>>> 0-management: connect returned 0
>>> [2016-03-30 20:32:07.135876] I
>>> [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting
>>> frame-timeout to 600
>>> [2016-03-30 20:32:07.136651] W [socket.c:870:__socket_keepalive]
>>> 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 13, Invalid
>>> argument
>>> [2016-03-30 20:32:07.136673] E [socket.c:2966:socket_connect]
>>> 0-management: Failed to set keep-alive: Invalid argument
>>> [2016-03-30 20:32:07.136908] I [MSGID: 106194]
>>> [glusterd-store.c:3523:glusterd_store_retrieve_missed_snaps_list]
>>> 0-management: No missed snaps list.
>>> Final graph:
>>> +------------------------------------------------------------------------------+
>>>
>>>
>>> 1: volume management
>>> 2: type mgmt/glusterd
>>> 3: option rpc-auth.auth-glusterfs on
>>> 4: option rpc-auth.auth-unix on
>>> 5: option rpc-auth.auth-null on
>>> 6: option rpc-auth-allow-insecure on
>>> 7: option transport.socket.listen-backlog 128
>>> 8: option event-threads 1
>>> 9: option ping-timeout 0
>>> 10: option transport.socket.read-fail-log off
>>> 11: option transport.socket.keepalive-interval 2
>>> 12: option transport.socket.keepalive-time 10
>>> 13: option transport-type rdma
>>> 14: option working-directory /var/lib/glusterd
>>> 15: end-volume
>>> 16:
>>> +------------------------------------------------------------------------------+
>>>
>>>
>>> [2016-03-30 20:32:07.138287] I [MSGID: 101190]
>>> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started
>>> thread with index 1
>>> [2016-03-30 20:32:07.138980] I [MSGID: 106544]
>>> [glusterd.c:159:glusterd_uuid_init] 0-management: retrieved UUID:
>>> ae191e96-9cd6-4e2b-acae-18f2cc45e6ed
>>> [2016-03-30 20:32:07.139422] I [MSGID: 106163]
>>> [glusterd-handshake.c:1194:__glusterd_mgmt_hndsk_versions_ack]
>>> 0-management: using the op-version 30501
>>> [2016-03-30 20:32:14.394056] I [MSGID: 106487]
>>> [glusterd-handler.c:1239:__glusterd_handle_cli_probe] 0-glusterd:
>>> Received CLI probe req nfs1 24007
>>> pending frames:
>>> frame : type(0) op(0)
>>> patchset: git://git.gluster.com/glusterfs.git
>>> signal received: 11
>>> time of crash:
>>> 2016-03-30 20:32:14
>>> configuration details:
>>> argp 1
>>> backtrace 1
>>> dlfcn 1
>>> libpthread 1
>>> llistxattr 1
>>> setfsid 1
>>> spinlock 1
>>> epoll.h 1
>>> xattr.h 1
>>> st_atim.tv_nsec 1
>>> package-string: glusterfs 3.7.9
>>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x92)[0x7f0401a78562]
>>>
>>>
>>> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f0401a9464d]
>>>
>>>
>>> /lib/x86_64-linux-gnu/libc.so.6(+0x36d40)[0x7f0400e76d40]
>>> /lib/x86_64-linux-gnu/libpthread.so.0(pthread_spin_lock+0x0)[0x7f04012120f0]
>>>
>>>
>>> ---------
>>>
>>>
>>> Both nodes are running GlusterFS 3.7.9 on Ubuntu Trusty Tahr (14.04
>>> LTS). Node 1 is running Linux kernel 3.13.0-55-generic #94-Ubuntu SMP,
>>> and node 3 is running Linux kernel 3.13.0-77-generic #121-Ubuntu SMP.
>>> To me, this seems to be the only difference between the systems,
>>> although the new node has the very latest version of the Gluster
>>> packages from the launchpad.net PPA. I would imagine that Node 1 has
>>> the same update, but it's hard to tell.
>>>
>>> Any help would be much appreciated.
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
More information about the Gluster-users
mailing list