[Gluster-users] Gluster server crashes with signal 11 after probing peers.

[Ernie Dunbar]

Hi everyone.

I'm trying to add a new Gluster node to our cluster, and when trying to 
probing the first node in the cluster, the new node crashes with the 
following report (logs start when the daemon starts):

---------
[2016-03-30 20:32:05.191659] I [MSGID: 100030] [glusterfsd.c:2332:main] 
0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.9 
(args: /usr/sbin/glusterd -p /var/run/glusterd.pid)
[2016-03-30 20:32:05.195695] I [MSGID: 106478] [glusterd.c:1337:init] 
0-management: Maximum allowed open file descriptors set to 65536
[2016-03-30 20:32:05.195752] I [MSGID: 106479] [glusterd.c:1386:init] 
0-management: Using /var/lib/glusterd as working directory
[2016-03-30 20:32:05.200609] W [MSGID: 103071] 
[rdma.c:4594:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event 
channel creation failed [No such device]
[2016-03-30 20:32:05.200648] W [MSGID: 103055] [rdma.c:4901:init] 
0-rdma.management: Failed to initialize IB Device
[2016-03-30 20:32:05.200662] W [rpc-transport.c:359:rpc_transport_load] 
0-rpc-transport: 'rdma' initialization failed
[2016-03-30 20:32:05.200723] W [rpcsvc.c:1597:rpcsvc_transport_create] 
0-rpc-service: cannot create listener, initing the transport failed
[2016-03-30 20:32:05.200743] E [MSGID: 106243] [glusterd.c:1610:init] 
0-management: creation of 1 listeners failed, continuing with succeeded 
transport
[2016-03-30 20:32:07.135310] I [MSGID: 106513] 
[glusterd-store.c:2062:glusterd_restore_op_version] 0-glusterd: 
retrieved op-version: 30501
[2016-03-30 20:32:07.135775] I [MSGID: 106498] 
[glusterd-handler.c:3640:glusterd_friend_add_from_peerinfo] 
0-management: connect returned 0
[2016-03-30 20:32:07.135876] I [rpc-clnt.c:984:rpc_clnt_connection_init] 
0-management: setting frame-timeout to 600
[2016-03-30 20:32:07.136651] W [socket.c:870:__socket_keepalive] 
0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 13, Invalid 
argument
[2016-03-30 20:32:07.136673] E [socket.c:2966:socket_connect] 
0-management: Failed to set keep-alive: Invalid argument
[2016-03-30 20:32:07.136908] I [MSGID: 106194] 
[glusterd-store.c:3523:glusterd_store_retrieve_missed_snaps_list] 
0-management: No missed snaps list.
Final graph:
+------------------------------------------------------------------------------+
   1: volume management
   2:     type mgmt/glusterd
   3:     option rpc-auth.auth-glusterfs on
   4:     option rpc-auth.auth-unix on
   5:     option rpc-auth.auth-null on
   6:     option rpc-auth-allow-insecure on
   7:     option transport.socket.listen-backlog 128
   8:     option event-threads 1
   9:     option ping-timeout 0
  10:     option transport.socket.read-fail-log off
  11:     option transport.socket.keepalive-interval 2
  12:     option transport.socket.keepalive-time 10
  13:     option transport-type rdma
  14:     option working-directory /var/lib/glusterd
  15: end-volume
  16:
+------------------------------------------------------------------------------+
[2016-03-30 20:32:07.138287] I [MSGID: 101190] 
[event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread 
with index 1
[2016-03-30 20:32:07.138980] I [MSGID: 106544] 
[glusterd.c:159:glusterd_uuid_init] 0-management: retrieved UUID: 
ae191e96-9cd6-4e2b-acae-18f2cc45e6ed
[2016-03-30 20:32:07.139422] I [MSGID: 106163] 
[glusterd-handshake.c:1194:__glusterd_mgmt_hndsk_versions_ack] 
0-management: using the op-version 30501
[2016-03-30 20:32:14.394056] I [MSGID: 106487] 
[glusterd-handler.c:1239:__glusterd_handle_cli_probe] 0-glusterd: 
Received CLI probe req nfs1 24007
pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash:
2016-03-30 20:32:14
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.7.9
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x92)[0x7f0401a78562]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f0401a9464d]
/lib/x86_64-linux-gnu/libc.so.6(+0x36d40)[0x7f0400e76d40]
/lib/x86_64-linux-gnu/libpthread.so.0(pthread_spin_lock+0x0)[0x7f04012120f0]
---------


Both nodes are running GlusterFS 3.7.9 on Ubuntu Trusty Tahr (14.04 
LTS). Node 1 is running Linux kernel 3.13.0-55-generic #94-Ubuntu SMP, 
and node 3 is running Linux kernel 3.13.0-77-generic #121-Ubuntu SMP. To 
me, this seems to be the only difference between the systems, although 
the new node has the very latest version of the Gluster packages from 
the launchpad.net PPA. I would imagine that Node 1 has the same update, 
but it's hard to tell.

Any help would be much appreciated.