[Bugs] [Bug 1739884] glusterfsd process crashes with SIGSEGV

Sun Aug 11 16:09:31 UTC 2019

https://bugzilla.redhat.com/show_bug.cgi?id=1739884


--- Comment #1 from Chad Feller <cfeller at gmail.com> ---
Less than 24 hours later, a second crash nearly identical with the exception
that this time around, the glusterfsd process on both nodes crashed within a
few seconds of each other. I had just added four new bricks and was about 8
hours into a rebalance when it crashed:

###########################
# log from first node:
[2019-08-09 17:06:48.926301] I [MSGID: 115036] [server.c:499:server_rpc_notify]
0-gv0-server: disconnecting connection from
CTX_ID:b36e17ac-62e5-4171-bfdc-1707aa5ec5a9-GRAPH_ID:0-PID:31299-HOST:gluster01-PC_NAME:gv0-client-2-RECON_NO:-0
[2019-08-09 17:06:48.926480] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-gv0-server: Shutting down connection
CTX_ID:b36e17ac-62e5-4171-bfdc-1707aa5ec5a9-GRAPH_ID:0-PID:31299-HOST:gluster01-PC_NAME:gv0-client-2-RECON_NO:-0
[2019-08-09 17:06:48.956140] I [dict.c:541:dict_get]
(-->/usr/lib64/glusterfs/6.4/xlator/features/worm.so(+0x7241) [0x7f5f80443241]
-->/usr/lib64/glusterfs/6.4/xlator/features/locks.so(+0x1c219) [0x7f5f80669219]
-->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f5f900860b4] ) 0-dict: !this ||
key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument]
[2019-08-09 17:06:49.038307] I [dict.c:541:dict_get]
(-->/lib64/libglusterfs.so.0(default_fremovexattr+0xe7) [0x7f5f901251f7]
-->/usr/lib64/glusterfs/6.4/xlator/features/locks.so(+0x1d5dd) [0x7f5f8066a5dd]
-->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f5f900860b4] ) 0-dict: !this ||
key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument]
[2019-08-09 17:06:49.099221] I [dict.c:541:dict_get]
(-->/usr/lib64/glusterfs/6.4/xlator/features/worm.so(+0x7241) [0x7f5f80443241]
-->/usr/lib64/glusterfs/6.4/xlator/features/locks.so(+0x1c219) [0x7f5f80669219]
-->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f5f900860b4] ) 0-dict: !this ||
key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument]
pending frames:
frame : type(1) op(INODELK)
patchset: git://git.gluster.org/glusterfs.git
signal received: 11
time of crash: 
2019-08-09 17:06:49
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 6.4
/lib64/libglusterfs.so.0(+0x26e00)[0x7f5f90091e00]
/lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f5f9009c804]
/lib64/libc.so.6(+0x36340)[0x7f5f8e6d1340]
/usr/lib64/glusterfs/6.4/rpc-transport/socket.so(+0xa4cc)[0x7f5f844564cc]
/lib64/libglusterfs.so.0(+0x8b4d6)[0x7f5f900f64d6]
/lib64/libpthread.so.0(+0x7dd5)[0x7f5f8eed1dd5]
/lib64/libc.so.6(clone+0x6d)[0x7f5f8e79902d]
---------

###########################
# log from second node:
[2019-08-09 17:06:48.926010] I [MSGID: 115036] [server.c:499:server_rpc_notify]
0-gv0-server: disconnecting connection from
CTX_ID:b36e17ac-62e5-4171-bfdc-1707aa5ec5a9-GRAPH_ID:0-PID:31299-HOST:gluster01-PC_NAME:gv0-client-3-RECON_NO:-0
[2019-08-09 17:06:48.926231] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-gv0-server: Shutting down connection
CTX_ID:b36e17ac-62e5-4171-bfdc-1707aa5ec5a9-GRAPH_ID:0-PID:31299-HOST:gluster01-PC_NAME:gv0-client-3-RECON_NO:-0
[2019-08-09 17:06:48.955802] I [dict.c:541:dict_get]
(-->/usr/lib64/glusterfs/6.4/xlator/features/worm.so(+0x7241) [0x7f47b8c64241]
-->/usr/lib64/glusterfs/6.4/xlator/features/locks.so(+0x1c219) [0x7f47b8e8a219]
-->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f47c88a70b4] ) 0-dict: !this ||
key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument]
[2019-08-09 17:06:49.038024] I [dict.c:541:dict_get]
(-->/lib64/libglusterfs.so.0(default_fremovexattr+0xe7) [0x7f47c89461f7]
-->/usr/lib64/glusterfs/6.4/xlator/features/locks.so(+0x1d5dd) [0x7f47b8e8b5dd]
-->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f47c88a70b4] ) 0-dict: !this ||
key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument]
[2019-08-09 17:06:49.098930] I [dict.c:541:dict_get]
(-->/usr/lib64/glusterfs/6.4/xlator/features/worm.so(+0x7241) [0x7f47b8c64241]
-->/usr/lib64/glusterfs/6.4/xlator/features/locks.so(+0x1c219) [0x7f47b8e8a219]
-->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f47c88a70b4] ) 0-dict: !this ||
key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument]
[2019-08-09 17:06:50.042220] I [MSGID: 115072]
[server-rpc-fops_v2.c:1681:server4_setattr_cbk] 0-gv0-server: 349203: SETATTR
/munin/var/lib (418e15b7-70d5-4f11-86ac-c1d8aeb6aa12), client:
CTX_ID:d6150421-3ac2-4dbc-a665-d619f13dbb15-GRAPH_ID:0-PID:538-HOST:munin-PC_NAME:gv0-client-3-RECON_NO:-0,
error-xlator: gv0-access-control [Operation not permitted]
[2019-08-09 17:06:50.962416] I [MSGID: 115072]
[server-rpc-fops_v2.c:1681:server4_setattr_cbk] 0-gv0-server: 349225: SETATTR
/munin/var/lib (418e15b7-70d5-4f11-86ac-c1d8aeb6aa12), client:
CTX_ID:d6150421-3ac2-4dbc-a665-d619f13dbb15-GRAPH_ID:0-PID:538-HOST:munin-PC_NAME:gv0-client-3-RECON_NO:-0,
error-xlator: gv0-access-control [Operation not permitted]
[2019-08-09 17:06:52.073700] I [dict.c:541:dict_get]
(-->/lib64/libglusterfs.so.0(default_fremovexattr+0xe7) [0x7f47c89461f7]
-->/usr/lib64/glusterfs/6.4/xlator/features/locks.so(+0x1d5dd) [0x7f47b8e8b5dd]
-->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f47c88a70b4] ) 0-dict: !this ||
key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument]
[2019-08-09 17:06:52.222678] I [dict.c:541:dict_get]
(-->/usr/lib64/glusterfs/6.4/xlator/features/worm.so(+0x7241) [0x7f47b8c64241]
-->/usr/lib64/glusterfs/6.4/xlator/features/locks.so(+0x1c219) [0x7f47b8e8a219]
-->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f47c88a70b4] ) 0-dict: !this ||
key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument]
pending frames:
patchset: git://git.gluster.org/glusterfs.git
signal received: 11
time of crash: 
2019-08-09 17:06:52
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 6.4
/lib64/libglusterfs.so.0(+0x26e00)[0x7f47c88b2e00]
/lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f47c88bd804]
/lib64/libc.so.6(+0x36340)[0x7f47c6ef2340]
/usr/lib64/glusterfs/6.4/rpc-transport/socket.so(+0xa4cc)[0x7f47bcc774cc]
/lib64/libglusterfs.so.0(+0x8b4d6)[0x7f47c89174d6]
/lib64/libpthread.so.0(+0x7dd5)[0x7f47c76f2dd5]
/lib64/libc.so.6(clone+0x6d)[0x7f47c6fba02d]
---------


###########################
# backtrace in core dump on first node:
(gdb) bt
#0  0x00007f5f844564cc in socket_event_handler () from
/usr/lib64/glusterfs/6.4/rpc-transport/socket.so
#1  0x00007f5f900f64d6 in event_dispatch_epoll_worker () from
/lib64/libglusterfs.so.0
#2  0x00007f5f8eed1dd5 in start_thread (arg=0x7f5f66ffd700) at
pthread_create.c:307
#3  0x00007f5f8e79902d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) 


###########################
# backtrace in core dump on second node:
(gdb) bt
#0  0x00007f47bcc774cc in socket_event_handler () from
/usr/lib64/glusterfs/6.4/rpc-transport/socket.so
#1  0x00007f47c89174d6 in event_dispatch_epoll_worker () from
/lib64/libglusterfs.so.0
#2  0x00007f47c76f2dd5 in start_thread (arg=0x7f47b1053700) at
pthread_create.c:307
#3  0x00007f47c6fba02d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.