[Bugs] [Bug 1654181] glusterd segmentation fault: glusterd_op_ac_brick_op_failed (event=0x7f44e0e63f40, ctx=0x0) at glusterd-op-sm.c:5606
bugzilla at redhat.com
bugzilla at redhat.com
Wed Nov 28 08:13:23 UTC 2018
https://bugzilla.redhat.com/show_bug.cgi?id=1654181
Sanju <srakonde at redhat.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |ASSIGNED
Assignee|bugs at gluster.org |srakonde at redhat.com
--- Comment #1 from Sanju <srakonde at redhat.com> ---
Description of problem:
Segmentation fault in glusterd.
Version-Release number of selected component (if applicable):
# rpm -qa | grep gluster
gluster-nagios-common-0.2.4-1.el7rhgs.noarch
glusterfs-client-xlators-3.12.2-18.el7rhgs.x86_64
glusterfs-geo-replication-3.12.2-18.el7rhgs.x86_64
glusterfs-3.12.2-18.el7rhgs.x86_64
glusterfs-fuse-3.12.2-18.el7rhgs.x86_64
gluster-nagios-addons-0.2.10-2.el7rhgs.x86_64
vdsm-gluster-4.19.43-2.3.el7rhgs.noarch
pcp-pmda-gluster-4.1.0-0.201805281909.git68ab4b18.el7.x86_64
glusterfs-cli-3.12.2-18.el7rhgs.x86_64
glusterfs-api-3.12.2-18.el7rhgs.x86_64
glusterfs-server-3.12.2-18.el7rhgs.x86_64
libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.7.x86_64
python2-gluster-3.12.2-18.el7rhgs.x86_64
glusterfs-rdma-3.12.2-18.el7rhgs.x86_64
glusterfs-libs-3.12.2-18.el7rhgs.x86_64
glusterfs-debuginfo-3.12.2-18.el7rhgs.x86_64
How reproducible:
unknown. Happened during normal operation
Steps to Reproduce:
1.
2.
3.
Actual results:
Expected results:
Additional info:
Backtrace:
Core was generated by `/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level
INFO'.
Program terminated with signal 11, Segmentation fault.
#0 glusterd_op_ac_brick_op_failed (event=0x7f44e0e63f40, ctx=0x0) at
glusterd-op-sm.c:5606
5606 ret = glusterd_remove_pending_entry (&opinfo.pending_bricks,
ev_ctx->pending_node->node);
Missing separate debuginfos, use: debuginfo-install
bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.170-4.el7.x86_64
elfutils-libs-0.170-4.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64
krb5-libs-1.15.1-19.el7.x86_64 libattr-2.4.46-13.el7.x86_64
libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-12.el7_5.x86_64
libselinux-2.5-12.el7.x86_64 libsepol-2.5-8.1.el7.x86_64
pcre-8.32-17.el7.x86_64 systemd-libs-219-57.el7_5.1.x86_64
xz-libs-5.2.2-1.el7.x86_64
(gdb) bt
#0 glusterd_op_ac_brick_op_failed (event=0x7f44e0e63f40, ctx=0x0) at
glusterd-op-sm.c:5606
#1 0x00007f452e3afc9f in glusterd_op_sm () at glusterd-op-sm.c:8391
#2 0x00007f452e3e357c in __glusterd_commit_op_cbk
(req=req at entry=0x7f4516afc4a0, iov=iov at entry=0x7f4516afc4e0,
count=count at entry=1, myframe=myframe at entry=0x7f4516af5300)
at glusterd-rpc-ops.c:1468
#3 0x00007f452e3e50ea in glusterd_big_locked_cbk (req=0x7f4516afc4a0,
iov=0x7f4516afc4e0, count=1, myframe=0x7f4516af5300, fn=0x7f452e3e3070
<__glusterd_commit_op_cbk>)
at glusterd-rpc-ops.c:223
#4 0x00007f45396d1960 in rpc_clnt_handle_reply
(clnt=clnt at entry=0x55d93a0d4d70, pollin=pollin at entry=0x7f44e0e03e40) at
rpc-clnt.c:778
#5 0x00007f45396d1d03 in rpc_clnt_notify (trans=<optimized out>,
mydata=0x55d93a0d4da0, event=<optimized out>, data=0x7f44e0e03e40) at
rpc-clnt.c:971
#6 0x00007f45396cda73 in rpc_transport_notify (this=this at entry=0x55d93a0d4f40,
event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7f44e0e03e40)
at rpc-transport.c:538
#7 0x00007f452b5db566 in socket_event_poll_in (this=this at entry=0x55d93a0d4f40,
notify_handled=notify_handled at entry=_gf_false) at socket.c:2315
#8 0x00007f452b5de242 in socket_poller (ctx=0x55d93a0d4f40) at socket.c:2590
#9 0x00007f4538768dd5 in start_thread (arg=0x7f45197fa700) at
pthread_create.c:308
#10 0x00007f4538031b3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) p ev_ctx
$1 = (glusterd_op_brick_rsp_ctx_t *) 0x0
(gdb) p *event
$2 = {list = {next = 0x7f44e0e63f40, prev = 0x7f44e0e63f40}, ctx = 0x0, event =
GD_OP_EVENT_RCVD_RJT,
txn_id = "\016\243\267]\037\370Kd\217\027\313\363\212\367\370", <incomplete
sequence \364>}
== glusterd.log:
[2018-10-15 18:47:36.539047] I [MSGID: 106494]
[glusterd-handler.c:3103:__glusterd_handle_cli_profile_volume] 0-management:
Received volume profile req for volume gluster_shared_storage
[2018-10-15 18:47:36.666572] E [MSGID: 106524]
[glusterd-op-sm.c:2104:glusterd_op_stage_stats_volume] 0-glusterd: Profile on
Volume gluster_shared_storage is not started
[2018-10-15 18:48:10.048205] E [MSGID: 106430]
[glusterd-utils.c:565:glusterd_submit_reply] 0-glusterd: Reply submission
failed
[2018-10-15 18:49:35.967695] I [MSGID: 106488]
[glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req
[2018-10-15 18:49:36.431171] I [MSGID: 106494]
[glusterd-handler.c:3103:__glusterd_handle_cli_profile_volume] 0-management:
Received volume profile req for volume gluster_shared_storage
[2018-10-15 18:49:36.600248] E [MSGID: 106524]
[glusterd-op-sm.c:2104:glusterd_op_stage_stats_volume] 0-glusterd: Profile on
Volume gluster_shared_storage is not started
[2018-10-15 18:49:36.600290] E [MSGID: 106301]
[glusterd-op-sm.c:4756:glusterd_op_ac_send_stage_op] 0-management: Staging of
operation 'Volume Profile' failed on localhost : Profile on Volum
e gluster_shared_storage is not started
[2018-10-15 18:49:37.029217] I [MSGID: 106494]
[glusterd-handler.c:3103:__glusterd_handle_cli_profile_volume] 0-management:
Received volume profile req for volume supervole1a01
[2018-10-15 18:49:37.029342] W [glusterd-locks.c:622:glusterd_mgmt_v3_lock]
(-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x258eb)
[0x7f452e3938eb] -->/usr/lib64/glusterfs/3.12.2/
xlator/mgmt/glusterd.so(+0x25148) [0x7f452e393148]
-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe96e6)
[0x7f452e4576e6] ) 0-management: Lock for supervole1a01 held by c1fb8d2e-f
3b7-498a-a11c-c4d79d061f2c
[2018-10-15 18:49:37.029369] E [MSGID: 106119]
[glusterd-handler.c:803:glusterd_op_txn_begin] 0-management: Unable to acquire
lock for supervole1a01
[2018-10-15 18:49:37.611858] I [MSGID: 106494]
[glusterd-handler.c:3103:__glusterd_handle_cli_profile_volume] 0-management:
Received volume profile req for volume supervole1a03
The message "I [MSGID: 106488]
[glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 3 times between [2018-10-15 18:49:35.967695] and
[2018-
10-15 18:49:35.980367]
[2018-10-15 18:50:35.931571] I [MSGID: 106488]
[glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req
[2018-10-15 18:50:37.541527] I [MSGID: 106494]
[glusterd-handler.c:3103:__glusterd_handle_cli_profile_volume] 0-management:
Received volume profile req for volume supervole1a01
[2018-10-15 18:50:37.542017] W [glusterd-locks.c:622:glusterd_mgmt_v3_lock]
(-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x258eb)
[0x7f452e3938eb] -->/usr/lib64/glusterfs/3.12.2/
xlator/mgmt/glusterd.so(+0x25148) [0x7f452e393148]
-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe96e6)
[0x7f452e4576e6] ) 0-management: Lock for supervole1a01 held by c1fb8d2e-f
3b7-498a-a11c-c4d79d061f2c
[2018-10-15 18:50:37.542040] E [MSGID: 106119]
[glusterd-handler.c:803:glusterd_op_txn_begin] 0-management: Unable to acquire
lock for supervole1a01
[2018-10-15 18:50:37.984898] I [MSGID: 106494]
[glusterd-handler.c:3103:__glusterd_handle_cli_profile_volume] 0-management:
Received volume profile req for volume supervole1a03
[2018-10-15 18:50:37.985003] W [glusterd-locks.c:622:glusterd_mgmt_v3_lock]
(-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x258eb)
[0x7f452e3938eb] -->/usr/lib64/glusterfs/3.12.2/
xlator/mgmt/glusterd.so(+0x25148) [0x7f452e393148]
-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe96e6)
[0x7f452e4576e6] ) 0-management: Lock for supervole1a03 held by c1fb8d2e-f
3b7-498a-a11c-c4d79d061f2c
[2018-10-15 18:50:37.985024] E [MSGID: 106119]
[glusterd-handler.c:803:glusterd_op_txn_begin] 0-management: Unable to acquire
lock for supervole1a03
[2018-10-15 18:50:38.106949] I
[glusterd-locks.c:732:gd_mgmt_v3_unlock_timer_cbk] 0-management: In
gd_mgmt_v3_unlock_timer_cbk
[2018-10-15 18:51:08.444708] E [MSGID: 106279]
[glusterd-rpc-ops.c:1387:__glusterd_commit_op_cbk] 0-management: Received
commit RJT from uuid: 721f68c8-5468-4776-bc2a-9f14c4250054
[2018-10-15 18:51:08.444818] E
[glusterd-op-sm.c:5603:glusterd_op_ac_brick_op_failed]
(-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x7557c)
[0x7f452e3e357c] -->/usr/lib64/gluster
fs/3.12.2/xlator/mgmt/glusterd.so(+0x41c9f) [0x7f452e3afc9f]
-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x41576)
[0x7f452e3af576] ) 0-: Assertion failed: ctx
The message "I [MSGID: 106488]
[glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 3 times between [2018-10-15 18:50:35.931571] and
[2018-
10-15 18:50:35.937377]
pending frames:
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
patchset: git://git.gluster.org/glusterfs.git
signal received: 11
time of crash:
2018-10-15 18:51:08
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.12.2
/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xa0)[0x7f4539909cc0]
/lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f4539913c04]
/lib64/libc.so.6(+0x36280)[0x7f4537f69280]
/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x41397)[0x7f452e3af397]
/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x41c9f)[0x7f452e3afc9f]
/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x7557c)[0x7f452e3e357c]
/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x770ea)[0x7f452e3e50ea]
/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0x90)[0x7f45396d1960]
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a3)[0x7f45396d1d03]
/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f45396cda73]
/usr/lib64/glusterfs/3.12.2/rpc-transport/socket.so(+0x7566)[0x7f452b5db566]
/usr/lib64/glusterfs/3.12.2/rpc-transport/socket.so(+0xa242)[0x7f452b5de242]
/lib64/libpthread.so.0(+0x7dd5)[0x7f4538768dd5]
/lib64/libc.so.6(clone+0x6d)[0x7f4538031b3d]
---------
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list