[Bugs] [Bug 1207146] New: BitRot:- bitd crashed
bugzilla at redhat.com
bugzilla at redhat.com
Mon Mar 30 10:18:45 UTC 2015
https://bugzilla.redhat.com/show_bug.cgi?id=1207146
Bug ID: 1207146
Summary: BitRot:- bitd crashed
Product: GlusterFS
Version: mainline
Component: bitrot
Severity: high
Assignee: bugs at gluster.org
Reporter: racpatel at redhat.com
CC: bugs at gluster.org
Docs Contact: bugs at gluster.org
Description of problem:
=======================
bitd crashed.
Version-Release number of selected component (if applicable):
=============================================================
0.803.gitf64666f.el6.x86_64
How reproducible:
=================
Intemittent
Steps to Reproduce:
===================
Don't know exact steps to reproduce.
1. created 2 volumes in cluster and enabled bitrot for that
[root at rhs-client37 ~]# gluster v info BitRot1
Volume Name: BitRot1
Type: Distributed-Replicate
Volume ID: a311984b-5978-4041-91fd-be627c616bea
Status: Started
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: rhs-client44:/pavanbrick6/br1
Brick2: rhs-client44:/pavanbrick6/br2
Brick3: rhs-client44:/pavanbrick6/br3
Brick4: rhs-client44:/pavanbrick6/br4
Brick5: rhs-client44:/pavanbrick6/br5
Brick6: rhs-client44:/pavanbrick6/br6
Options Reconfigured:
features.bitrot: on
performance.open-behind: off
[root at rhs-client37 ~]# gluster v info rac1
Volume Name: rac1
Type: Distribute
Volume ID: d462f6c7-809f-4eb1-9517-7947527c5415
Status: Started
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: rhs-client44:/pavanbrick7/r1
Brick2: rhs-client37:/pavanbrick7/r1
Brick3: rhs-client38:/pavanbrick7/r1
Options Reconfigured:
features.bitrot: on
2. created few files and while verifying bitrot functionality found that on one
of the machine (rhs-client44) bitd crashed
Actual results:
===============
bitd crashed
Additional info:
================
bt:-
#0 0x00007f63fc558434 in gf_changelog_reborp_rpcsvc_notify (rpc=<value
optimized out>, mydata=0x7f63c40016e0,
event=<value optimized out>, data=<value optimized out>) at
gf-changelog-reborp.c:161
161 gf_log (this->name, GF_LOG_WARNING, "failed to
unlink "
Missing separate debuginfos, use: debuginfo-install
glibc-2.12-1.149.el6_6.5.x86_64 keyutils-libs-1.4-5.el6.x86_64
krb5-libs-1.10.3-33.el6.x86_64 libcom_err-1.41.12-21.el6.x86_64
libgcc-4.4.7-11.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64
openssl-1.0.1e-30.el6_6.5.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0 0x00007f63fc558434 in gf_changelog_reborp_rpcsvc_notify (rpc=<value
optimized out>, mydata=0x7f63c40016e0,
event=<value optimized out>, data=<value optimized out>) at
gf-changelog-reborp.c:161
#1 0x0000003712c09ea4 in rpcsvc_program_notify (trans=<value optimized out>,
mydata=<value optimized out>,
event=<value optimized out>, data=0x7f63b8039900) at rpcsvc.c:327
#2 rpcsvc_accept (trans=<value optimized out>, mydata=<value optimized out>,
event=<value optimized out>, data=0x7f63b8039900)
at rpcsvc.c:350
#3 rpcsvc_notify (trans=<value optimized out>, mydata=<value optimized out>,
event=<value optimized out>, data=0x7f63b8039900)
at rpcsvc.c:775
#4 0x0000003712c0b7c8 in rpc_transport_notify (this=<value optimized out>,
event=<value optimized out>, data=<value optimized out>)
at rpc-transport.c:543
#5 0x00007f63fd7d032e in socket_server_event_handler (fd=<value optimized
out>, idx=<value optimized out>, data=0x7f63c4008a60,
poll_in=<value optimized out>, poll_out=<value optimized out>,
poll_err=<value optimized out>) at socket.c:2820
#6 0x000000371247cee0 in event_dispatch_epoll_handler (data=0x7f63f8002250) at
event-epoll.c:572
#7 event_dispatch_epoll_worker (data=0x7f63f8002250) at event-epoll.c:674
#8 0x0000003ac86079d1 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003ac7ee88fd in clone () from /lib64/libc.so.6
log snippet:-
[2015-03-26 13:55:57.601516] E [socket.c:823:__socket_server_bind]
0-socket.gfchangelog: binding to failed: Address already in use
[2015-03-26 13:55:57.601534] E [socket.c:826:__socket_server_bind]
0-socket.gfchangelog: Port is already in use
[2015-03-26 13:55:57.601546] W [rpcsvc.c:1583:rpcsvc_transport_create]
0-rpc-service: listening on transport failed
pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 7
time of crash:
2015-03-26 13:55:57
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.7dev
[2015-03-26 13:55:57.601685] E
[gf-changelog.c:543:gf_changelog_register_generic] 0-gfchangelog: Error
registering with changelog xlator
pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
[2015-03-26 13:55:57.601873] E [bit-rot.c:1018:br_enact_signer] 0-bit-rot:
Register to changelog failed [Reason: Address already in use]
[2015-03-26 13:55:57.602010] E [bit-rot.c:1166:br_handle_events] 0-bit-rot:
failed to connect to the child (subvolume: BitRot1-client-0)
/usr/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb6)[0x7f92d80e1126]
/usr/lib64/libglusterfs.so.0(gf_print_trace+0x33f)[0x7f92d80fcd5f]
/lib64/libc.so.6[0x3ac7e326a0]
/usr/lib64/libgfchangelog.so.0(gf_changelog_reborp_rpcsvc_notify+0xd4)[0x7f92ccc7a434]
/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0xa4)[0x7f92d7eaeea4]
/usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7f92d7eb07c8]
/usr/lib64/glusterfs/3.7dev/rpc-transport/socket.so(+0x832e)[0x7f92cdef232e]
/usr/lib64/libglusterfs.so.0(+0x7cee0)[0x7f92d813bee0]
/lib64/libpthread.so.0[0x3ac86079d1]
/lib64/libc.so.6(clone+0x6d)[0x3ac7ee88fd]
[2015-03-27 06:03:28.691590] I [MSGID: 100030] [glusterfsd.c:2288:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs ver
sion 3.7dev (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/bitd
-p /var/lib/glusterd/bitd/run/bitd.pid -l /var/log/glust
erfs/bitd.log -S /var/run/gluster/a2f42fcde27fccc09d3f5318ab8b9ed2.socket)
[2015-03-27 06:03:28.868332] I [event-epoll.c:629:event_dispatch_epoll_worker]
0-epoll: Started thread with index 1
[2015-03-27 06:03:30.205733] I [bit-rot.c:1419:init] 0-bit-rot: bit-rot xlator
loaded in "SIGNER" mode
[2015-03-27 06:03:30.205955] I [event-epoll.c:629:event_dispatch_epoll_worker]
0-epoll: Started thread with index 2
[2015-03-27 06:03:30.208236] I [client.c:2389:notify] 0-BitRot1-client-0:
parent translators are ready, attempting connect on transpor
t
[2015-03-27 06:03:30.212406] I [client.c:2389:notify] 0-BitRot1-client-1:
parent translators are ready, attempting connect on transpor
t
[2015-03-27 06:03:30.212669] I [rpc-clnt.c:1806:rpc_clnt_reconfig]
0-BitRot1-client-0: changing port to 49154 (from 0)
[2015-03-27 06:03:30.216228] I [client.c:2389:notify] 0-BitRot1-client-2:
parent translators are ready, attempting connect on transpor
t
[2015-03-27 06:03:30.219695] I
[client-handshake.c:1414:select_server_supported_programs] 0-BitRot1-client-0:
Using Program GlusterFS
3.3, Num (1298437), Version (330)
[2015-03-27 06:03:30.219788] I [rpc-clnt.c:1806:rpc_clnt_reconfig]
0-BitRot1-client-1: changing port to 49155 (from 0)
[2015-03-27 06:03:30.222783] I [client.c:2389:notify] 0-BitRot1-client-3:
parent translators are ready, attempting connect on transpor
t
[2015-03-27 06:03:30.226078] I [client-handshake.c:1202:client_setvolume_cbk]
0-BitRot1-client-0: Connected to BitRot1-client-0, attac
hed to remote volume '/pavanbrick6/br1'.
[2015-03-27 06:03:30.226102] I [client-handshake.c:1212:client_setvolume_cbk]
0-BitRot1-client-0: Server and Client lk-version numbers
are not same, reopening the fds
[2015-03-27 06:03:30.227889] I
[client-handshake.c:187:client_set_lk_version_cbk] 0-BitRot1-client-0: Server
lk version = 1
[2015-03-27 06:03:30.228003] I
[client-handshake.c:1414:select_server_supported_programs] 0-BitRot1-client-1:
Using Program GlusterFS
3.3, Num (1298437), Version (330)
[2015-03-27 06:03:30.228077] I [rpc-clnt.c:1806:rpc_clnt_reconfig]
0-BitRot1-client-2: changing port to 49156 (from 0)
[2015-03-27 06:03:30.229337] I [client.c:2389:notify] 0-BitRot1-client-4:
parent translators are ready, attempting connect on transpor
t
[2015-03-27 06:03:30.232713] I [client-handshake.c:1202:client_setvolume_cbk]
0-BitRot1-client-1: Connected to BitRot1-client-1, attac
hed to remote volume '/pavanbrick6/br2'.
[2015-03-27 06:03:30.232746] I [client-handshake.c:1212:client_setvolume_cbk]
0-BitRot1-client-1: Server and Client lk-version numbers
are not same, reopening the fds
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
You are the Docs Contact for the bug.
More information about the Bugs
mailing list