[Gluster-users] glusterd 3.2.7 crash on start

[Qaiser Shah (Banking Division)]

Hi
I have a problem with glusterd 3.2.7.  I have two nodes each configured with one brick in a replicate configuration.  Both have been running for 9 months.  One node recently started to crash with kernel core dump. The issue was a corrupt ext4 filesystem on the glusterfs brick.  I unmounted the volume and fixed the filesystem.  Now I have problems starting glusterd:

[2014-06-19 10:20:33.526714] I [glusterfsd.c:1493:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.
2.7
[2014-06-19 10:20:33.538144] I [glusterd.c:550:init] 0-management: Using /var/lib/glusterd as working directory
[2014-06-19 10:20:33.539148] E [rpc-transport.c:677:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.2.7/rpc-tra
nsport/rdma.so: cannot open shared object file: No such file or directory
[2014-06-19 10:20:33.539175] E [rpc-transport.c:681:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transpor
t-type 'rdma' is not valid or not found on this machine
[2014-06-19 10:20:33.539183] W [rpcsvc.c:1288:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the t
ransport failed
[2014-06-19 10:20:33.539263] I [glusterd.c:88:glusterd_uuid_init] 0-glusterd: retrieved UUID: 8b6e691c-e4c1-4107-944f-fee9c
fb55269
[2014-06-19 10:20:34.854729] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2014-06-19 10:20:34.854778] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2014-06-19 10:20:35.104082] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0
[2014-06-19 10:20:35.107779] E [common-utils.c:125:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not know
n)
[2014-06-19 10:20:35.107843] E [name.c:253:af_inet_client_get_remote_sockaddr] 0-management: DNS resolution failed on host
all
pending frames:

patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2014-06-19 10:20:35
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.2.7
/lib64/libc.so.6(+0x32920)[0x7f10fe1ff920]
/usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_friend_sm+0x25)[0x7f10fc761055]
/usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_peer_rpc_notify+0x1d6)[0x7f10fc749026]
/usr/lib64/libgfrpc.so.0(rpc_clnt_start+0x17)[0x7f10feba38b7]
/usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_rpc_create+0x108)[0x7f10fc749f18]
/usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_friend_add+0x33c)[0x7f10fc74a30c]
/usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_store_retrieve_peers+0x4d6)[0x7f10fc7932f6]
/usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(glusterd_restore+0x78)[0x7f10fc794cb8]
/usr/lib64/glusterfs/3.2.7/xlator/mgmt/glusterd.so(init+0xbec)[0x7f10fc74884c]
/usr/lib64/libglusterfs.so.0(xlator_init+0x52)[0x7f10fedc4af2]
/usr/lib64/libglusterfs.so.0(glusterfs_graph_init+0x31)[0x7f10fedf57a1]
/usr/lib64/libglusterfs.so.0(glusterfs_graph_activate+0x88)[0x7f10fedf5938]
/usr/sbin/glusterd(glusterfs_process_volfp+0xd5)[0x404b35]
/usr/sbin/glusterd(glusterfs_volumes_init+0x145)[0x404d45]
/usr/sbin/glusterd(main+0x295)[0x405e85]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f10fe1ebcdd]
/usr/sbin/glusterd[0x4036c9]


Any pointers on solving this would be helpful.

--------------------------------------------------
PLEASE CONSIDER THE ENVIRONMENT BEFORE PRINTING THIS E-MAIL

Close Brothers Limited (Company No. 195626) and Close Brothers Group plc (Company No. 520241) are registered in England and Wales, and their registered office is 10 Crown Place, London EC2A 4FT.

Close Brothers Limited is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and Prudential Regulation Authority. Firm reference number 124750.
 
The information in this e-mail and any files transmitted with it is confidential and may be legally privileged. It is intended solely for the addressee and others authorised to receive it. If you are not the intended recipient, any disclosure, copying, distribution or action taken in reliance on its contents isprohibited and may be unlawful.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140619/3217ad74/attachment.html>