[Gluster-devel] 3.1.3 crash on volume start

Mark "Naoki" Rogers mrogers at valuecommerce.co.jp
Tue Apr 12 07:07:19 UTC 2011


I had an interesting situation with a (3x3) distribute+replicate setup. 
One of the machines died (memory problem, hard lock) and gluster didn't 
exactly like it (although that's another story).

Anyway I decided to keep on truckin' and use a 5 node distribute (since 
this is mainly a network test anyway) and the volume creation was ok but 
startup failed with the old "cli op: 16" message (note: would be nice if 
that actually said lock file issue if that's what it is - seems to be 
solved by a restart anyway).

I restarted glusterd but everytime I try and "volume start" it just 
crashes immediately:

[2011-04-12 15:36:39.861099] I 
[glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 
af4007ac-804c-433f-a6e9-0a7d6cafe432, host: pdbsearch12
[2011-04-12 15:36:39.861123] I 
[glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 
5e4ef932-cca7-4440-a8e3-4415b84c2307, host: pdbsearch13
[2011-04-12 15:36:39.861148] I 
[glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 
df6c1fb1-cf5d-4a6b-ba15-fb26319e37db, host: pdbsearch14
[2011-04-12 15:36:39.861172] I 
[glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 
2cc51957-2c92-4d24-a53c-c7ded19e80b7, host: pdbsearch15
[2011-04-12 15:36:39.861695] I 
[glusterd-rpc-ops.c:626:glusterd3_1_friend_update_cbk] 0-glusterd: 
Received ACC from uuid:
[2011-04-12 15:36:39.861737] I 
[glusterd-rpc-ops.c:626:glusterd3_1_friend_update_cbk] 0-glusterd: 
Received ACC from uuid:
[2011-04-12 15:36:39.861776] I 
[glusterd-rpc-ops.c:626:glusterd3_1_friend_update_cbk] 0-glusterd: 
Received ACC from uuid:
[2011-04-12 15:36:39.861809] I 
[glusterd-rpc-ops.c:626:glusterd3_1_friend_update_cbk] 0-glusterd: 
Received ACC from uuid:
[2011-04-12 15:36:45.725525] I 
[glusterd-handler.c:1088:glusterd_handle_cli_start_volume] 0-glusterd: 
Received start vol reqfor volume distribute
[2011-04-12 15:36:45.725677] I [glusterd-utils.c:242:glusterd_lock] 
0-glusterd: Cluster lock held by 1b6130e7-f83e-4ec6-90f7-46706dbffc4f
[2011-04-12 15:36:45.725702] I 
[glusterd-handler.c:825:glusterd_op_txn_begin] 0-glusterd: Acquired 
local lock
pending frames:

patchset: v3.1.3
signal received: 11
time of crash: 2011-04-12 15:36:45
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.1.3
/lib64/libc.so.6(+0x33140)[0x7fc7e2337140]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/mgmt/glusterd.so(+0x2a3e9)[0x7fc7e0b853e9]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/mgmt/glusterd.so(glusterd_op_sm+0xee)[0x7fc7e0b8b8ce]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/mgmt/glusterd.so(glusterd_handle_cli_start_volume+0x8a)[0x7fc7e0b8197a]
/opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x271)[0x7fc7e2cde061]
/opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpcsvc_notify+0x93)[0x7fc7e2cde183]
/opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpc_transport_notify+0x2d)[0x7fc7e2cde77d]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/rpc-transport/socket.so(socket_event_poll_in+0x34)[0x7fc7e09545b4]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/rpc-transport/socket.so(socket_event_handler+0x19d)[0x7fc7e095476d]
/opt/glusterfs/3.1.3/lib64/libglusterfs.so.0(+0x39422)[0x7fc7e2f23422]
/opt/glusterfs/3.1.3/sbin/glusterd(main+0x245)[0x405235]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7fc7e2322e5d]
/opt/glusterfs/3.1.3/sbin/glusterd[0x402ea9]

Restarted all nodes and tried again:

patchset: v3.1.3
signal received: 11
time of crash: 2011-04-12 15:47:44
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.1.3
/lib64/libc.so.6(+0x33140)[0x7fb8f65bb140]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/mgmt/glusterd.so(+0x2a3e9)[0x7fb8f4e093e9]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/mgmt/glusterd.so(glusterd_op_sm+0xee)[0x7fb8f4e0f8ce]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/mgmt/glusterd.so(glusterd_handle_cli_start_volume+0x8a)[0x7fb8f4e0597a]
/opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x271)[0x7fb8f6f62061]
/opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpcsvc_notify+0x93)[0x7fb8f6f62183]
/opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpc_transport_notify+0x2d)[0x7fb8f6f6277d]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/rpc-transport/socket.so(socket_event_poll_in+0x34)[0x7fb8f4bd85b4]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/rpc-transport/socket.so(socket_event_handler+0x19d)[0x7fb8f4bd876d]
/opt/glusterfs/3.1.3/lib64/libglusterfs.so.0(+0x39422)[0x7fb8f71a7422]
/opt/glusterfs/3.1.3/sbin/glusterd(main+0x245)[0x405235]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7fb8f65a6e5d]
/opt/glusterfs/3.1.3/sbin/glusterd[0x402ea9]


I'm going to probably give a 3.1 (qa4) and/or 3.2 (qa11) release a try 
just in the meantime.
Cheers.




More information about the Gluster-devel mailing list