[Gluster-devel] 3.1.3 crash on volume start

Tue Apr 12 07:36:37 UTC 2011

Hi -
> I'm going to probably give a 3.1 (qa4) and/or 3.2 (qa11) release a try

Please try with stable version(3.1.x) instead of (3.1.xQAx) QA releases.
Also provide the output of command - "gluster volume info" along with server and client logs.

-- 
----
Cheers,
Lakshmipathi.G
FOSS Programmer.

----- Original Message -----
From: "Mark \"Naoki\" Rogers" <mrogers at valuecommerce.co.jp>
To: gluster-devel at nongnu.org
Sent: Tuesday, April 12, 2011 12:48:34 PM
Subject: Re: [Gluster-devel] 3.1.3 crash on volume start

Sadly no go with 3.1.4qa3:

patchset: v3.1.4qa3
signal received: 11
time of crash: 2011-04-12 16:16:34
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.1.4qa3
/lib64/libc.so.6(+0x33140)[0x7f68d48c8140]
/opt/glusterfs/3.1.4qa3/lib64/glusterfs/3.1.4qa3/xlator/mgmt/glusterd.so(+0x3323b)[0x7f68d311e23b]
/opt/glusterfs/3.1.4qa3/lib64/glusterfs/3.1.4qa3/xlator/mgmt/glusterd.so(glusterd_op_sm+0xee)[0x7f68d311d6ce]
/opt/glusterfs/3.1.4qa3/lib64/glusterfs/3.1.4qa3/xlator/mgmt/glusterd.so(glusterd_handle_cli_start_volume+0x8f)[0x7f68d310c91f]
/opt/glusterfs/3.1.4qa3/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x28f)[0x7f68d526e01f]
/opt/glusterfs/3.1.4qa3/lib64/libgfrpc.so.0(rpcsvc_notify+0x123)[0x7f68d526e653]
/opt/glusterfs/3.1.4qa3/lib64/libgfrpc.so.0(rpc_transport_notify+0x2d)[0x7f68d527221d]
/opt/glusterfs/3.1.4qa3/lib64/glusterfs/3.1.4qa3/rpc-transport/socket.so(socket_event_poll_in+0x34)[0x7f68d2ee3b44]
/opt/glusterfs/3.1.4qa3/lib64/glusterfs/3.1.4qa3/rpc-transport/socket.so(socket_event_handler+0xb3)[0x7f68d2ee3de3]
/opt/glusterfs/3.1.4qa3/lib64/libglusterfs.so.0(+0x37d42)[0x7f68d54b2d42]
/opt/glusterfs/3.1.4qa3/sbin/glusterd(main+0x263)[0x4054b3]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f68d48b3e5d]
/opt/glusterfs/3.1.4qa3/sbin/glusterd[0x402db9]

On 04/12/2011 04:07 PM, Mark "Naoki" Rogers wrote:
> I had an interesting situation with a (3x3) distribute+replicate setup.
> One of the machines died (memory problem, hard lock) and gluster didn't
> exactly like it (although that's another story).
>
> Anyway I decided to keep on truckin' and use a 5 node distribute (since
> this is mainly a network test anyway) and the volume creation was ok but
> startup failed with the old "cli op: 16" message (note: would be nice if
> that actually said lock file issue if that's what it is - seems to be
> solved by a restart anyway).
>
> I restarted glusterd but everytime I try and "volume start" it just
> crashes immediately:
>
> [2011-04-12 15:36:39.861099] I
> [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid:
> af4007ac-804c-433f-a6e9-0a7d6cafe432, host: pdbsearch12
> [2011-04-12 15:36:39.861123] I
> [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid:
> 5e4ef932-cca7-4440-a8e3-4415b84c2307, host: pdbsearch13
> [2011-04-12 15:36:39.861148] I
> [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid:
> df6c1fb1-cf5d-4a6b-ba15-fb26319e37db, host: pdbsearch14
> [2011-04-12 15:36:39.861172] I
> [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid:
> 2cc51957-2c92-4d24-a53c-c7ded19e80b7, host: pdbsearch15
> [2011-04-12 15:36:39.861695] I
> [glusterd-rpc-ops.c:626:glusterd3_1_friend_update_cbk] 0-glusterd:
> Received ACC from uuid:
> [2011-04-12 15:36:39.861737] I
> [glusterd-rpc-ops.c:626:glusterd3_1_friend_update_cbk] 0-glusterd:
> Received ACC from uuid:
> [2011-04-12 15:36:39.861776] I
> [glusterd-rpc-ops.c:626:glusterd3_1_friend_update_cbk] 0-glusterd:
> Received ACC from uuid:
> [2011-04-12 15:36:39.861809] I
> [glusterd-rpc-ops.c:626:glusterd3_1_friend_update_cbk] 0-glusterd:
> Received ACC from uuid:
> [2011-04-12 15:36:45.725525] I
> [glusterd-handler.c:1088:glusterd_handle_cli_start_volume] 0-glusterd:
> Received start vol reqfor volume distribute
> [2011-04-12 15:36:45.725677] I [glusterd-utils.c:242:glusterd_lock]
> 0-glusterd: Cluster lock held by 1b6130e7-f83e-4ec6-90f7-46706dbffc4f
> [2011-04-12 15:36:45.725702] I
> [glusterd-handler.c:825:glusterd_op_txn_begin] 0-glusterd: Acquired
> local lock
> pending frames:
>
> patchset: v3.1.3
> signal received: 11
> time of crash: 2011-04-12 15:36:45
> configuration details:
> argp 1
> backtrace 1
> dlfcn 1
> fdatasync 1
> libpthread 1
> llistxattr 1
> setfsid 1
> spinlock 1
> epoll.h 1
> xattr.h 1
> st_atim.tv_nsec 1
> package-string: glusterfs 3.1.3
> /lib64/libc.so.6(+0x33140)[0x7fc7e2337140]
> /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/mgmt/glusterd.so(+0x2a3e9)[0x7fc7e0b853e9]
> /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/mgmt/glusterd.so(glusterd_op_sm+0xee)[0x7fc7e0b8b8ce]
> /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/mgmt/glusterd.so(glusterd_handle_cli_start_volume+0x8a)[0x7fc7e0b8197a]
> /opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x271)[0x7fc7e2cde061]
> /opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpcsvc_notify+0x93)[0x7fc7e2cde183]
> /opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpc_transport_notify+0x2d)[0x7fc7e2cde77d]
> /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/rpc-transport/socket.so(socket_event_poll_in+0x34)[0x7fc7e09545b4]
> /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/rpc-transport/socket.so(socket_event_handler+0x19d)[0x7fc7e095476d]
> /opt/glusterfs/3.1.3/lib64/libglusterfs.so.0(+0x39422)[0x7fc7e2f23422]
> /opt/glusterfs/3.1.3/sbin/glusterd(main+0x245)[0x405235]
> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7fc7e2322e5d]
> /opt/glusterfs/3.1.3/sbin/glusterd[0x402ea9]
>
> Restarted all nodes and tried again:
>
> patchset: v3.1.3
> signal received: 11
> time of crash: 2011-04-12 15:47:44
> configuration details:
> argp 1
> backtrace 1
> dlfcn 1
> fdatasync 1
> libpthread 1
> llistxattr 1
> setfsid 1
> spinlock 1
> epoll.h 1
> xattr.h 1
> st_atim.tv_nsec 1
> package-string: glusterfs 3.1.3
> /lib64/libc.so.6(+0x33140)[0x7fb8f65bb140]
> /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/mgmt/glusterd.so(+0x2a3e9)[0x7fb8f4e093e9]
> /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/mgmt/glusterd.so(glusterd_op_sm+0xee)[0x7fb8f4e0f8ce]
> /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/mgmt/glusterd.so(glusterd_handle_cli_start_volume+0x8a)[0x7fb8f4e0597a]
> /opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x271)[0x7fb8f6f62061]
> /opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpcsvc_notify+0x93)[0x7fb8f6f62183]
> /opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpc_transport_notify+0x2d)[0x7fb8f6f6277d]
> /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/rpc-transport/socket.so(socket_event_poll_in+0x34)[0x7fb8f4bd85b4]
> /opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/rpc-transport/socket.so(socket_event_handler+0x19d)[0x7fb8f4bd876d]
> /opt/glusterfs/3.1.3/lib64/libglusterfs.so.0(+0x39422)[0x7fb8f71a7422]
> /opt/glusterfs/3.1.3/sbin/glusterd(main+0x245)[0x405235]
> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7fb8f65a6e5d]
> /opt/glusterfs/3.1.3/sbin/glusterd[0x402ea9]
>
>
> I'm going to probably give a 3.1 (qa4) and/or 3.2 (qa11) release a try
> just in the meantime.
> Cheers.
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel

_______________________________________________
Gluster-devel mailing list
Gluster-devel at nongnu.org
http://lists.nongnu.org/mailman/listinfo/gluster-devel