[Gluster-users] Gluster 3.5 wont start if disks are full

[Franco Broi]

Hi

I have a 2 node test system, on one of the nodes the volume disks are
full and gluster will not start after an upgrade to 3.5. Previous
version was  3.4.3. I am assuming that gluster wont start because the
disks are full although I can't see anything in the log that might
suggest that that is the case.

Here's a grep of the errors from the log file:

grep ' E ' /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
[2014-04-28 09:19:24.616070] E [rpc-clnt.c:368:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13d) [0x7fb38cd01c8d] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x7fb38cd017f3] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fb38cd0170e]))) 0-management: forced unwinding frame type(Gluster MGMT Handshake) op(MGMT-VERS(1)) called at 2014-04-28 09:19:24.614255 (xid=0x28x)
[2014-04-28 09:19:24.616084] E [glusterd-handshake.c:862:__glusterd_mgmt_hndsk_version_cbk] 0-management: Error through RPC layer, retry again later
[2014-04-28 09:19:27.234725] E [socket.c:2157:socket_connect_finish] 0-management: connection to 192.168.34.6:24007 failed (Connection refused)
[2014-04-28 09:21:47.063528] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.5.0/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2014-04-28 09:21:47.104301] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2014-04-28 09:21:47.104329] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2014-04-28 09:21:47.104342] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2
[2014-04-28 09:21:47.104353] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-3
[2014-04-28 09:21:47.104364] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-4
[2014-04-28 09:21:47.104375] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-5
[2014-04-28 09:21:47.104386] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-6
[2014-04-28 09:21:47.104397] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-7
[2014-04-28 09:22:09.971037] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.5.0/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2014-04-28 09:22:10.185569] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2014-04-28 09:22:10.185611] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2014-04-28 09:22:10.185624] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2
[2014-04-28 09:22:10.185634] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-3
[2014-04-28 09:22:10.185645] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-4
[2014-04-28 09:22:10.185655] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-5
[2014-04-28 09:22:10.185666] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-6
[2014-04-28 09:22:10.185677] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-7
[2014-04-28 09:22:13.519169] E [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/61ba1ffcacede69af7df2ca0a0000f0c.socket error: Permission denied


Status of volume: gluster_test
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick nas5-10g:/data17/gvol				N/A	N	N/A
Brick nas5-10g:/data18/gvol				N/A	N	N/A
Brick nas5-10g:/data19/gvol				N/A	N	N/A
Brick nas5-10g:/data20/gvol				N/A	N	N/A
Brick nas6-10g:/data21/gvol				49152	Y	447
Brick nas6-10g:/data22/gvol				49153	Y	453
Brick nas6-10g:/data23/gvol				49154	Y	460
Brick nas6-10g:/data24/gvol				49155	Y	465
NFS Server on localhost					2049	Y	473
NFS Server on nas5-10g					2049	Y	20880
 
Task Status of Volume gluster_test
------------------------------------------------------------------------------
There are no active volume tasks

Cheers,