[Gluster-users] Extremely high load after 100% full bricks

[Dan Bretherton]

Dear All-
A replicated pair of servers in my GlusterFS 3.3.0 cluster have been 
experiencing extremely high load for the past few days after a 
replicated brick pair became 100% full.  The GlusterFS related load on 
one of the servers was fluctuating at around 60, and this high load 
would swap to the other server periodically.  When I noticed the full 
bricks I quickly extended the volume by creating new bricks on another 
server, and manually moved some data off the full bricks to create space 
for write operations.  The fix-layout operation seemed to start normally 
but the load then increased even further.  The server with the high load 
(then up to about 80) became very slow to respond and I noticed a lot of 
errors in the VOLNAME-rebalance.log files like the following.

[2012-10-22 00:35:52.070364] W 
[socket.c:1512:__socket_proto_state_machine] 0-atmos-client-10: reading 
from socket failed. Error (Transport endpoint is not connected), peer 
(192.171.166.92:24052)
[2012-10-22 00:35:52.070446] E [rpc-clnt.c:373:saved_frames_unwind] 
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xe7) [0x2b3fd905c547] 
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb2) 
[0x2b3fd905bf42] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) 
[0x2b3fd905bbfe]))) 0-atmos-client-10: forced unwinding frame 
type(GlusterFS 3.1) op(INODELK(29)) called at 2012-10-22 00:35:45.454529 
(xid=0x285951x)

There have also been occasional errors like the following, referring to 
the pair of bricks that became 100% full.

[2012-10-22 01:32:52.827044] W [client3_1-fops.c:5517:client3_1_readdir] 
0-atmos-client-15:  (00000000-0000-0000-0000-000000000000) remote_fd is 
-1. EBADFD
[2012-10-22 09:49:21.103066] W 
[client3_1-fops.c:5628:client3_1_readdirp] 0-atmos-client-14:  
(00000000-0000-0000-0000-000000000000) remote_fd is -1. EBADFD

The log files from the bricks that were 100% full have a lot of these 
errors in, from the period after I freed up some space on them.

[2012-10-22 00:40:56.246075] E [server.c:176:server_submit_reply] 
(-->/usr/lib64/libglusterfs.so.0(default_inodelk_cbk+0xa4) 
[0x361da23e84] 
(-->/usr/lib64/glusterfs/3.3.0/xlator/debug/io-stats.so(io_stats_inodelk_cbk+0xd8) 
[0x2aaaabd74d48] 
(-->/usr/lib64/glusterfs/3.3.0/xlator/protocol/server.so(server_inodelk_cbk+0x10b) 
[0x2aaaabf9742b]))) 0-: Reply submission failed
[2012-10-22 00:40:56.246117] I 
[server-helpers.c:629:server_connection_destroy] 0-atmos-server: 
destroyed connection of 
bdan10.nerc-essc.ac.uk-13609-2012/10/21-23:04:53:323865-atmos-client-15-0

All these errors have only occurred on the replicated pair of servers 
that had suffered from 100% full bricks.  I don't know if the errors are 
being caused by the high load (resulting in poor communication with 
other peers for example) or if the high load is the result of 
replication and/or distribution errors.  I have tried various things to 
bring the load down, including un-mounting the volume and stopping the 
fix-layout operation, but the only thing that works is stopping the 
volume. Obviously I can't do that for long because people need to use 
the data, but with the load as high as it is data access is very slow 
and users are experiencing a lot of temporary I/O errors.   Bricks from 
several volumes are on those servers so everybody in the department is 
being affected by this problem.  I thought at first that the load was 
being caused by self-heal operations fixing errors caused by write 
failures that occurred when the bricks were full, but it is glusterfs 
threads that are causing the high load, not glustershd.

Can anyone suggest a way to bring the load down so people can access the 
data properly again?  Also, can I trust GlusterFS to eventually 
self-heal the errors causing the above error messages?

Regards,
-Dan.