[Gluster-users] Failed rebalance - lost files, inaccessible files, permission issues

[Shawn Heisey]

On 11/9/2013 2:39 AM, Shawn Heisey wrote:
> They are from the same log file - the one that I put on my dropbox
> account and linked in the original message.  They are consecutive log
> entries.

Further info from our developer that is looking deeper into these problems:



------------
Ouch.  I know why the rebalance stopped.  The host simply ran out of 
memory.  From the messages file:

Nov  2 21:55:30 slc01dfs001a kernel: VFS: file-max limit 2438308 reached
Nov  2 21:55:31 slc01dfs001a kernel: automount invoked oom-killer: 
gfp_mask=0xd0, order=1, oom_adj=0, oom_score_adj=0
Nov  2 21:55:31 slc01dfs001a kernel: automount cpuset=/ mems_allowed=0
Nov  2 21:55:31 slc01dfs001a kernel: Pid: 2810, comm: automount Not 
tainted 2.6.32-358.2.1.el6.centos.plus.x86_64 #1

That "file max limit" line actually goes back to the beginning of Nov. 
2, and happened on all four hosts.  It is because of a file descriptor 
leak and was fixed in 3.3.2: 
https://bugzilla.redhat.com/show_bug.cgi?id=928631

This is unconnected to the file corruption/loss which started much 
earlier.  I'm still trying to understand this part.  I noticed that 
three of the hosts reported successful rebalancing on the same day we 
started losing files.  I am not sure how rebalancing was distributed 
among the hosts, and if the load on the other hosts was enough to keep 
things stable until they stopped.
------------



I gather that we should be at least on 3.3.2, but I suspect that a 
number of other bugs might be a problem unless we go to 3.4.1.  The 
rebalance status output is below.  All hosts except "localhost" on this 
status were reading "completed" a very short time after I started the 
rebalance.  The "localhost" line continued to increment until the 
rebalance died four days after starting.

[root at slc01dfs001a ~]# gluster volume rebalance mdfs status
                                     Node Rebalanced-files          size 
       scanned      failures         status
                                ---------      -----------   ----------- 
   -----------   -----------   ------------
                                localhost          1121514         1.5TB 
       9020514       1777661         failed
                                slc01nas1                0        0Bytes 
      13638699             0      completed
                             slc01dfs002a                0        0Bytes 
      13638699             1      completed
                             slc01dfs001b                0        0Bytes 
      13638699             0      completed
                             slc01dfs002b                0        0Bytes 
      13638700             0      completed
                                slc01nas2                0        0Bytes 
      13638699             0      completed

Thanks,
Shawn