[Gluster-devel] Feature review: Improved rebalance performance

[Xavier Hernandez]

Hi Shyam,

On Thursday 26 June 2014 14:41:13 Shyamsundar Ranganathan wrote:
> It also touches upon a rebalance on access like mechanism where we could
> potentially, move data out of existing bricks to a newer brick faster, in
> the case of brick addition, and vice versa for brick removal, and heal the
> rest of the data on access.
> 
Will this "rebalance on access" feature be enabled always or only during a 
brick addition/removal to move files that do not go to the affected brick 
while the main rebalance is populating or removing files from the brick ?

I like all the proposed ideas. I think they would improve the performance of 
the rebalance operation considerably. Probably we will need to define some 
policies to limit the amount of bandwidth that rebalance is allowed to use and 
at which hours, but this can be determined later.

I would also consider using index or changelog xlators to track renames and 
let rebalance consume it. Currently a file or directory rename makes that 
files correctly placed in the right brick need to be moved to another brick. A 
full rebalance crawling all the file system seems too expensive for this kind 
of local changes (the effects of this are orders of magnitude smaller than 
adding or removing a brick). Having a way to list pending moves due to rename 
without scanning all the file system would be great.

Another thing to consider for future versions is to modify the current DHT to 
a consistent hashing and even the hash value (using gfid instead of a hash of 
the name would solve the rename problem). The consistent hashing would 
drastically reduce the number of files that need to be moved and already 
solves some of the current problems. This change needs a lot of thinking 
though.

Xavi