[Gluster-devel] Feature review: Improved rebalance performance

[Joe Julian]

On 06/30/2014 02:00 AM, Xavier Hernandez wrote:
> Hi Shyam,
>
> On Thursday 26 June 2014 14:41:13 Shyamsundar Ranganathan wrote:
>> It also touches upon a rebalance on access like mechanism where we could
>> potentially, move data out of existing bricks to a newer brick faster, in
>> the case of brick addition, and vice versa for brick removal, and heal the
>> rest of the data on access.
>>
> Will this "rebalance on access" feature be enabled always or only during a
> brick addition/removal to move files that do not go to the affected brick
> while the main rebalance is populating or removing files from the brick ?
>
> I like all the proposed ideas. I think they would improve the performance of
> the rebalance operation considerably. Probably we will need to define some
> policies to limit the amount of bandwidth that rebalance is allowed to use and
> at which hours, but this can be determined later.
>
> I would also consider using index or changelog xlators to track renames and
> let rebalance consume it. Currently a file or directory rename makes that
> files correctly placed in the right brick need to be moved to another brick. A
> full rebalance crawling all the file system seems too expensive for this kind
> of local changes (the effects of this are orders of magnitude smaller than
> adding or removing a brick). Having a way to list pending moves due to rename
> without scanning all the file system would be great.
>
> Another thing to consider for future versions is to modify the current DHT to
> a consistent hashing and even the hash value (using gfid instead of a hash of
> the name would solve the rename problem). The consistent hashing would
> drastically reduce the number of files that need to be moved and already
> solves some of the current problems. This change needs a lot of thinking
> though.
>
Besides bandwidth limits, there also needs to be monitors on brick 
latency. We don't want so many queued iops that operating performance is 
impacted.