[Gluster-users] Rebalancing after adding larger bricks

[Joe Julian]

I've written an example of how gluster's dht works on my blog at 
https://joejulian.name/blog/dht-misses-are-expensive/ which might make 
it clear why the end result is not what you expected.

By setting cluster.min-free-disk (defaults to 10%) you can, at least, 
ensure that your new bricks are utilized as needed to prevent over 
filling your smaller bricks.

On 10/10/2016 10:13 AM, Jackie Tung wrote:
> Hi,
>
> We have a 2 node, distributed replicated setup (11 bricks on each 
> node).  Each of these bricks are 6TB in size.
>
> node_A:/brick1 replicates node_B:/brick1
> node_A:/brick2 replicates node_B:/brick2
> node_A:/brick3 replicates node_B:/brick3
> …
> …
> node_A:/brick11 replicates node_B:/brick11
>
> We recently added 5 more bricks to make it 16 bricks on each node in 
> total.  Each of these new bricks are 8TB in size.
>
> We completed a full rebalance operation (status says “completed”).
>
> However the end result is somewhat unexpected:
> */dev/sdl1 7.3T 2.2T 5.2T 29%*
> */dev/sdk1 7.3T 2.0T 5.3T 28%*
> */dev/sdj1 7.3T 2.0T 5.3T 28%*
> */dev/sdn1 7.3T 2.2T 5.2T 30%*
> */dev/sdp1 7.3T 2.2T 5.2T 30%*
> /dev/sdc1 5.5T 2.3T 3.2T 42%
> /dev/sdf1 5.5T 2.3T 3.2T 43%
> /dev/sdo1 5.5T 2.3T 3.2T 42%
> /dev/sda1 5.5T 2.3T 3.2T 43%
> /dev/sdi1 5.5T 2.3T 3.2T 42%
> /dev/sdh1 5.5T 2.3T 3.2T 43%
> /dev/sde1 5.5T 2.3T 3.2T 42%
> /dev/sdb1 5.5T 2.3T 3.2T 42%
> /dev/sdm1 5.5T 2.3T 3.2T 42%
> /dev/sdg1 5.5T 2.3T 3.2T 42%
> /dev/sdd1 5.5T 2.3T 3.2T 42%
>
> The df output in *bold* are the new 8TB drives.
> Was I wrong to expect the % usage to be roughly equal?  Is there some 
> parameter I need to tweak to make rebalance account for disk sizes 
> properly?
>
> I’m using Gluster 3.8 on Ubuntu.
>
> Thanks,
> Jackie
>
> The information in this email is confidential and may be legally 
> privileged. It is intended solely for the addressee. Access to this 
> email by anyone else is unauthorized. If you are not the intended 
> recipient, any disclosure, copying, distribution or any action taken 
> or omitted to be taken in reliance on it, is prohibited and may be 
> unlawful.
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161010/f40e8cb0/attachment.html>