[Gluster-users] Is Gluster the wrong solution for us?

[James]

On Wed, Dec 11, 2013 at 8:15 PM, Scott Smith <ssmith at mainstreamdata.com> wrote:
> In trying to figure out what was going on with our GlusterFS system after
> the disastrous rebalance, we ran across two posts.  The first one was
> http://hekafs.org/index.php/2012/03/glusterfs-algorithms-distribution/.  If
> we understand it correctly, anytime you add new storage servers to your
> cluster, you have to do a rebalance and that rebalance will require a
> minimum of 50% of the data in the cluster to be moved to make the hashing
> algorithms work.  This means that when we have a 320TB cluster and add
> another 80TB, we have to move at least 160TB just to get things back into
> balance.  Our estimate is that that will take months.  It probably won’t
> finish before we need to add another 80TB.

As other have alluded to / mentioned, you might want to add new
bricks, but _not_ run a straight rebalance. You might want to try
running _just_ fix layout, and letting files settle to approximate
equilibrium over time...