[Gluster-users] gluster rebalance taking multiple days

[Jacob Shucart]

Mike,

I know Allen had sent you something within the support system regarding
this.  Based on his email, it sounds to me like it is possible something
underneath Gluster has caused the rebalance to hang.  This isn't any sort
of fatal problem as you can simply restart the rebalance once that has
been resolved and it will resume what it was doing.  The amount of time it
takes to rebalance depends on a few things like the number of files, the
size of the data, the network speed, and also the extent to which things
were unbalanced(if you are adding a lot more servers as opposed to a
couple it might take longer).  Generally speaking, if you double the
number of servers then the rebalance will probably move half of your data,
so it will take as long as that would normally take.

Jacob Shucart | Gluster
Systems Engineer
E-Mail	: Jacob at gluster.com
Direct	: (408)770-1504

-----Original Message-----
From: gluster-users-bounces at gluster.org
[mailto:gluster-users-bounces at gluster.org] On Behalf Of Michael Robbert
Sent: Monday, December 06, 2010 4:51 PM
To: gluster-users at gluster.org
Subject: [Gluster-users] gluster rebalance taking multiple days

How long should a rebalance take? I know that it depends so lets take this
example. 4 servers, 1 brick per server. here is the df -i output from the
servers:

[root at ra5 ~]# pdsh -g rack7 "df -i|grep brick"
iosrv-7-1:                      366288896 2720139 363568757    1%
/mnt/brick1
iosrv-7-4:                      366288896 3240868 363048028    1%
/mnt/brick4
iosrv-7-2:                      366288896 2594165 363694731    1%
/mnt/brick2
iosrv-7-3:                      366288896 3267152 363021744    1%
/mnt/brick3

So, it looks like there are roughly 10 million files. I have a rebalance
running on one of the servers since last Friday and this is what the
status looks like right now:

[root at iosrv-7-2 ~]# gluster volume rebalance gluster-test status
rebalance step 1: layout fix in progress: fixed layout 149531740

As a side note I started this rebalance when I noticed that about half of
my clients are missing a certain set of files. Upon further investigation
I found that a different set of clients are missing different data. This
problem happened after many problems getting an upgrade to 3.1.1 working.
Unfortunately I don't remember which version was running when I was last
able to write to this volume.

Any thoughts?

Mike

_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users