[Gluster-users] Rebalancing with no new bricks.
hjmangalam at gmail.com
Wed Jun 12 15:39:40 UTC 2013
Thru some happy coincidences, a few brutal threats, and some executive
liquidations, the problem was averted. We dropped 40TB in an hour and the
storage load over the bricks equalized as it did so. So we're good for a
while. However, I'll look into your suggestions (THANKS!!) and experiment
with them on some test files.
Re: the SvK's suggestions, I often do on-brick file manipulations to
resolve problems and especially do recursive ops. It's not a user-level
solution, but it does allow sysadmin flexibility. ie, we use a small util
to fork recursive 'du's to each brick and then sum them afterwards. It's
easily 10-1000x faster than doing it via gluster, especially on our ZOT
file dirs. As is the usual case, it was writ for our specific case, but if
anyone wants it as skel for their instance, happy to share.
On Wed, Jun 12, 2013 at 6:04 AM, Jeff Darcy <jdarcy at redhat.com> wrote:
> On 06/11/2013 05:55 PM, Harry Mangalam wrote:
> > I understand that gluster does not allow gluster-rebalancing in the same
> > config (ie, without adding a brick), but would moving some offending
> > files to another fs and then copying them back tend to rebalance the FS
> > by distributing the copied files more evenly?
> The problem is that no files will move unless the layout (the hash-range
> information used to place files) changes, and the layout won't change
> unless the set of bricks does, so a rebalance without adding/removing
> bricks would end up being a no-op. However, there are two other ways
> this kind of imbalance can be addressed.
> Firstly, we can observe that GlusterFS will still find files even if
> they're in the "wrong" place according to the layout. It is possible to
> move files manually from one brick to another, so long as appropriate
> care is taken e.g. to ensure that all extended attributes are preserved.
> The problem with this approach is that such careful placement might be
> undone the next time you do a rebalance for other reasons.
> The second approach avoids that problem, but is a bit trickier in other
> ways. There is a feature that allows a user to define their own layout
> for a directory, so that it will be left alone by rebalance. There's
> even a script in your friendly neighborhood source tree
> (extras/rebalance.py) that will use this to set a layout that's weighted
> according to the size of bricks (or free space) instead of just treating
> all as equal. You can use that as an example of how to construct and
> apply a valid user-defined layout, plus other techniques that are likely
> to be useful. Lastly, we have a feature that setting the fake
> "distribute.migrate-data" xattr on a file will cause use to re-evaluate
> where it should go and migrate it to the correct brick if necessary.
> Since you seem to be dealing with a relatively small number of files
> that need to be moved, it shouldn't be too hard to combine this little
> bag of tricks into a solution that meets your needs. Just let me know
> if you'd like me to assist.
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Gluster-users