[Gluster-users] Phasing out replace-brick for data migration in favor of remove-brick.

[James]

On Fri, 2013-09-27 at 00:35 -0700, Anand Avati wrote:
> Hello all,
Hey,

Interesting timing for this post...
I've actually started working on automatic brick addition/removal. (I'm
planning to add this to puppet-gluster of course.) I was hoping you
could help out with the algorithm. I think it's a bit different if
there's no replace-brick command as you are proposing.

Here's the problem:
Given a logically optimal initial volume:

volA: rep=2; h1:/b1 h2:/b1 h3:/b1 h4:/b1 h1:/b2 h2:/b2 h3:/b2 h4:/b2

suppose I know that I want to add/remove bricks such that my new volume
(if I had created it new) looks like:

volB: rep=2; h1:/b1 h3:/b1 h4:/b1 h5:/b1 h6:/b1 h1:/b2 h3:/b2 h4:/b2
h5:/b2 h6:/b2

What is the optimal algorithm for determining the correct sequence of
transforms that are needed to accomplish this task. Obviously there are
some simpler corner cases, but I'd like to solve the general case.

The transforms are obviously things like running the add-brick {...} and
remove-brick {...} commands.

Obviously we have to take into account that it's better to add bricks
and rebalance before we remove bricks and risk the file system if a
replica is missing. The algorithm should work for any replica N. We want
to make sure the new layout makes sense to replicate the data on
different servers. In many cases, this will require creating a circular
"chain" of bricks as illustrated in the bottom of this image:
http://joejulian.name/media/uploads/images/replica_expansion.png
for example. I'd like to optimize for safety first, and then time, I
imagine.

Many thanks in advance.

James

Some comments below, although I'm a bit tired so I hope I said it all
right.

> DHT's remove-brick + rebalance has been enhanced in the last couple of
> releases to be quite sophisticated. It can handle graceful decommissioning
> of bricks, including open file descriptors and hard links.
Sweet

> 
> This in a way is a feature overlap with replace-brick's data migration
> functionality. Replace-brick's data migration is currently also used for
> planned decommissioning of a brick.
> 
> Reasons to remove replace-brick (or why remove-brick is better):
> 
> - There are two methods of moving data. It is confusing for the users and
> hard for developers to maintain.
> 
> - If server being replaced is a member of a replica set, neither
> remove-brick nor replace-brick data migration is necessary, because
> self-healing itself will recreate the data (replace-brick actually uses
> self-heal internally)
> 
> - In a non-replicated config if a server is getting replaced by a new one,
> add-brick <new> + remove-brick <old> "start" achieves the same goal as
> replace-brick <old> <new> "start".
> 
> - In a non-replicated config, <replace-brick> is NOT glitch free
> (applications witness ENOTCONN if they are accessing data) whereas
> add-brick <new> + remove-brick <old> is completely transparent.
> 
> - Replace brick strictly requires a server with enough free space to hold
> the data of the old brick, whereas remove-brick will evenly spread out the
> data of the bring being removed amongst the remaining servers.

Can you talk more about the replica = N case (where N is 2 or 3?)
With remove brick, add brick you will need add/remove N (replica count)
bricks at a time, right? With replace brick, you could just swap out
one, right? Isn't that a missing feature if you remove replace brick?

> 
> - Replace-brick code is complex and messy (the real reason :p).
> 
> - No clear reason why replace-brick's data migration is better in any way
> to remove-brick's data migration.
> 
> I plan to send out patches to remove all traces of replace-brick data
> migration code by 3.5 branch time.
> 
> NOTE that replace-brick command itself will still exist, and you can
> replace on server with another in case a server dies. It is only the data
> migration functionality being phased out.
> 
> Please do ask any questions / raise concerns at this stage :)
I heard with 3.4 you can somehow change the replica count when adding
new bricks... What's the full story here please?

Thanks!
James

> 
> Avati
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130927/6a9d839e/attachment.sig>