[Gluster-users] Expand distributed replicated volume

[Anuradha Talur]

----- Original Message -----
> From: "Gandalf Corvotempesta" <gandalf.corvotempesta at gmail.com>
> To: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> Cc: "Anuradha Talur" <atalur at redhat.com>, "gluster-users" <Gluster-users at gluster.org>
> Sent: Tuesday, July 12, 2016 4:32:57 PM
> Subject: Re: [Gluster-users] Expand distributed replicated volume
> 
> 2016-07-12 12:56 GMT+02:00 Pranith Kumar Karampuri <pkarampu at redhat.com>:
> > For adding new bricks into a replica set you need each brick in the replica
> > set to be from different machine. So you can't add all bricks directly from
> > just one machine. So how do you get the extra bricks that can be combined
> > with the bricks on new machine? From old servers. How do you get those?
> > Using replace-brick. I hope with this info if you can go through the
> > example
> > once again, it probably may clarify your doubts. Please feel free to ask
> > any
> > questions you may have.
> 
> I'm new to gluster, so let me try to explain and sorry for my dump question.
> 
> If I have a replica made with:
> 
> (s1b1, s2b1, s3b1)
> (s1b2, s2b2, s3b2)
> (s1b3, s2b3, s3b3)
> 
> Why I cant add a new server getting this result:
> 
> (s1b1, s2b1, s3b1, s4b1)
> (s1b2, s2b2, s3b2, s4b2)
> (s1b3, s2b3, s3b3, s4b3)
> 
> ?? In this case , I've added a new server with 3 bricks.
> 

Some clarification here :

In Pranith's example, each replica set is represented as comma separated values.
Meaning, (s1b1, s2b1, s3b1) will all contain the same data replicated thrice.

If you add new server with 3 bricks like you've represented :
(s1b1, s2b1, s3b1, s4b1) as per Pranith's representation it means 
there are 4 bricks which are replica of each other. Meaning they all have the same data,
so there is 4 fold replication here.

>From my understanding, you say that your job should only be to add a node with 3
bricks and gluster should internally take care of distributing the data. This is a
perfectly valid requirement, but right now, gluster makes its replica set based on the
order of bricks specified in the cli. When you perform add-brick, and mention 3 new
bricks, they will become replica of each other automatically. Which is why steps mentioned
by Pranith will help in solving your issue.

I'm repeating the same example that Pranith gave. Skip the next section if you have already
understood.
What Pranith is trying to say is, at the end you would have technically added 3 bricks
to your cluster (all residing in the same node) but, the distribution will not happen automatically.
It will be your responsibility (until heketi is usable to do the same) to make sure that the
newly added bricks are arranged such that you don't lose redundancy in case of node failures.

Taking your example here:

Step 1) You have :
        (s1b1, s2b1, s3b1)
        (s1b2, s2b2, s3b2)
        (s1b3, s2b3, s3b3)

Step 2) You want to add 3 more disks/bricks to increase volume size.
Condition is that you want to add only one new node. As you don't want to lose
redundancy in case of node failure, you have to ensure that no 2 bricks which
contain the same data reside in the same node. As this arrangement is not taken care by
gluster yet, using the same resources, there is only one way to do it:

Empty one brick from each existing node so that we have gained distribution of our
bricks to all nodes. And now substitute each of these bricks with the 3 new bricks
from the latest node. This is done using replace-brick option.

At this point, your cluster will look like :
        (s4b1, s2b1, s3b1)
        (s1b2, s4b2, s3b2)
        (s1b3, s2b3, s4b3)

So instead of s4b1, s4b2 and s4b3 being free, you have s1b1, s2b2 and s3b3 free.

Step 3)
You can now add the three free bricks to form a new 3 way replica set. Do this,
using add-brick command.

This way, you have essentially added only one node containing 3 bricks, and also
achieved redundancy.

I realize I just repeated the same example in different words but I hope this answers
your questions. So do let us know if something is not clear.

-- 
Thanks,
Anuradha.