[Gluster-users] Replace brick of a dead node

[Mohit Anchlia]

On Fri, Aug 12, 2011 at 4:49 PM, Rajat Chopra <rchopra at redhat.com> wrote:
>
> I am afraid the 'replace-brick' procedure does not work well if the node is dead. Here is the (long-ish) step-wise procedure for the dead-end that I run into...
>
>  [node-1 $] service glusterd start
>  [node-1 $] gluster volume create my-vol replica 2 node-1:/srv-node-1-first node-1:/srv-node-1-second
>  [node-1 $] gluster volume start my-vol
> # this began my gluster service on first node with two bricks replicated but sourcing from the same node
> # next I add a new node and replace one of the bricks with a new brick location on second node
> # the purpose is to achieve failover redundancy
>  [node-2 $] service glusterd start
>  [node-1 $] gluster peer probe node-2
>  [node-2 $] gluster peer probe node-1
>  [node-2 $] gluster volume replace-brick my-vol node-1:/srv-node-1-second node-2:/srv-node-2-third start
> # this starts the replace operation and after a while I can do volume info from either node
>  [node-2 $] gluster volume info
> Volume Name: my-vol
> Type: Replicate
> Status: Started
> Number of Bricks: 2
> Transport-type: tcp
> Bricks:
> Brick1: node-1:/srv-node-1-first
> Brick2: node-2:/srv-node-2-third
>
> # all good so far... now node-1 dies (no EBS, no disk, no data... just not reachable.. its a pvt cloud and the machine running the vm had a hardware failure)
> # good gluster serves well from node-2 to all the clients nicely too
> # now I want to replace the node-1 brick to another brick in node-2 so that I can pass it on to new nodes later
>
> # so according to the suggestion, I ran replace-brick command
> [node-2 $] gluster volume replace-brick my-vol node-1:/srv-node-1-first node-2:/srv-node-2-fourth start


Did you run $ gluster volume replace-brick my-vol
node-1:/srv-node-1-first node-2:/srv-node-2-fourth commit ?

If not, try running this additional command commit. This will make
necessary changes to the config. So don't check status but run commit
right after because dead node is not around.

> # the command succeeds without errors, so I check status...
> [node-2 $] gluster volume replace-brick my-vol node-1:/srv-node-1-first node-2:/srv-node-2-fourth status
> # this command is supposed to return the status, but it returns nothing
> # I check with gluster volume info on node-2
> [node-2 $] gluster volume info
> No volumes present
> # wha?? where did my volume go?
> # Note that all this while.. my mounted client is working fine, so no downtime
>
> Since 'gluster volume info' returned with 'No volumes present', I assume that the procedure does not work. Is there something wrong in my procedure, or was it not supposed to work anyway?
>
> I am using v3.1.1
>
> Again, I really appreciate the help, but I seem to be stuck. The email suggested the procedure in the following link
>      [ http://gluster.com/community/documentation/index.php/Gluster_3.2:_Brick_Restoration_-_Replace_Crashed_Server ]
> It seems like a better way of replacing dead-nodes, but then it would seem that I cant replace the brick from dead node to a newly created path on an existing node, because I should have the hostname matched. That is fine too, if its a requirement, but can I assume that it will work with 3.1.1 or do I have to upgrade to 3.2 for it?
>
> Thanks again for the assistance.
> Rajat
>
>
>
>
>
> ----- Original Message -----
> From: "Mohit Anchlia" <mohitanchlia at gmail.com>
> To: "Rajat Chopra" <rchopra at redhat.com>
> Cc: "Harshavardhana" <harsha at gluster.com>, gluster-users at gluster.org
> Sent: Friday, August 12, 2011 3:07:59 PM
> Subject: Re: [Gluster-users] Replace brick of a dead node
>
> On Fri, Aug 12, 2011 at 2:35 PM, Rajat Chopra <rchopra at redhat.com> wrote:
>>
>> Thank you Harsha for the quick response.
>>
>> Unfortunately, the infrastructure is in the cloud. So, I cant get the dead node's disk.
>> Since I have replication 'ON', there is no downtime as the brick on the second node serves well, but I want the redundancy/replication to be restored with the introduction of a new node (#3) in the cluster.
>
> One way is http://gluster.com/community/documentation/index.php/Gluster_3.2:_Brick_Restoration_-_Replace_Crashed_Server
>
> Other way is to use replace-brick. You should be able to use it even
> if the node is dead.
>
>>
>> I would hope there is a gluster command to just forget about the dead node's brick, and pick up the new brick and start replicating/serving from the new location (in conjunction with the one existing brick on the #2 node). Is that the self heal feature? I am using v3.11 as of now.
>>
>> Rajat
>>
>>
>>
>>
>> ----- Original Message -----
>> From: "Harshavardhana" <harsha at gluster.com>
>> To: "Rajat Chopra" <rchopra at redhat.com>
>> Cc: gluster-users at gluster.org
>> Sent: Friday, August 12, 2011 2:06:14 PM
>> Subject: Re: [Gluster-users] Replace brick of a dead node
>>
>>> I have a two node cluster, with two bricks replicated, one on each node.
>>> Lets say one of the node dies and is unreachable.
>>
>> If you have the disk from the dead node, then all have to do is plug
>> it in new system and start running following commands.
>>
>> gluster volume replace-brick <volname> <old-brick> <new-brick> start
>> gluster volume replace-brick <volname> <old-brick> <new-brick> commit
>>
>> You don't have to migrate the data, this works as expected.
>>
>> Since you have a replicate you wouldn't see a downtime,  but mind you
>> self-heal will kick in as of 3.2 it will be blocking, wait for 3.3 you
>> have non-blocking self-healing capabilities.
>>
>>> I want to be able to spin a new node and replace the dead node's brick to a location on the new node.
>>
>> This is out of Gluster's hand, if you already have mechanisms to
>> decommission a brick and reattach it on new node then above steps are
>> fairly simple.
>>
>> Go ahead and try it, it should work.
>>
>> -Harsha
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>