[Gluster-users] How to recover after one node breakdown

Mon Mar 21 08:00:31 UTC 2016

Hi，
Thank you for your reply.
Could you help me to answer my questions as below.

 Now C node has replace the B node in replicate volume.
 And then I want to use B node to replace C node.

 13.shutdown C node
 14.may continue create files on mount point
 15.power up B node (the old files in /var/lib/gluster/ are still there)
 16.B disk is mount on B node(the disk are non-volatile)

 at this point my question:
 1.Must I remove all files in /var/lib/gluster/ and create a glusterd.info in dirctory /var/lib/glusterd/ based on peers of A node on B node?
 2.how to sync the two bricks?Can heal full work?
 3."heal full" must be run on the node whose uuid is biggest in volume?
Thanks,
Xin

At 2016-03-21 14:04:25, "Atin Mukherjee" <amukherj at redhat.com> wrote:
>
>
>On 03/19/2016 06:50 AM, songxin wrote:
>> Hi Gaurav Garg,
>> 
>> Thank you for you reply.It is very helpful for me.
>> And I have a little more questions as below.
>> 
>> precondition:
>> 1.I have three node, A node, B node and C node.
>> 2.A node is always runing.
>> 3.B node and C node has same ip,so just only one can be running.
>> 4.there are two external hard disks, A disk and B disk
>> 5.At first the rootfs are emtpy on all nodes
>> 
>> 
>> reprodue steps:
>> 1.A disk is mounted on A node as A brick
>> 2.B disk is mounted on B node as B brick
>> 3.start glusterd on A node
>> 4.start glusterd on B node
>> 5.create a replicate volume use A brich and B brick
>> 6.start volume
>> 7.a directory is mounted on volume
>> 8.create some files on mount point
>> 9.shutdown B node
>> 10.may continue create files on mount point
>> 11.power up C node (rootfs is empty on C node)
>> 12.Bdisk is mount on C node (the disk are non-volatile )
>> 
>> at this point my question:
>> 1.Must I create a glusterd.info in dirctory /var/lib/glusterd/ based on
>> peers of A node on C node, if I want to use the C node replace the B node?
>> I found that the glusterd will create a new glusterd.info after I start
>> glusterd on C node if I can't create a glusterd.info.Is that ok?
>This is where all your problem starts as per the multiple mails I have
>seen from you and now it makes sense why you/Abhishek see multiple UUIDs
>for the same host.
>
>You'd need to copy glusterd.info file from Node B and place it on Node C
>to indicate Node C is just a replacement server and for that You'd need
>to follow the steps which Gaurav had mentioned in the mail thread.
>
>Post that if gluster cluster is operational (peers are in sync) I don't
>see any reason why heal full won't work.
>> 2.how to sync the two bricks?Can heal full work?
>> 
>> Now C node has replace the B node in replicate volume.
>> And then I want to use B node to replace C node.
>> 
>> 13.shutdown C node
>> 14.may continue create files on mount point
>> 15.power up B node (the old files in /var/lib/gluster/ are still there)
>> 12.Bdisk is mount on B node(the disk are non-volatile)
>> 
>> at this point my question:
>> 1.Must I remove all files in /var/lib/gluster/ and create a
>> glusterd.info in dirctory /var/lib/glusterd/ based on peers of A node on
>> B node?
>> 2.how to sync the two bricks?Can heal full work?
>> 3."heal full" must be run on the node whose uuid is biggest in volume?
>> 
>> 
>> 
>> Thanks,
>> Xin
>> 
>> 
>> 
>> 
>> At 2016-03-19 02:25:58, "Gaurav Garg" <ggarg at redhat.com> wrote:
>>>Hi songxin,
>>>
>>>both method are almost same for recovering the replicated volume. i forgot to mentioned one steps:
>>>
>>> #gluster volume heal $vol full
>>>
>>>IMO this solution should also apply for gluster 3.4
>>>
>>>Thanks,
>>>
>>>Regards,
>>>Gaurav
>>>
>>>----- Original Message -----
>>>From: "songxin" <songxin_1980 at 126.com>
>>>To: "Gaurav Garg" <ggarg at redhat.com>
>>>Cc: "gluster-user" <gluster-users at gluster.org>, "Alastair Neil" <ajneil.tech at gmail.com>
>>>Sent: Friday, March 18, 2016 7:26:50 AM
>>>Subject: Re:Re: [Gluster-users] How to recover after one node breakdown
>>>
>>>Hi,
>>>
>>>Thank you for your reply.
>>>And I found a link that is about how to use a server to replace a old one as below.
>>>
>>>
>>>http://www.gluster.org/community/documentation/index.php/Gluster_3.4:_Brick_Restoration_-_Replace_Crashed_Server
>>>
>>>
>>>My question is following.
>>>1.I found that your recovery step is a little different with the link about sync the volume.I want to know if both methods are  right for recovering the replicate volume?
>>>2.Is this link is just fit for glusterfs 3.4?My glusterfs version is 3.7.6.Is the the link fit the glusterfs 3.7.6?
>>>
>>>
>>>Thanks,
>>>Xin
>>>
>>>
>>>
>>>
>>>
>>>
>>>At 2016-03-17 14:51:34, "Gaurav Garg" <ggarg at redhat.com> wrote:
>>>>>> Could I run some glusterfs command on good node to recover the replicate volume， if I don't copy the files ,including  glusterd.info and other files,from good node to new node.
>>>>
>>>>
>>>>running glusterfs command is not enough to recover the replicate volume. for recovery you need to follow following steps.
>>>>
>>>>1) remove /var/lib/glusterd/* data from new node (if in any case it present) then start glusterd on new node
>>>>1) kill glusterd on new node
>>>>2) from 1st node (which is in good condition) execute #gluster peer status command and copy the uuid from the peer status (you will see one failed node entry with hostname and uuid) and replace this UUID in node file /var/lib/glusterd/glusterd.info
>>>>from the 1st node you can also get the uuid of failed node by doing #cat /var/lib/glusterd/peer/* it will show uuid of failed node along with hostname/ip-address of failed node.
>>>>3) copy /var/lib/glusterd/peers/* to new node.
>>>>4) you need to rename one of the /var/lib/glusterd/peers/* file (you can find that file in new node by just matching the uuid of new node (/var/lib/glusterd/glusterd.info) with /var/lib/glusterd/peers/* file name) with the uuid of 1st node (/var/lib/glusterd/glusterd.info) and modify the content of the same file with having uuid of 1st node and hostname of 1st node.
>>>>5) now start glusterd on new node.
>>>>6) your volume will recover.
>>>>>
>>>>above steps are mandatory steps to recover failed node.
>>>>
>>>>Thanks,
>>>>
>>>>Regards,
>>>>Gaurav
>>>>
>>>>----- Original Message -----
>>>>From: "songxin" <songxin_1980 at 126.com>
>>>>To: "Alastair Neil" <ajneil.tech at gmail.com>
>>>>Cc: gluster-users at gluster.org
>>>>Sent: Thursday, March 17, 2016 8:56:58 AM
>>>>Subject: Re: [Gluster-users] How to recover after one node breakdown
>>>>
>>>>Thank you very much for your reply. 
>>>>
>>>>In fact it is that I use a new node ,of which rootfs is new , to replace the failed node. 
>>>>And the new node has same IP address with the failed one. 
>>>>
>>>>The brick is on a external hard disk.Because the hard disk is mounted on the node ，so the data on the brick of failed node will not be loss but may be async with the brick of good node.And the brick of failed node will be mounted on the new node. 
>>>>
>>>>Now my recovery steps is run some glusterfs command on good node as below, after starting the glusterd on new node. 
>>>>1.remove brick of new node from volume(the volume type is changed from replicate to distribute) 
>>>>2.peer detach the new node ip(the new node ip is same as failed node) 
>>>>3.peer probe the new node ip 
>>>>3.add brick of new node to volume(the volume type is change to replicate) 
>>>>
>>>>But many problem,like data async or peer state is error etc, will happen. 
>>>>
>>>>My question is below. 
>>>>
>>>>Could I run some glusterfs command on good node to recover the replicate volume， if I don't copy the files ,including glusterd.info and other files,from good node to new node. 
>>>>
>>>>Thanks 
>>>>Xin 
>>>>
>>>>
>>>>
>>>>
>>>>发自我的 iPhone 
>>>>
>>>>在 2016年3月17日，04:54，Alastair Neil < ajneil.tech at gmail.com > 写道： 
>>>>
>>>>
>>>>
>>>>
>>>>hopefully you have a back up of /var/lib/glusterd/ glusterd.info and /var/lib/glusterd/peers, if so I think you can copy them back to and restart glusterd and the volume info should get populated from the other node. If not you can probably reconstruct these from these files on the other node. 
>>>>
>>>>i.e: 
>>>>On the unaffected node the peers directory should have an entry for the failed node containing the uuid of the failed node. The glusterd.info file should enable you to recreate the peer file on the failed node. 
>>>>
>>>>
>>>>On 16 March 2016 at 09:25, songxin < songxin_1980 at 126.com > wrote: 
>>>>
>>>>
>>>>
>>>>Hi, 
>>>>Now I face a problem. 
>>>>Reproduc step is as below. 
>>>>1.I create a replicate volume using two brick on two board 
>>>>2.start the volume 
>>>>3.one board is breakdown and all 
>>>>files in the rootfs ,including /var/lib/glusterd/*,are lost. 
>>>>4.reboot the board and ip is not change. 
>>>>
>>>>My question: 
>>>>How to recovery the replicate volume? 
>>>>
>>>>Thanks, 
>>>>Xin 
>>>>
>>>>
>>>>
>>>>
>>>>_______________________________________________ 
>>>>Gluster-users mailing list 
>>>>Gluster-users at gluster.org 
>>>>http://www.gluster.org/mailman/listinfo/gluster-users 
>>>>
>>>>
>>>>_______________________________________________
>>>>Gluster-users mailing list
>>>>Gluster-users at gluster.org
>>>>http://www.gluster.org/mailman/listinfo/gluster-users
>> 
>> 
>> 
>>  
>> 
>> 
>> 
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160321/66e4a01a/attachment.html>