[Gluster-users] Disk failed, how do I remove brick?
Nithya Balachandran
nbalacha at redhat.com
Wed Jun 15 04:49:45 UTC 2016
On Fri, Jun 10, 2016 at 1:25 AM, Phil Dumont <phil at solidstatescientific.com>
wrote:
> Just started trying gluster, to decide if we want to put it into
> production.
>
> Running version 3.7.11-1
>
> Replicated, distributed volume, two servers, 20 bricks per server:
>
> [root at storinator1 ~]# gluster volume status gv0
> Status of volume: gv0
> Gluster process TCP Port RDMA Port Online
> Pid
>
> ------------------------------------------------------------------------------
> Brick storinator1:/export/brick1/gv0 49153 0 Y
> 2554
> Brick storinator2:/export/brick1/gv0 49153 0 Y
> 9686
> Brick storinator1:/export/brick2/gv0 49154 0 Y
> 2562
> Brick storinator2:/export/brick2/gv0 49154 0 Y
> 9708
> Brick storinator1:/export/brick3/gv0 49155 0 Y
> 2568
> Brick storinator2:/export/brick3/gv0 49155 0 Y
> 9692
> Brick storinator1:/export/brick4/gv0 49156 0 Y
> 2574
> Brick storinator2:/export/brick4/gv0 49156 0 Y
> 9765
> Brick storinator1:/export/brick5/gv0 49173 0 Y
> 16901
> Brick storinator2:/export/brick5/gv0 49173 0 Y
> 9727
> Brick storinator1:/export/brick6/gv0 49174 0 Y
> 16920
> Brick storinator2:/export/brick6/gv0 49174 0 Y
> 9733
> Brick storinator1:/export/brick7/gv0 49175 0 Y
> 16939
> Brick storinator2:/export/brick7/gv0 49175 0 Y
> 9739
> Brick storinator1:/export/brick8/gv0 49176 0 Y
> 16958
> Brick storinator2:/export/brick8/gv0 49176 0 Y
> 9703
> Brick storinator1:/export/brick9/gv0 49177 0 Y
> 16977
> Brick storinator2:/export/brick9/gv0 49177 0 Y
> 9713
> Brick storinator1:/export/brick10/gv0 49178 0 Y
> 16996
> Brick storinator2:/export/brick10/gv0 49178 0 Y
> 9718
> Brick storinator1:/export/brick11/gv0 49179 0 Y
> 17015
> Brick storinator2:/export/brick11/gv0 49179 0 Y
> 9746
> Brick storinator1:/export/brick12/gv0 49180 0 Y
> 17034
> Brick storinator2:/export/brick12/gv0 49180 0 Y
> 9792
> Brick storinator1:/export/brick13/gv0 49181 0 Y
> 17053
> Brick storinator2:/export/brick13/gv0 49181 0 Y
> 9755
> Brick storinator1:/export/brick14/gv0 49182 0 Y
> 17072
> Brick storinator2:/export/brick14/gv0 49182 0 Y
> 9767
> Brick storinator1:/export/brick15/gv0 49183 0 Y
> 17091
> Brick storinator2:/export/brick15/gv0 N/A N/A N
> N/A
> Brick storinator1:/export/brick16/gv0 49184 0 Y
> 17110
> Brick storinator2:/export/brick16/gv0 49184 0 Y
> 9791
> Brick storinator1:/export/brick17/gv0 49185 0 Y
> 17129
> Brick storinator2:/export/brick17/gv0 49185 0 Y
> 9756
> Brick storinator1:/export/brick18/gv0 49186 0 Y
> 17148
> Brick storinator2:/export/brick18/gv0 49186 0 Y
> 9766
> Brick storinator1:/export/brick19/gv0 49187 0 Y
> 17167
> Brick storinator2:/export/brick19/gv0 49187 0 Y
> 9745
> Brick storinator1:/export/brick20/gv0 49188 0 Y
> 17186
> Brick storinator2:/export/brick20/gv0 49188 0 Y
> 9783
> NFS Server on localhost 2049 0 Y
> 17206
> Self-heal Daemon on localhost N/A N/A Y
> 17214
> NFS Server on storinator2 2049 0 Y
> 9657
> Self-heal Daemon on storinator2 N/A N/A Y
> 9677
>
> Task Status of Volume gv0
>
> ------------------------------------------------------------------------------
> Task : Rebalance
> ID : 28c733e9-d618-44fc-873f-405d3b29a609
> Status : completed
>
>
> Wouldn't you know it, within a week or two of pulling the hardware
> together and getting gluster installed and configured, a disk dies. Note
> the dead process for brick15 on server storinator2.
>
> I would like to remove (not replace) the failed brick (and its replica).
> (I don't have a spare disk handy, and there's plenty of room on the other
> bricks.) But gluster doesn't seem to want to remove a brick if the brick
> is dead:
>
> [root at storinator1 ~]# gluster volume remove-brick gv0
> storinator{1..2}:/export/brick15/gv0 start
> volume remove-brick start: failed: Staging failed on storinator2. Error:
> Found stopped brick storinator2:/export/brick15/gv0
>
>
> So what do I do? I can't remove the brick while the brick is bad, but I
> want to remove the brick *because* the brick is bad. Bit of a Catch-22.
>
Hi Phil,
This looks like a bug - please open a new BZ for this issue.
You can remove these bricks using
gluster v remove-brick <vol> <bricks> force
However, that will not copy the data from the bricks that have been
removed. You will need to copy the files manually from the brick to the
gluster volume mount point.
Thanks,
Nithya
> Thanks in advance for any help you can give.
>
> phil
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160615/a0a30c63/attachment.html>
More information about the Gluster-users
mailing list