[Gluster-users] How can i replace died hard disk (a brick) with new hard disk (new brick).
Pranith Kumar Karampuri
pkarampu at redhat.com
Wed Aug 20 13:12:30 UTC 2014
On 08/15/2014 08:52 AM, Phong Tran wrote:
> Hi, I have a problem, try to search how to fix. I have one brick
> failure in replicate volume. How can i replace with new hard disk?
I guess you need to replace that brick:
Steps:
1) Check that all the bricks are running. It is okay if the brick that
is going to be replaced is down.
2) Bring the brick that is going to be replaced down if not already.
- Get the pid of the brick by executing 'gluster volume <volname>
status'
12:37:49 ? gluster volume status
Status of volume: r2
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick pranithk-laptop:/home/gfs/r2_0 49152 Y 5342 <<---
this is the brick we want to replace lets say.
Brick pranithk-laptop:/home/gfs/r2_1 49153 Y 5354
Brick pranithk-laptop:/home/gfs/r2_2 49154 Y 5365
Brick pranithk-laptop:/home/gfs/r2_3 49155 Y 5376
....
- Login to the machine where the brick is running and kill the brick.
root at pranithk-laptop - /mnt/r2
12:38:33 ? kill -9 5342
- Confirm that the brick is not running anymore and the other
bricks are running fine.
12:38:38 ? gluster volume status
Status of volume: r2
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick pranithk-laptop:/home/gfs/r2_0 N/A N 5342 <<----
brick is not running, others are running fine.
Brick pranithk-laptop:/home/gfs/r2_1 49153 Y 5354
Brick pranithk-laptop:/home/gfs/r2_2 49154 Y 5365
Brick pranithk-laptop:/home/gfs/r2_3 49155 Y 5376
....
3) Set up metadata so that heal will happen from the other brick in
replica pair to the one that is going to be replaced (In this case it is
from /home/gfs/r2_1 -> /home/gfs/r2_5):
- Create a directory on the mount point that doesn't already
exist. Then delete that directory, do the same for metadata changelog by
doing setfattr. This operation marks the pending changelog which will
tell self-heal damon/mounts to perform self-heal from /home/gfs/r2_1 to
/home/gfs/r2_5.
mkdir /mnt/r2/<name-of-nonexistent-dir>
rmdir /mnt/r2/<name-of-nonexistent-dir>
setfattr -n trusted.non-existent-key -v abc /mnt/r2
setfattr -x trusted.non-existent-key /mnt/r2
NOTE: '/mnt/r2' is the mount path.
- Check that there are pending xattrs:
getfattr -d -m. -e hex /home/gfs/r2_1
# file: home/gfs/r2_1
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.r2-client-0=0x000000000000000300000002 <<---- xattrs are
marked from source brick pranithk-laptop:/home/gfs/r2_1
trusted.afr.r2-client-1=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
trusted.glusterfs.volume-id=0xde822e25ebd049ea83bfaa3c4be2b440
3) Replace the brick with 'commit force' option
- Execute replace-brick command
root at pranithk-laptop - /mnt/r2
12:58:46 ? gluster volume replace-brick r2 `hostname`:/home/gfs/r2_0
`hostname`:/home/gfs/r2_5 commit force
volume replace-brick: success: replace-brick commit successful
- Check that the new brick is now online
root at pranithk-laptop - /mnt/r2
12:59:21 ? gluster volume status
Status of volume: r2
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick pranithk-laptop:/home/gfs/r2_5 49156 Y 5731
<<<---- new brick is online
Brick pranithk-laptop:/home/gfs/r2_1 49153 Y 5354
Brick pranithk-laptop:/home/gfs/r2_2 49154 Y 5365
Brick pranithk-laptop:/home/gfs/r2_3 49155 Y 5376
...
- Once self-heal completes the changelogs will be removed.
12:59:27 ? getfattr -d -m. -e hex /home/gfs/r2_1
getfattr: Removing leading '/' from absolute path names
# file: home/gfs/r2_1
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.r2-client-0=0x000000000000000000000000 <<---- Pending
changelogs are cleared.
trusted.afr.r2-client-1=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
trusted.glusterfs.volume-id=0xde822e25ebd049ea83bfaa3c4be2b440
Pranith
>
>
>
> [root at brick05 ~]# gluster volume status
> Status of volume: hdviet
> Gluster process Port Online Pid
> ------------------------------------------------------------------------------
> Brick brick01:/export/hdd00/hdd00 49152 Y 2515
> Brick brick02:/export/hdd00/hdd00 49152 Y 2358
> Brick brick01:/export/hdd01/hdd01 49153 Y 2526
> Brick brick02:/export/hdd01/hdd01 49153 Y 2369
> Brick brick01:/export/hdd02/hdd02 49154 Y 2537
> Brick brick02:/export/hdd02/hdd02 49154 Y 2380
> Brick brick01:/export/hdd03/hdd03 49155 Y 2548
> Brick brick02:/export/hdd03/hdd03 49155 Y 2391
> Brick brick01:/export/hdd04/hdd04 49156 Y 2559
> Brick brick02:/export/hdd04/hdd04 49156 Y 2402
> Brick brick01:/export/hdd05/hdd05 49157 Y 2570
> Brick brick02:/export/hdd05/hdd05 49157 Y 2413
> Brick brick01:/export/hdd06/hdd06 49158 Y 2581
> Brick brick02:/export/hdd06/hdd06 49158 Y 2424
> Brick brick01:/export/hdd07/hdd07 49159 Y 2592
> Brick brick02:/export/hdd07/hdd07 49159 Y 2435
> Brick brick03:/export/hdd00/hdd00 49152 Y 2208
> Brick brick04:/export/hdd00/hdd00 49152 Y 2352
> Brick brick03:/export/hdd01/hdd01 49153 Y 2219
> Brick brick04:/export/hdd01/hdd01 49153 Y 2363
> Brick brick03:/export/hdd02/hdd02 49154 Y 2230
> Brick brick04:/export/hdd02/hdd02 49154 Y 2374
> Brick brick03:/export/hdd03/hdd03 49155 Y 2241
> Brick brick04:/export/hdd03/hdd03 49155 Y 2385
> Brick brick03:/export/hdd04/hdd04 49156 Y 2252
> Brick brick04:/export/hdd04/hdd04 49156 Y 2396
> Brick brick03:/export/hdd05/hdd05 49157 Y 2263
> Brick brick04:/export/hdd05/hdd05 49157 Y 2407
> Brick brick03:/export/hdd06/hdd06 49158 Y 2274
> Brick brick04:/export/hdd06/hdd06 49158 Y 2418
> Brick brick03:/export/hdd07/hdd07 49159 Y 2285
> Brick brick04:/export/hdd07/hdd07 49159 Y 2429
> Brick brick05:/export/hdd00/hdd00 49152 Y 2321
> Brick brick06:/export/hdd00/hdd00 49152 Y 2232
> Brick brick05:/export/hdd01/hdd01 49153 Y 2332
> Brick brick06:/export/hdd01/hdd01 49153 Y 2243
> *Brick brick05:/export/hdd02/hdd02 N/A N N/A*
> Brick brick06:/export/hdd02/hdd02 49154 Y
> 13976
> Brick brick05:/export/hdd03/hdd03 49155 Y 2354
> Brick brick06:/export/hdd03/hdd03 49155 Y 2265
> Brick brick05:/export/hdd04/hdd04 49156 Y 2365
> Brick brick06:/export/hdd04/hdd04 49156 Y 2276
> Brick brick05:/export/hdd05/hdd05 49157 Y 2376
> Brick brick06:/export/hdd05/hdd05 49157 Y 2287
> Brick brick05:/export/hdd06/hdd06 49158 Y 2387
> Brick brick06:/export/hdd06/hdd06 49158 Y 2298
> Brick brick05:/export/hdd07/hdd07 49159 Y 2398
> Brick brick06:/export/hdd07/hdd07 49159 Y 2309
> Brick brick07:/export/hdd00/hdd00 49152 Y 2357
> Brick brick08:/export/hdd00/hdd00 49152 Y 2261
> Brick brick07:/export/hdd01/hdd01 49153 Y 2368
> Brick brick08:/export/hdd01/hdd01 49153 Y 2272
> Brick brick07:/export/hdd02/hdd02 49154 Y 2379
> Brick brick08:/export/hdd02/hdd02 49154 Y 2283
> Brick brick07:/export/hdd03/hdd03 49155 Y 2390
> Brick brick08:/export/hdd03/hdd03 49155 Y 2294
> Brick brick07:/export/hdd04/hdd04 49156 Y 2401
> Brick brick08:/export/hdd04/hdd04 49156 Y 2305
> Brick brick07:/export/hdd05/hdd05 49157 Y 2412
> Brick brick08:/export/hdd05/hdd05 49157 Y 2316
> Brick brick07:/export/hdd06/hdd06 49158 Y 2423
> Brick brick08:/export/hdd06/hdd06 49158 Y 2327
> Brick brick07:/export/hdd07/hdd07 49159 Y 2434
> Brick brick08:/export/hdd07/hdd07 49159 Y 2338
> NFS Server on localhost 2049 Y
> 15604
> Self-heal Daemon on localhost N/A Y
> 15614
> NFS Server on brick04 2049 Y 2443
> Self-heal Daemon on brick04 N/A Y 2447
> NFS Server on brick03 2049 Y 2300
> Self-heal Daemon on brick03 N/A Y 2304
> NFS Server on brick02 2049 Y 2449
> Self-heal Daemon on brick02 N/A Y 2453
> NFS Server on 192.168.200.1 2049 Y 2606
> Self-heal Daemon on 192.168.200.1 N/A Y 2610
> NFS Server on brick06 2049 Y
> 14021
> Self-heal Daemon on brick06 N/A Y 14028
> NFS Server on brick08 2049 Y 2352
> Self-heal Daemon on brick08 N/A Y 2356
> NFS Server on brick07 2049 Y 2448
> Self-heal Daemon on brick07 N/A Y 2452
>
> Task Status of Volume hdviet
> ------------------------------------------------------------------------------
> There are no active volume tasks
>
>
>
>
> [root at brick05 ~]# gluster volume info
>
> Volume Name: hdviet
> Type: Distributed-Replicate
> Volume ID: fe3a2ed8-d727-499b-9cc6-b11ffb80fc5d
> Status: Started
> Number of Bricks: 32 x 2 = 64
> Transport-type: tcp
> Bricks:
> Brick1: brick01:/export/hdd00/hdd00
> Brick2: brick02:/export/hdd00/hdd00
> Brick3: brick01:/export/hdd01/hdd01
> Brick4: brick02:/export/hdd01/hdd01
> Brick5: brick01:/export/hdd02/hdd02
> Brick6: brick02:/export/hdd02/hdd02
> Brick7: brick01:/export/hdd03/hdd03
> Brick8: brick02:/export/hdd03/hdd03
> Brick9: brick01:/export/hdd04/hdd04
> Brick10: brick02:/export/hdd04/hdd04
> Brick11: brick01:/export/hdd05/hdd05
> Brick12: brick02:/export/hdd05/hdd05
> Brick13: brick01:/export/hdd06/hdd06
> Brick14: brick02:/export/hdd06/hdd06
> Brick15: brick01:/export/hdd07/hdd07
> Brick16: brick02:/export/hdd07/hdd07
> Brick17: brick03:/export/hdd00/hdd00
> Brick18: brick04:/export/hdd00/hdd00
> Brick19: brick03:/export/hdd01/hdd01
> Brick20: brick04:/export/hdd01/hdd01
> Brick21: brick03:/export/hdd02/hdd02
> Brick22: brick04:/export/hdd02/hdd02
> Brick23: brick03:/export/hdd03/hdd03
> Brick24: brick04:/export/hdd03/hdd03
> Brick25: brick03:/export/hdd04/hdd04
> Brick26: brick04:/export/hdd04/hdd04
> Brick27: brick03:/export/hdd05/hdd05
> Brick28: brick04:/export/hdd05/hdd05
> Brick29: brick03:/export/hdd06/hdd06
> Brick30: brick04:/export/hdd06/hdd06
> Brick31: brick03:/export/hdd07/hdd07
> Brick32: brick04:/export/hdd07/hdd07
> Brick33: brick05:/export/hdd00/hdd00
> Brick34: brick06:/export/hdd00/hdd00
> Brick35: brick05:/export/hdd01/hdd01
> Brick36: brick06:/export/hdd01/hdd01
> *Brick37: brick05:/export/hdd02/hdd02*
> Brick38: brick06:/export/hdd02/hdd02
> Brick39: brick05:/export/hdd03/hdd03
> Brick40: brick06:/export/hdd03/hdd03
> Brick41: brick05:/export/hdd04/hdd04
> Brick42: brick06:/export/hdd04/hdd04
> Brick43: brick05:/export/hdd05/hdd05
> Brick44: brick06:/export/hdd05/hdd05
> Brick45: brick05:/export/hdd06/hdd06
> Brick46: brick06:/export/hdd06/hdd06
> Brick47: brick05:/export/hdd07/hdd07
> Brick48: brick06:/export/hdd07/hdd07
> Brick49: brick07:/export/hdd00/hdd00
> Brick50: brick08:/export/hdd00/hdd00
> Brick51: brick07:/export/hdd01/hdd01
> Brick52: brick08:/export/hdd01/hdd01
> Brick53: brick07:/export/hdd02/hdd02
> Brick54: brick08:/export/hdd02/hdd02
> Brick55: brick07:/export/hdd03/hdd03
> Brick56: brick08:/export/hdd03/hdd03
> Brick57: brick07:/export/hdd04/hdd04
> Brick58: brick08:/export/hdd04/hdd04
> Brick59: brick07:/export/hdd05/hdd05
> Brick60: brick08:/export/hdd05/hdd05
> Brick61: brick07:/export/hdd06/hdd06
> Brick62: brick08:/export/hdd06/hdd06
> Brick63: brick07:/export/hdd07/hdd07
> Brick64: brick08:/export/hdd07/hdd07
>
>
> [root at brick05 ~]# gluster peer status
> Number of Peers: 7
>
> Hostname: brick08
> Uuid: ae52d2c7-6966-4261-9d51-b789010c78c7
> State: Peer in Cluster (Connected)
>
> Hostname: brick06
> Uuid: 88910c4e-3b3c-4797-adfd-9236f161051a
> State: Peer in Cluster (Connected)
>
> Hostname: brick03
> Uuid: b30eb4f4-19a8-4309-9c14-02893a52f0b8
> State: Peer in Cluster (Connected)
>
> Hostname: brick04
> Uuid: b0cd18a8-b5b1-4bf2-b6d1-2803be86e955
> State: Peer in Cluster (Connected)
>
> Hostname: 192.168.200.1
> Uuid: 574acf46-22b0-45f4-a4d0-768417202bf5
> State: Peer in Cluster (Connected)
>
> Hostname: brick02
> Uuid: 068389b6-8f4c-4eaf-be91-f7aac490078b
> State: Peer in Cluster (Connected)
>
> Hostname: brick07
> Uuid: 7ff99e83-31fb-4eac-9b82-5a0e54feb761
> State: Peer in Cluster (Connected)
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140820/9548f905/attachment.html>
More information about the Gluster-users
mailing list