[Gluster-users] Migrating bricks from a 2 drive raid 0 to a single HD
marascio at gmail.com
Fri Jul 17 18:21:47 UTC 2015
I currently have a 30 TB distributed-replicate gluster volume with 10 bricks
and a replica count of 2 (so 5x2 configuration). The cluster is made up of 5
nodes, and each node has 4 3.5" HD slots. Each node has 2 bricks and each
brick is comprised of 2 3 TB hard drives in a raid 0 configuration. As you
can see, each brick is 6 TB.
The volume is getting quite full and I need to stretch the hardware as long as
possible, so my plan is to migrate away from bricks build around 2 3 TB drives
in raid 0 to a single 6 TB drive. By doing this I will free up 2 HD slots per
node and can then add new bricks doubling the capacity of the volume to 60 TB.
To the crux of the matter: I'm looking for any advice or thoughts on doing
this migration. After some discussion on IRC shyam and I came to the
conclussion that the following procedure was a good option (downtime is
1. Run a heal operation on the entire volume.
2. Shutdown all cluster nodes.
3. Pull sda and sdb (slot1 and slot2) from the node to upgrade. In my
config sd[ab] = md1 = brick1. Due to how my replication is setup I will
be doing the hard drive swap starting with brick2.
4. Insert a new 6 TB disk into slot1 (where sda was).
5. Boot the node from a livecd or equivalent. We should have a 6 TB drive
as sda and brick2 should be md2 (comprising sdc and sdd).
6. Clone /dev/md2 onto /dev/sda. I will do this with either dd or cat.
Suffice it to say I will create a bit-perfect copy of /dev/md2 onto
/dev/sda thus cloning all of brick2's data.
7. Zero out the mdadm super block on /dev/sda (I believe the superblock is
going to be picked up as part of the clone and we don't need it
8. My fstab uses labels for mount points so I will ensure the label of the
new 6 TB sda drive is the same as the old md1 (just cloned, so it SHOULD
9. Shutdown the node.
10. Pull sdc and sdb (slot3 and slot4). Move newly cloned 6 TB drive from
slot1 to slot3 (where sdc was). Replace original sda and sdb into their
original slots (remember, these are for brick1 and not being modified at
all at this point). I should now have brick1 back, brick2 should exist
on a single 6 TB hard drive, and there should be a single free slot for
our new brick, brick3, to be added later after ensuring we have migrated
11. Boot the nodes. Gluster should see no change as the 6 TB drive is
mounted at the same spot as the previous md1 brick was.
At this point adding the new 6 TB brick into the newly freed slot is
straightforward enough. Since I have a replica 2 setup I will obviously do the
above steps on two nodes at a time so that the new 6 TB brick has somewhere to
Can anyone see any problems with this approach? Is there a better way?
Thanks (especially to shyam for helping out on IRC)!
note: the bricks are actually 5.4TB each and the volume is actually 27 TB as
a small bit of each drive is carved out for the boot partition. I omit these
numbers above to keep things simple and the numbers round.
Louis R. Marascio
More information about the Gluster-users