[Gluster-users] Replica bricks fungible?

[Zenon Panoussis]

> So you copy from the brick to the FUSE via rsync , 
> but what is the idea behind that move ?

No, not from the brick. I copy from my external non-gluster
datasource to the gluster volume via the fuse mount of its
one and only brick. This is how I added the new data to the
volume.

I can list the process schematically for better clarity:

1. Take a replica 3+ volume without arbiters. It may already
   contain data or not, it doesn't matter at all. Make sure
   that nothing needs healing[i].
2. Unmount all the clients except the one that you are
   going to load data to. Leave the servers running.
3. Run remove-brick for all the bricks except the one that
   you are going to load data to. Use force if needed, there
   is nothing to rebalance anyway.
4. Now you have an active replica volume with only one brick.
   Use its client (fuse, nfs, whatever) to copy data to it.
5. Copy the active brick with the new data to all the other
   nodes. Make sure that you preserve hard links.
6. Run add-brick force for all the bricks that you previously
   removed.
7. Remount the clients that you previously unmounted.

[i] If something needs healing on the bricks that you are
    about to remove, you can just go ahead and it will be
    healed when you're done. But if something needs healing
    on the brick that you will keep active and load data to,
    that something will be permanently unhealable after this
    process.

It should also work if you have arbiter(s), but I haven't
tested that. In that case you copy the newly-loaded brick
to all the non-arbiter nodes and then you let shd heal the
arbiters. If you have a 4+2 or bigger arbiter configuration
you will probably have to stop the volume before step 3,
then start it for step 4, stop it again before step 5 and
finally start it again after step 6. You will also have to
make sure that you add back the bricks in the correct order,
so that your old arbiters are still arbiters after step 6.

A note about rsync: it is generally inefficient and slow
for copying to gluster volumes, especially to distributed
volumes where it can take longer to read and chunk up a file
over the network in order to do a partial delta transfer,
than to just copy the whole file anew. There is a very good
explanation with workaround suggestions here:

https://developers.redhat.com/blog/2018/08/14/improving-rsync-performance-with-glusterfs

However, those downsides are completely negligible when
copying new data locally onto a single-brick gluster volume
and then rsync is very convenient, especially for copying
the bricks themselves.

Z