[Gluster-users] Advice for running out of space on a replicated 4-brick gluster

[Artem Russakovskii]

Hi Strahil,

We have 4 main servers, and I wanted to run gluster on all of them, with
everything working even if 3/4 are down, so I set up a replica 4 with
quorum at 1. It's been working well for several years now, and I can lose 3
out of 4 servers to outages and remain up.

Amar, so to clarify, right now I set up the volume using "gluster v create
$GLUSTER_VOL replica 4 server1:brick1 server2:brick2 server3:brick3
server4:brick4".
In order to turn it into a replica 4 but distributed across 4 old bricks
and 4 new bricks (say server1:brick5 server2:brick6 server3:brick7
server4:brick8), what exact commands do I need to issue?

The docs are a bit confusing for this case IMO:

> volume add-brick <VOLNAME> [<stripe|replica> <COUNT> [arbiter <COUNT>]]
> <NEW-BRICK> ... [force] - add brick to volume <VOLNAME>


Do I need to specify a stripe? Do I need to repeat the replica param and
keep it at 4? I.e.:

> gluster v add-brick $GLUSTER_VOL replicate 4 server1:brick5
> gluster v add-brick $GLUSTER_VOL replicate 4 server2:brick6
> gluster v add-brick $GLUSTER_VOL replicate 4 server3:brick7
> gluster v add-brick $GLUSTER_VOL replicate 4 server4:brick8
> gluster v rebalance $GLUSTER_VOL fix-layout start


My reservations about going with this new approach also include the fact
that right now I can back up and restore just the brick data itself as each
brick contains the full copy of the data, and it's a loooot faster to
access the brick data during backups (probably an order of magnitude due to
unresolved list issues). If I go distributed replicated, my current backup
strategy will need to shift to backing up the gluster volume itself (not
sure what kind of additional load that would put on the servers), or maybe
backing up one brick from each replica would work too, though it's unclear
if I'd be able to restore by just copying the data from such backups back
into one restore location to recreate the full set of data (would that
work?).

Thanks again for your answers.

Sincerely,
Artem

--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | @ArtemR
<http://twitter.com/ArtemR>


On Mon, Feb 17, 2020 at 3:29 PM Strahil Nikolov <hunter86_bg at yahoo.com>
wrote:

> On February 18, 2020 1:16:19 AM GMT+02:00, Artem Russakovskii <
> archon810 at gmail.com> wrote:
> >Hi all,
> >
> >We currently have an 8TB 4-brick replicated volume on our 4 servers,
> >and
> >are at 80% capacity. The max disk size on our host is 10TB. I'm
> >starting to
> >think about what happens closer to 100% and see 2 options.
> >
> >Either we go with another new 4-brick replicated volume and start
> >dealing
> >with symlinks in our webapp to make sure it knows which volumes the
> >data is
> >on, which is a bit of a pain (but not too much) on the sysops side of
> >things. Right now the whole volume mount is symlinked to a single
> >location
> >in the webapps (an uploads/ directory) and life is good. After such a
> >split, I'd have to split uploads into yeardir symlinks, make sure
> >future
> >yeardir symlinks are created ahead of time and point to the right
> >volume,
> >etc).
> >
> >The other direction would be converting the replicated volume to a
> >distributed replicated one
> >
> https://docs.gluster.org/en/latest/Administrator%20Guide/Setting%20Up%20Volumes/#creating-distributed-replicated-volumes
> ,
> >but I'm a bit scared to do it with production data (even after testing,
> >of
> >course), and having never dealt with a distributed replicated volume.
> >
> >1. Is it possible to convert our existing volume on the fly by adding 4
> >   bricks but keeping the replica count at 4?
> >2. What happens if bricks 5-8 which contain the replicated volume #2 go
> >down for whatever reason or can't meet their quorum, but the replicated
> >   volume #1 is still up? Does the whole main combined volume become
> >unavailable or only a portion of it which has data residing on
> >replicated
> >   volume #2?
> >   3. Any other gotchas?
> >
> >Thank you very much in advance.
> >
> >Sincerely,
> >Artem
> >
> >--
> >Founder, Android Police <http://www.androidpolice.com>, APK Mirror
> ><http://www.apkmirror.com/>, Illogical Robot LLC
> >beerpla.net | @ArtemR
> ><http://twitter.com/ArtemR>
>
> Distributed replicated sounds more reasonable.
>
> Out of curiocity, why did you decide to have an even number of bricks in
> the replica - it can still suffer from split-brain?
>
> 1.  It should be OK, but I have never done it. Test on some VMs before
> proceeding.
> Rebalance might take some time, so keep that in mind.
>
> 2.All files on replica 5-8 will be unavailable untill yoiu recover that
> set of bricks.
>
> Best Regards,
> Strahil Nikolov
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200218/86b02718/attachment.html>