<div dir="ltr">Hi Strahil,<div><br></div><div>Actually, I'd be attaching the additional block storage to the same hosts, just under different mounts, that's why in my replies, I referenced servers1 through 4, and there would be no servers 5-8.</div><div><br></div><div>server1:brick1 server2:brick2 server3:brick3 server4:brick4 </div><div>server1:brick5 server2:brick6 server3:brick7 server4:brick8 <br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><br></div><div>So no peer probe needed, and the backup can still go to the same machine, just with the new brick selected. Good to know.</div><div><br></div><div>I'll test all of this, but it looks trivial in retrospect. Hope it doesn't bite us in the butt!</div><div><br></div><div>Thanks, all.</div><div dir="ltr"><br>Sincerely,<br>Artem<br><br>--<br>Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8px" target="_blank">APK Mirror</a><span style="font-size:12.8px">, Illogical Robot LLC</span></div><div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 18, 2020 at 11:56 PM Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com">hunter86_bg@yahoo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On February 19, 2020 1:59:12 AM GMT+02:00, Artem Russakovskii <<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>> wrote:<br>
>Hi Strahil,<br>
><br>
>We have 4 main servers, and I wanted to run gluster on all of them,<br>
>with<br>
>everything working even if 3/4 are down, so I set up a replica 4 with<br>
>quorum at 1. It's been working well for several years now, and I can<br>
>lose 3<br>
>out of 4 servers to outages and remain up.<br>
><br>
>Amar, so to clarify, right now I set up the volume using "gluster v<br>
>create<br>
>$GLUSTER_VOL replica 4 server1:brick1 server2:brick2 server3:brick3<br>
>server4:brick4".<br>
>In order to turn it into a replica 4 but distributed across 4 old<br>
>bricks<br>
>and 4 new bricks (say server1:brick5 server2:brick6 server3:brick7<br>
>server4:brick8), what exact commands do I need to issue?<br>
><br>
>The docs are a bit confusing for this case IMO:<br>
><br>
>> volume add-brick <VOLNAME> [<stripe|replica> <COUNT> [arbiter<br>
><COUNT>]]<br>
>> <NEW-BRICK> ... [force] - add brick to volume <VOLNAME><br>
><br>
><br>
>Do I need to specify a stripe? Do I need to repeat the replica param<br>
>and<br>
>keep it at 4? I.e.:<br>
><br>
>> gluster v add-brick $GLUSTER_VOL replicate 4 server1:brick5<br>
>> gluster v add-brick $GLUSTER_VOL replicate 4 server2:brick6<br>
>> gluster v add-brick $GLUSTER_VOL replicate 4 server3:brick7<br>
>> gluster v add-brick $GLUSTER_VOL replicate 4 server4:brick8<br>
>> gluster v rebalance $GLUSTER_VOL fix-layout start<br>
><br>
><br>
>My reservations about going with this new approach also include the<br>
>fact<br>
>that right now I can back up and restore just the brick data itself as<br>
>each<br>
>brick contains the full copy of the data, and it's a loooot faster to<br>
>access the brick data during backups (probably an order of magnitude<br>
>due to<br>
>unresolved list issues). If I go distributed replicated, my current<br>
>backup<br>
>strategy will need to shift to backing up the gluster volume itself<br>
>(not<br>
>sure what kind of additional load that would put on the servers), or<br>
>maybe<br>
>backing up one brick from each replica would work too, though it's<br>
>unclear<br>
>if I'd be able to restore by just copying the data from such backups<br>
>back<br>
>into one restore location to recreate the full set of data (would that<br>
>work?).<br>
><br>
>Thanks again for your answers.<br>
><br>
>Sincerely,<br>
>Artem<br>
><br>
>--<br>
>Founder, Android Police <<a href="http://www.androidpolice.com" rel="noreferrer" target="_blank">http://www.androidpolice.com</a>>, APK Mirror<br>
><<a href="http://www.apkmirror.com/" rel="noreferrer" target="_blank">http://www.apkmirror.com/</a>>, Illogical Robot LLC<br>
><a href="http://beerpla.net" rel="noreferrer" target="_blank">beerpla.net</a> | @ArtemR<br>
><<a href="http://twitter.com/ArtemR" rel="noreferrer" target="_blank">http://twitter.com/ArtemR</a>><br>
><br>
><br>
>On Mon, Feb 17, 2020 at 3:29 PM Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" target="_blank">hunter86_bg@yahoo.com</a>><br>
>wrote:<br>
><br>
>> On February 18, 2020 1:16:19 AM GMT+02:00, Artem Russakovskii <<br>
>> <a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>> wrote:<br>
>> >Hi all,<br>
>> ><br>
>> >We currently have an 8TB 4-brick replicated volume on our 4 servers,<br>
>> >and<br>
>> >are at 80% capacity. The max disk size on our host is 10TB. I'm<br>
>> >starting to<br>
>> >think about what happens closer to 100% and see 2 options.<br>
>> ><br>
>> >Either we go with another new 4-brick replicated volume and start<br>
>> >dealing<br>
>> >with symlinks in our webapp to make sure it knows which volumes the<br>
>> >data is<br>
>> >on, which is a bit of a pain (but not too much) on the sysops side<br>
>of<br>
>> >things. Right now the whole volume mount is symlinked to a single<br>
>> >location<br>
>> >in the webapps (an uploads/ directory) and life is good. After such<br>
>a<br>
>> >split, I'd have to split uploads into yeardir symlinks, make sure<br>
>> >future<br>
>> >yeardir symlinks are created ahead of time and point to the right<br>
>> >volume,<br>
>> >etc).<br>
>> ><br>
>> >The other direction would be converting the replicated volume to a<br>
>> >distributed replicated one<br>
>> ><br>
>><br>
><a href="https://docs.gluster.org/en/latest/Administrator%20Guide/Setting%20Up%20Volumes/#creating-distributed-replicated-volumes" rel="noreferrer" target="_blank">https://docs.gluster.org/en/latest/Administrator%20Guide/Setting%20Up%20Volumes/#creating-distributed-replicated-volumes</a><br>
>> ,<br>
>> >but I'm a bit scared to do it with production data (even after<br>
>testing,<br>
>> >of<br>
>> >course), and having never dealt with a distributed replicated<br>
>volume.<br>
>> ><br>
>> >1. Is it possible to convert our existing volume on the fly by<br>
>adding 4<br>
>> > bricks but keeping the replica count at 4?<br>
>> >2. What happens if bricks 5-8 which contain the replicated volume #2<br>
>go<br>
>> >down for whatever reason or can't meet their quorum, but the<br>
>replicated<br>
>> > volume #1 is still up? Does the whole main combined volume become<br>
>> >unavailable or only a portion of it which has data residing on<br>
>> >replicated<br>
>> > volume #2?<br>
>> > 3. Any other gotchas?<br>
>> ><br>
>> >Thank you very much in advance.<br>
>> ><br>
>> >Sincerely,<br>
>> >Artem<br>
>> ><br>
>> >--<br>
>> >Founder, Android Police <<a href="http://www.androidpolice.com" rel="noreferrer" target="_blank">http://www.androidpolice.com</a>>, APK Mirror<br>
>> ><<a href="http://www.apkmirror.com/" rel="noreferrer" target="_blank">http://www.apkmirror.com/</a>>, Illogical Robot LLC<br>
>> ><a href="http://beerpla.net" rel="noreferrer" target="_blank">beerpla.net</a> | @ArtemR<br>
>> ><<a href="http://twitter.com/ArtemR" rel="noreferrer" target="_blank">http://twitter.com/ArtemR</a>><br>
>><br>
>> Distributed replicated sounds more reasonable.<br>
>><br>
>> Out of curiocity, why did you decide to have an even number of bricks<br>
>in<br>
>> the replica - it can still suffer from split-brain?<br>
>><br>
>> 1. It should be OK, but I have never done it. Test on some VMs<br>
>before<br>
>> proceeding.<br>
>> Rebalance might take some time, so keep that in mind.<br>
>><br>
>> 2.All files on replica 5-8 will be unavailable untill yoiu recover<br>
>that<br>
>> set of bricks.<br>
>><br>
>> Best Regards,<br>
>> Strahil Nikolov<br>
>><br>
>><br>
<br>
Hi Artem,<br>
<br>
That's interesting...<br>
In order to extend the volume you will need to:<br>
gluster peer probe node5<br>
gluster peer probe node6<br>
gluster peer probe node7<br>
gluster peer probe node8<br>
<br>
gluster volume add-brick replica 4 node{5..8}:/brick/path <br>
<br>
Note: Asuming that the brick paths are the same.<br>
<br>
For the backup, you can still backup via the gluster bricks, but you need to pick 1 node per replica set - as your data will be like this:<br>
Node1..4-> A<br>
Node5..8-> B<br>
<br>
So, you need to backup from 2 hosts instead of one.<br>
<br>
Best Regards,<br>
Strahil Nikolov<br>
</blockquote></div>