<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I did explore CEPH a bit, and that might be an option as well, still doing exploration on gluster. Hopefully no one hates you for making the suggestion
<span id="🙂">🙂</span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I haven't tried NFS Ganesha yet. I was under the impression it was maybe a little unstable yet, and found the docs a little limited for it. If that solves the issue that might also be a good option. I've heard others suggest performance is better for it
than the FUSE client as well.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I don't know how other systems deal with it currently, but it seems like even just leveraging the volfile itself as a source for backups would work well. There are still likely issues where things could lapse, but that seems like an improvement at least.
I'll try and dig into what other's are using, though maybe they don't have this issue at all since they tend to use metadata servers?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Tim<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Strahil Nikolov <hunter86_bg@yahoo.com><br>
<b>Sent:</b> Wednesday, October 16, 2019 11:49 AM<br>
<b>To:</b> gluster-users <gluster-users@gluster.org>; Timothy Orme <torme@ancestry.com><br>
<b>Subject:</b> [EXTERNAL] Re: Re: [Gluster-users] Client Handling of Elastic Clusters</font>
<div> </div>
</div>
<div>
<div class="x_ydpca9c9238yahoo-style-wrap" style="font-family:courier new,courier,monaco,monospace,sans-serif; font-size:16px">
<div></div>
<div dir="ltr">Most probably current version never supported (maybe there was no such need until now) such elasticity and the only option is to use Highly-Available NFS Ganesha as the built-in NFS is deprecated.</div>
<div dir="ltr">What about scaling on the same system ? Nowadays , servers have a lot of hot-plug disk slots and you can keep the number of servers the same ... still the server bandwidth will be a limit at some point .</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">I'm not sure how other SDS deal with such elasticity . I guess many users in the list will hate me for saying this , but have you checked CEPH for your needs ?</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Best Regards,</div>
<div dir="ltr">Strahil Nikolov</div>
<div><br>
</div>
</div>
<div id="x_ydp4d15a416yahoo_quoted_1801640462" class="x_ydp4d15a416yahoo_quoted">
<div style="font-family:'Helvetica Neue',Helvetica,Arial,sans-serif; font-size:13px; color:#26282a">
<div>В сряда, 16 октомври 2019 г., 21:13:58 ч. Гринуич+3, Timothy Orme <torme@ancestry.com> написа:
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div id="x_ydp4d15a416yiv3571518655">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Yes, this makes the issue less likely, but doesn't make it impossible for something that is fully elastic.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br clear="none">
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
For instance, if I had instead just started with A,B,C and then scaled out and in twice, all volfile servers would have potentially be destroyed and replaced. I think the problem is that the selection of volfile servers is determined at mounting, rather than
updating as the cluster changes. There are ways to greatly reduce this issue, such as adding more backup servers, but it's still a possibility.
<br clear="none">
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br clear="none">
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I think more important then, for me at least, is to have the option of failing when no volfile servers are remaining as it can produce incomplete views of the data.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br clear="none">
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thanks!</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Tim<br clear="none">
</div>
<div id="x_ydp4d15a416yiv3571518655appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div class="x_ydp4d15a416yiv3571518655yqt1979744634" id="x_ydp4d15a416yiv3571518655yqt81335">
<div dir="ltr" id="x_ydp4d15a416yiv3571518655divRplyFwdMsg"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Strahil <hunter86_bg@yahoo.com><br clear="none">
<b>Sent:</b> Tuesday, October 15, 2019 8:46 PM<br clear="none">
<b>To:</b> Timothy Orme <torme@ancestry.com>; gluster-users <gluster-users@gluster.org><br clear="none">
<b>Subject:</b> [EXTERNAL] Re: [Gluster-users] Client Handling of Elastic Clusters</font>
<div> </div>
</div>
<div>
<p dir="ltr">Hi Timothy,</p>
<p dir="ltr">Have you tried to mount on the client via all servers :</p>
<p dir="ltr">mount -t glusterfs -o backup-volfile-servers=B:C:D:E:F A:/volume /destination</p>
<p dir="ltr">Best Regards,<br clear="none">
Strahil Nikolov</p>
<div class="x_ydp4d15a416yiv3571518655x_quote">On Oct 15, 2019 22:05, Timothy Orme <torme@ancestry.com> wrote:<br clear="none">
<blockquote class="x_ydp4d15a416yiv3571518655x_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
Hello,</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
<br clear="none">
</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
I'm trying to setup an elastic gluster cluster and am running into a few odd edge cases that I'm unsure how to address. I'll try and walk through the setup as best I can.</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
<br clear="none">
</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
If I have a replica 3 distributed-replicated volume, with 2 replicated volumes to start:</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
<br clear="none">
</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
MyVolume</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
Replica 1<br clear="none">
</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
serverA</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
serverB</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
serverC</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
Replica 2</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
serverD</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
serverE</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
serverF<br clear="none">
</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
<br clear="none">
</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
And the client mounts the volume with serverA as the primary volfile server, and B & C as the backups.</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
<br clear="none">
</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
Then, if I perform a scale down event, it selects the first replica volume as the one to remove. So I end up with a configuration like:</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
<br clear="none">
</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
MyVolume<br clear="none">
Replica 2</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
serverD</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
serverE</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
serverF<br clear="none">
<br clear="none">
Everything rebalances and works great. However, at this point, the client has lost any connection with a volfile server. It knows about D, E, and F, so my data is all fine, but it can no longer retrieve a volfile. In the logs I see:<br clear="none">
<br clear="none">
[2019-10-15 17:21:59.232819] I [glusterfsd-mgmt.c:2463:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers<br clear="none">
</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
<br clear="none">
</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
This becomes problematic when I try and scale back up, and add a replicated volume back in:<br clear="none">
<br clear="none">
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt">
MyVolume<br clear="none">
Replica 2</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt">
serverD</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt">
serverE</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt">
serverF</div>
Replica 3</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
serverG</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
serverH</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
serverI</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
<br clear="none">
</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
And then rebalance the volume. Now, I have all my data present, but the client only knows about D,E,F, so when I run an `ls` on a directory, only about half of the files are returned, since the other half live on G,H,I which the client doesn't know about.
The data is still there, but it would require a re-mount at one of the new servers.</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
<br clear="none">
</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
My question then, is there a way to have a more dynamic set of volfile servers? What would be great is if there was a way to tell the mount to fall back on the servers returned in the volfile itself in case the primary one goes away.</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
<br clear="none">
</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
If there's not an easy way to do this, is there a flag on the mount helper that can cause the mount to die or error out in the event that it is unable to retrieve volfiles? The problem now is that it sort of silently fails and returns incomplete file listings,
which for my use cases can cause improper processing of that data. I'd rather have it hard error than provide bad results silently obviously.</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
<br clear="none">
</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
Hope that makes sense, if you need further clarity please let me know. <br clear="none">
</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
<br clear="none">
</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
Thanks,</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
Tim<br clear="none">
</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
<br clear="none">
</div>
<div style="font-family:'calibri' ,'arial' ,'helvetica' ,sans-serif; font-size:12pt; color:rgb(0 ,0 ,0)">
<br clear="none">
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>