<div dir="ltr">Been using Gluster since the 3.3.x days, been burned a few times and if it was not for the help of the community (one specific time saved big by Joe Julian), I would had not continued using it.<div><br>My main use was originally as a self hosted engine via NFS and a file server for windows clients with Samba. After a couple of years, decided to get rid of virtualization so now exclusively used as a file server. I never dared turn on sharding because of the horror stories I saw in the old days for people who turned it on, then back off and then lost VMs to corruption.<br><br></div><div>The 3.3.x to 3.7.x days were good. Burned somewhere in the post 3.7.x version. Mostly problems with samba VFS gluster and also gluster mounts for the file server. Since I stopped using VFS gluster in samba due to problems with specific files breaking and no longer working form the Windows side, I had to permanently go to using gluster mounts. So since the > 3.7.x and throughout all the 4.x versions I lived with memory leaks related to the gluster mounts, where I had to reboot servers once a month or even once every 15 days. Memory would start at around 11GB from the get go, and slowly go up till it would crash the server after eating up all 64GB of RAM in 15 to 30 days.</div><div><br></div><div>After finally upgrading to 6.6.x I no longer have the memory leak. I have seen many improvements in speed but really did suffer with very slow performance for a long time. Nowadays, with 6.6.x and the sam gluster mounts, memory usage stays at around 4-5GB total even after 80 days of uptime!</div><div><br></div><div>I can definitively echo the ease of use and setup that others have mentioned, but I really cannot vouch for the stability until now, with 6.6.x. I have not dared going back to using VFS gluster diue to the bad experience I had, so I guess I could try again and probably even get some performance gains, but why try to fix what ain't broken at the moment and make my life harder?</div><div><br></div><div>I have tried other solutions and they are fast (significantly faster than gluster for as a file server), reliable and stable. Unfortunately, they are not free (well, the version with automatic failover isn't free), otherwise, I would gladly give up glusterfs for MooseFS in my file server use case.</div><div><br></div><div>Diego</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 23, 2020 at 4:13 AM Mahdi Adnan <<a href="mailto:maadnan@earthlink.iq">maadnan@earthlink.iq</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello Gionatan,<div><br></div><div> Using Gluster brick in a RAID configuration might be safer and require less work from Gluster admins but, it is a waste of disk space.</div><div>Gluster bricks are replicated "assuming you're creating a distributed-replica volume" so when brick went down, it should be easy to recover it and should not affect the client's IO.</div><div>We are using JBOD in all of our Gluster setups, overall, performance is good, and replacing a brick would work "most" of the time without issues.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jun 21, 2020 at 8:43 PM Gionatan Danti <<a href="mailto:g.danti@assyoma.it" target="_blank">g.danti@assyoma.it</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Il 2020-06-21 14:20 Strahil Nikolov ha scritto:<br>
> With every community project , you are in the position of a Betta<br>
> Tester - no matter Fedora, Gluster or CEPH. So far , I had<br>
> issues with upstream projects only diring and immediately after<br>
> patching - but this is properly mitigated with a reasonable<br>
> patching strategy (patch test environment and several months later<br>
> patch prod with the same repos).<br>
> Enterprise Linux breaks (and alot) having 10-times more users and<br>
> use cases, so you cannot expect to start to use Gluster and assume<br>
> that a free peoject won't break at all.<br>
> Our part in this project is to help the devs to create a test case for<br>
> our workload , so regressions will be reduced to minimum.<br>
<br>
Well, this is true, and both devs & community deserve a big thanks for <br>
all the work done.<br>
<br>
> In the past 2 years, we got 2 major issues with VMware VSAN and 1<br>
> major issue with a Enterprise Storage cluster (both solutions are<br>
> quite expensive) - so I always recommend proper testing of your<br>
> software .<br>
<br>
Interesting, I am almost tempted to ask you what issue you had with <br>
vSAN, but this is not the right mailing list ;)<br>
<br>
> From my observations, almost nobody is complaining about Ganesha in<br>
> the mailing list -> 50% are having issues with geo replication,20%<br>
> are having issues with small file performance and the rest have<br>
> issues with very old version of gluster -> v5 or older.<br>
<br>
Mmm, I would swear to have read quite a few posts where the problem was <br>
solved by migrating away from NFS Ganesha. Still, for hyperconverged <br>
setup a problem remains: NFS on loopback/localhost is not 100% supported <br>
(or, at least, RH is not willing to declare it supportable/production <br>
ready [1]). A fuse mount would be the more natural way to access the <br>
underlying data.<br>
<br>
> I can't say that a replace-brick on a 'replica 3' volume is more<br>
> riskier than a rebuild of a raid, but I have noticed that nobody is<br>
> following Red Hat's guide to use either:<br>
> - a Raid6 of 12 Disks (2-3 TB big)<br>
> - a Raid10 of 12 Disks (2-3 TB big)<br>
> - JBOD disks in 'replica 3' mode (i'm not sure about the size RH<br>
> recommends, most probably 2-3 TB)<br>
> So far, I didn' have the opportunity to run on JBODs.<br>
<br>
For the RAID6/10 setup, I found no issues: simply replace the broken <br>
disk without involing Gluster at all. However, this also means facing <br>
the "iops wall" I described earlier for single-brick node. Going <br>
full-Guster with JBODs would be interesting from a performance <br>
standpoint, but this complicate eventual recovery from bad disks.<br>
<br>
Does someone use Gluster in JBOD mode? If so, can you share your <br>
experience?<br>
Thanks.<br>
<br>
[1] <a href="https://access.redhat.com/solutions/22231" rel="noreferrer" target="_blank">https://access.redhat.com/solutions/22231</a> (accound required)<br>
[2] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=489889" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=489889</a> (old, but I can <br>
not find anything newer)<br>
<br>
-- <br>
Danti Gionatan<br>
Supporto Tecnico<br>
Assyoma S.r.l. - <a href="http://www.assyoma.it" rel="noreferrer" target="_blank">www.assyoma.it</a> [1]<br>
email: <a href="mailto:g.danti@assyoma.it" target="_blank">g.danti@assyoma.it</a> - <a href="mailto:info@assyoma.it" target="_blank">info@assyoma.it</a><br>
GPG public key ID: FF5F32A8<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><br><div dir="ltr" style="margin-left:0pt" align="left"><table style="border:medium none;border-collapse:collapse"><colgroup><col width="90"><col width="305"><col width="15"></colgroup><tbody><tr style="height:99pt"><td rowspan="2" style="border-right:0.75pt solid rgb(0,121,172);vertical-align:top;background-color:rgb(255,255,255);padding:0pt 7.5pt 0pt 0pt;overflow:hidden"><br><span style="border:medium none;display:inline-block;overflow:hidden;width:70px;height:66px"><img alt="photograph" src="https://lh4.googleusercontent.com/6Cjtg9hXPLLZfWnzWeaKFmcpvxc07qfi8gvwO2b5GP3vqCUkVsYmMxy1aTTh495vksEh2aTJrX-A8zIlmJ5rtjRb5Ufm9-YrOSXyJdITQIV9MMApOHvsicSAigLhfxA3LClLW6mulTO4eo8MoQ" style="margin-left: 0px; margin-top: 0px;" width="70" height="66"></span></td><td style="border-left:0.75pt solid rgb(0,121,172);vertical-align:middle;background-color:rgb(255,255,255);overflow:hidden"><br><div dir="ltr" style="margin-left:0pt" align="left"><table style="border:medium none;border-collapse:collapse"><colgroup><col width="305"></colgroup><tbody><tr style="height:45pt"><td style="vertical-align:top;padding:0pt 0pt 3.75pt 7.5pt;overflow:hidden"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14pt;font-family:Arial;color:rgb(0,121,172);background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Mahdi Adnan</span><span style="font-size:14pt;font-family:Arial;color:rgb(0,121,172);background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><br></span><span style="font-size:10pt;font-family:Arial;color:rgb(68,68,68);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">IT Manager <br></span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:Arial;color:rgb(68,68,68);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Information Technology</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:Arial;color:rgb(68,68,68);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">EarthLink</span></p></td></tr><tr style="height:41pt"><td style="vertical-align:top;padding:3.75pt 0pt 3.75pt 7.5pt;overflow:hidden"><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:Arial;color:rgb(0,121,172);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">VoIP: 69</span><span style="font-size:10pt;font-family:Arial;color:rgb(0,121,172);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><br></span><span style="font-size:10pt;font-family:Arial;color:rgb(0,121,172);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Cell: 07903316180</span></p><p dir="ltr" style="line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:Arial;color:rgb(0,121,172);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Website: </span><a style="text-decoration:none"><span style="font-size:10pt;font-family:Arial;color:rgb(5,99,193);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">www.earthlink.iq</span></a></p></td></tr></tbody></table></div><br></td></tr><tr style="height:4pt"><td colspan="2" style="border-left:0.75pt solid rgb(0,121,172);vertical-align:middle;background-color:rgb(255,255,255);overflow:hidden"><br></td></tr></tbody></table></div><br></div></div>
<br>--
<br>This message has been scanned for viruses and
<br>dangerous content and is believed to be clean.
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div>