<div dir="ltr"><div>Hi,</div><div><br></div><div>Stefan, sorry to hear that things are breaking a lot in your cluster, please file a bug(s) with the necessary information so we can take a look.</div><div>If already filed, share it here so we are reminded of it. Fixing broken cluster state should be easy with Gluster. <br></div><div>There are a few older threads you should be able to find regarding the same. <br></div><div><br></div><div>Do consider the facts that the devs are limited in bandwidth. we do look at the issues and are fixing them actively.</div><div>We may take some time expecting the community to help each other as well. If they couldn't resolve it we get in try to sort it out.<br></div><div>FYI: You can see dozens of bugs being worked on even in the past 2 days: <a href="https://review.gluster.org/#/q/status:open+project:glusterfs">https://review.gluster.org/#/q/status:open+project:glusterfs</a></div><div>And there are other activities happening around as well to make gluster project healthier. Like Glusto. We are working on this testing framework <br></div><div>to cover as many cases as possible. If you can send out a test case, it will be beneficial for you as well as the community.<br></div><div><br></div><div>We don't see many people sending out mails that their cluster is healthy and they are happy (not sure if they think they are spamming. <br></div><div>which they won't be. It helps us understand how well things are going).</div><div>Thanks Erik and Strahi, for sharing your experience. It means a lot to us :)<br></div><div>People usually prefer to send a mail when something breaks and that's one main reason all the threads you read are creating negativity.</div><div><br></div><div>Do let us know what is the issue and we will try our best to help you out.</div><div><br></div><div>Regards,</div><div>Hari.<br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 12, 2020 at 11:58 AM Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com">hunter86_bg@yahoo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On February 12, 2020 12:28:14 AM GMT+02:00, Erik Jacobson <<a href="mailto:erik.jacobson@hpe.com" target="_blank">erik.jacobson@hpe.com</a>> wrote:<br>
>> looking through the last couple of week on this mailing list and<br>
>reflecting our own experiences, I have to ask: what is the status of<br>
>GlusterFS? So many people here reporting bugs and no solutions are in<br>
>sight. GlusterFS clusters break left and right, reboots of a node have<br>
>become a warrant for instability and broken clusters, no way to fix<br>
>broken clusters. And all of that with recommended settings, and in our<br>
>case, enterprise hardware underneath.<br>
><br>
><br>
>I have been one of the people asking questions. I sometimes get an<br>
>answer, which I appreciate. Other times not. But I'm not paying for<br>
>support in this forum so I appreciate what I can get. My questions<br>
>are sometimes very hard to summarize and I can't say I've been offering<br>
>help as much as I ask. I think I will try to do better.<br>
><br>
><br>
>Just to counter with something cool....<br>
>As we speak now, I'm working on a 2,000 node cluster that will soon be<br>
>a<br>
>5120 node cluster. We're validating it with the newest version of our<br>
>cluster manager.<br>
><br>
>It has 12 leader nodes (soon to have 24) that are gluster servers and<br>
>gnfs servers.<br>
><br>
>I am validating Gluster7.2 (updating from 4.6). Things are looking very<br>
>good. 5120 nodes using RO NFS root with RW NFS overmounts (for things<br>
>like /var, /etc, ...)...<br>
>- boot 1 (where each node creates a RW XFS image on top of NFS for its<br>
> writable area then syncs /var, /etc, etc) -- full boot is 15-16<br>
> minutes for 2007 nodes.<br>
>- boot 2 (where the writable area pre-exists and is reused, just<br>
> re-rsynced) -- 8-9 minutes to boot 2007 nodes.<br>
><br>
>This is similar to gluster 4, but I think it's saying something to not<br>
>have had any failures in this setup on the bleeding edge release level.<br>
><br>
>We also use a different volume shared between the leaders and the head<br>
>node for shared-storage consoles and system logs. It's working great.<br>
><br>
>I haven't had time to test other solutions. Our old solution from SGI<br>
>days (ICE, ICE X, etc) was a different model where each leader served<br>
>a set of nodes and NFS-booted 288 or so. No shared storage.<br>
><br>
>Like you, I've wondered if something else matches this solution. We<br>
>like<br>
>the shared storage and the ability for a leader to drop and not take<br>
>288 noes with it.<br>
><br>
>(All nodes running RHEL8.0, Glusterfs 72, CTDB 4.9.1)<br>
><br>
><br>
><br>
>So we can say gluster is providing the network boot solution for now<br>
>two<br>
>supercomputers.<br>
><br>
><br>
><br>
>Erik<br>
>________<br>
><br>
>Community Meeting Calendar:<br>
><br>
>APAC Schedule -<br>
>Every 2nd and 4th Tuesday at 11:30 AM IST<br>
>Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
><br>
>NA/EMEA Schedule -<br>
>Every 1st and 3rd Tuesday at 01:00 PM EDT<br>
>Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
><br>
>Gluster-users mailing list<br>
><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
<br>
Hi Stefan,<br>
<br>
It seems that devs are not so active in the mailing lists, but based on my experience the bugs will be fixed in a reasonable timeframe. I admit that I was quite frustrated when my Gluster v6.5 to v6.6 upgrade made my lab useless for 2 weeks and the only help came from oVirt Dev, while gluster-users/devel were semi-silent.<br>
Yet, I'm not paying for any support and I know that any help here is just a good will.<br>
I hope this has nothing in common with the recent acquisition from IBM, but we will see.<br>
<br>
<br>
There is a reason why Red Hat clients are still using Gluster v3 (even with backports) - it is the most tested version in Gluster.<br>
For me Gluster v4+ compared to v3 is like Fedora to RHEL. After all, the upstream is not so well tested and Gluster community is taking over here - reporting bugs, sharing workarounds, giving advices .<br>
<br>
Of course, if you need rock-solid Gluster environment - you definately need the enterprise solution with it's 24/7 support.<br>
<br>
Keep in mind that even the most expensive storage arrays break after an upgrade (it happened 3 times for less than 2 weeks where 2k+ machines were read-only, before the vendor provided a new patch), so the issues in Gluster are nothing new and we should not forget that Gluster is free (and doesn't costs millions like some arrays).<br>
The only mitigation is to thoroughly test each patch on a cluster that provides storage for your dev/test clients.<br>
<br>
I hope you didn't understand me wrong - just lower your expectations -> even arrays for millions break , so Gluster is not an exclusion , but at least it's OpenSource and free.<br>
<br>
Best Regards,<br>
Strahil Nikolov<br>
<br>
________<br>
<br>
Community Meeting Calendar:<br>
<br>
APAC Schedule -<br>
Every 2nd and 4th Tuesday at 11:30 AM IST<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
NA/EMEA Schedule -<br>
Every 1st and 3rd Tuesday at 01:00 PM EDT<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature">Regards,<br>Hari Gowtham.</div>