<div dir="ltr">Hmm, in our case of running gluster across Linode block storage (which itself runs inside Ceph, as I found out), the only thing that helped with the hangs so far was defragmenting xfs.<div><br></div><div>I tried changing many things, including the scheduler to "none" and this performance.write-behind-window-size setting, and nothing seemed to help or provide any meaningful difference.<br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><br>Sincerely,<br>Artem<br><br>--<br>Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8px" target="_blank">APK Mirror</a><span style="font-size:12.8px">, Illogical Robot LLC</span></div><div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 7, 2020 at 11:28 AM Computerisms Corporation <<a href="mailto:bob@computerisms.ca">bob@computerisms.ca</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Artem and others,<br>
<br>
Happy to report the system has been relatively stable for the remainder <br>
of the week. I have one wordpress site that seems to get hung processes <br>
when someone logs in with an incorrect password. Since it is only one, <br>
and reliably reproduceable, I am not sure if the issue is to do with <br>
Gluster or Wordpress itself, but afaik it was not doing it some months <br>
back before the system was using Gluster so I am guessing some combo of <br>
both.<br>
<br>
Regardless, that is the one and only time apache processes stacked up to <br>
over 150, and that still only brought the load average up to just under <br>
25; the system did go a bit sluggish, but remained fairly responsive <br>
throughout until I restarted apache. Otherwise 15 minute load average <br>
consistently runs between 8 and 11 during peak hours and between 4 and 7 <br>
during off hours, and other than the one time I have not seen the <br>
one-minute load average go over 15. all resources still spike to full <br>
capacity from time to time, but it never remains that way for long like <br>
it did before.<br>
<br>
For site responsiveness, first visit to any given site is quite slow, <br>
like 3-5 seconds on straight html pages, 10-15 seconds for some of the <br>
more bloated WP themes, but clicking links within the site after the <br>
first page is loaded is relatively quick, like 1 second on straight html <br>
pages, and ~5-6 seconds on the bloated themes. Again, not sure if that <br>
is a Gluster related thing or something else.<br>
<br>
So, still holding my breath a bit, but seems this solution is working, <br>
at least for me. I haven't played with any of the other settings yet to <br>
see if I can improve it further, probably will next week. thinking to <br>
increase the write behind window size further to see what happens, as <br>
well as play with the settings suggested by Strahil.<br>
<br>
On 2020-08-05 5:28 p.m., Artem Russakovskii wrote:<br>
> I'm very curious whether these improvements hold up over the next few <br>
> days. Please report back.<br>
> <br>
> Sincerely,<br>
> Artem<br>
> <br>
> --<br>
> Founder, Android Police <<a href="http://www.androidpolice.com" rel="noreferrer" target="_blank">http://www.androidpolice.com</a>>, APK Mirror <br>
> <<a href="http://www.apkmirror.com/" rel="noreferrer" target="_blank">http://www.apkmirror.com/</a>>, Illogical Robot LLC<br>
> <a href="http://beerpla.net" rel="noreferrer" target="_blank">beerpla.net</a> <<a href="http://beerpla.net/" rel="noreferrer" target="_blank">http://beerpla.net/</a>> | @ArtemR <<a href="http://twitter.com/ArtemR" rel="noreferrer" target="_blank">http://twitter.com/ArtemR</a>><br>
> <br>
> <br>
> On Wed, Aug 5, 2020 at 9:44 AM Computerisms Corporation <br>
> <<a href="mailto:bob@computerisms.ca" target="_blank">bob@computerisms.ca</a> <mailto:<a href="mailto:bob@computerisms.ca" target="_blank">bob@computerisms.ca</a>>> wrote:<br>
> <br>
> Hi List,<br>
> <br>
> > So, we just moved into a quieter time of the day, but maybe I just<br>
> > stumbled onto something. I was trying to figure out if/how I could<br>
> > throw more RAM at the problem. gluster docs says write behind is<br>
> not a<br>
> > cache unless flush-behind is on. So seems that is a way to throw<br>
> ram to<br>
> > it? I put performance.write-behind-window-size: 512MB and<br>
> > performance.flush-behind: on and the whole system calmed down pretty<br>
> > much immediately. could be just timing, though, will have to see<br>
> > tomorrow during business hours whether the system stays at a<br>
> reasonable<br>
> > load.<br>
> <br>
> so reporting back that this seems to have definitely had a significant<br>
> positive effect.<br>
> <br>
> So far today I have not seen the load average climb over 13 with the<br>
> 15minute average hovering around 7. cpus are still spiking from<br>
> time to<br>
> time, but they are not staying maxed out all the time, and frequently I<br>
> am seeing brief periods of up to 80% idle. glusterfs process still<br>
> spiking up to 180% or so, but consistently running around 70%, and the<br>
> brick processes still spiking up to 70-80%, but consistently running<br>
> around 20%. Disk has only been above 50% in atop once so far today<br>
> when<br>
> it spiked up to 92%, and still lots of RAM left over. So far nload<br>
> even<br>
> seems indicates I could get away with a 100Mbit network connection.<br>
> Websites are snappy relative to what they were, still a bit sluggish on<br>
> the first page of any given site, but tolerable or close to. Apache<br>
> processes are opening and closing right away, instead of stacking up.<br>
> <br>
> Overall, system is performing pretty much like I would expect it to<br>
> without gluster. I haven't played with any of the other settings yet,<br>
> just going to leave it like this for a day.<br>
> <br>
> I have to admit I am a little bit suspicious. I have been arguing with<br>
> Gluster for a very long time, and I have never known it to play this<br>
> nice. kind feels like when your girl tells you she is "fine";<br>
> conversation has stopped, but you aren't really sure if it's done...<br>
> <br>
> ><br>
> > I will still test the other options you suggested tonight,<br>
> though, this<br>
> > is probably too good to be true.<br>
> ><br>
> > Can't thank you enough for your input, Strahil, your help is truly<br>
> > appreciated!<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> >><br>
> >>>><br>
> >>>><br>
> >>>> Best Regards,<br>
> >>>> Strahil Nikolov<br>
> >>>><br>
> >>> ________<br>
> >>><br>
> >>><br>
> >>><br>
> >>> Community Meeting Calendar:<br>
> >>><br>
> >>> Schedule -<br>
> >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
> >>> Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
> >>><br>
> >>> Gluster-users mailing list<br>
> >>> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>><br>
> >>> <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
> > ________<br>
> ><br>
> ><br>
> ><br>
> > Community Meeting Calendar:<br>
> ><br>
> > Schedule -<br>
> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
> > Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
> ><br>
> > Gluster-users mailing list<br>
> > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>><br>
> > <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
> ________<br>
> <br>
> <br>
> <br>
> Community Meeting Calendar:<br>
> <br>
> Schedule -<br>
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
> Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
> <br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>><br>
> <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
> <br>
</blockquote></div>