[Gluster-users] performance

Artem Russakovskii archon810 at gmail.com
Wed Aug 12 22:08:14 UTC 2020 

Hmm, in our case of running gluster across Linode block storage (which
itself runs inside Ceph, as I found out), the only thing that helped with
the hangs so far was defragmenting xfs.

I tried changing many things, including the scheduler to "none" and
this performance.write-behind-window-size setting, and nothing seemed to
help or provide any meaningful difference.

Sincerely,
Artem

--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | @ArtemR <http://twitter.com/ArtemR>


On Fri, Aug 7, 2020 at 11:28 AM Computerisms Corporation <
bob at computerisms.ca> wrote:

> Hi Artem and others,
>
> Happy to report the system has been relatively stable for the remainder
> of the week.  I have one wordpress site that seems to get hung processes
> when someone logs in with an incorrect password.  Since it is only one,
> and reliably reproduceable, I am not sure if the issue is to do with
> Gluster or Wordpress itself, but afaik it was not doing it some months
> back before the system was using Gluster so I am guessing some combo of
> both.
>
> Regardless, that is the one and only time apache processes stacked up to
> over 150, and that still only brought the load average up to just under
> 25; the system did go a bit sluggish, but remained fairly responsive
> throughout until I restarted apache.  Otherwise 15 minute load average
> consistently runs between 8 and 11 during peak hours and between 4 and 7
> during off hours, and other than the one time I have not seen the
> one-minute load average go over 15.  all resources still spike to full
> capacity from time to time, but it never remains that way for long like
> it did before.
>
> For site responsiveness, first visit to any given site is quite slow,
> like 3-5 seconds on straight html pages, 10-15 seconds for some of the
> more bloated WP themes, but clicking links within the site after the
> first page is loaded is relatively quick, like 1 second on straight html
> pages, and ~5-6 seconds on the bloated themes.  Again, not sure if that
> is a Gluster related thing or something else.
>
> So, still holding my breath a bit, but seems this solution is working,
> at least for me.  I haven't played with any of the other settings yet to
> see if I can improve it further, probably will next week.  thinking to
> increase the write behind window size further to see what happens, as
> well as play with the settings suggested by Strahil.
>
> On 2020-08-05 5:28 p.m., Artem Russakovskii wrote:
> > I'm very curious whether these improvements hold up over the next few
> > days. Please report back.
> >
> > Sincerely,
> > Artem
> >
> > --
> > Founder, Android Police <http://www.androidpolice.com>, APK Mirror
> > <http://www.apkmirror.com/>, Illogical Robot LLC
> > beerpla.net <http://beerpla.net/> | @ArtemR <http://twitter.com/ArtemR>
> >
> >
> > On Wed, Aug 5, 2020 at 9:44 AM Computerisms Corporation
> > <bob at computerisms.ca <mailto:bob at computerisms.ca>> wrote:
> >
> >     Hi List,
> >
> >      > So, we just moved into a quieter time of the day, but maybe I just
> >      > stumbled onto something.  I was trying to figure out if/how I
> could
> >      > throw more RAM at the problem.  gluster docs says write behind is
> >     not a
> >      > cache unless flush-behind is on.  So seems that is a way to throw
> >     ram to
> >      > it?  I put performance.write-behind-window-size: 512MB and
> >      > performance.flush-behind: on and the whole system calmed down
> pretty
> >      > much immediately.  could be just timing, though, will have to see
> >      > tomorrow during business hours whether the system stays at a
> >     reasonable
> >      > load.
> >
> >     so reporting back that this seems to have definitely had a
> significant
> >     positive effect.
> >
> >     So far today I have not seen the load average climb over 13 with the
> >     15minute average hovering around 7.  cpus are still spiking from
> >     time to
> >     time, but they are not staying maxed out all the time, and
> frequently I
> >     am seeing brief periods of up to 80% idle.  glusterfs process still
> >     spiking up to 180% or so, but consistently running around 70%, and
> the
> >     brick processes still spiking up to 70-80%, but consistently running
> >     around 20%.  Disk has only been above 50% in atop once so far today
> >     when
> >     it spiked up to 92%, and still lots of RAM left over.  So far nload
> >     even
> >     seems indicates I could get away with a 100Mbit network connection.
> >     Websites are snappy relative to what they were, still a bit sluggish
> on
> >     the first page of any given site, but tolerable or close to.  Apache
> >     processes are opening and closing right away, instead of stacking up.
> >
> >     Overall, system is performing pretty much like I would expect it to
> >     without gluster.  I haven't played with any of the other settings
> yet,
> >     just going to leave it like this for a day.
> >
> >     I have to admit I am a little bit suspicious.  I have been arguing
> with
> >     Gluster for a very long time, and I have never known it to play this
> >     nice.  kind feels like when your girl tells you she is "fine";
> >     conversation has stopped, but you aren't really sure if it's done...
> >
> >      >
> >      > I will still test the other options you suggested tonight,
> >     though, this
> >      > is probably too good to be true.
> >      >
> >      > Can't thank you enough for your input, Strahil, your help is truly
> >      > appreciated!
> >      >
> >      >
> >      >
> >      >
> >      >
> >      >
> >      >>
> >      >>>>
> >      >>>>
> >      >>>> Best Regards,
> >      >>>> Strahil Nikolov
> >      >>>>
> >      >>> ________
> >      >>>
> >      >>>
> >      >>>
> >      >>> Community Meeting Calendar:
> >      >>>
> >      >>> Schedule -
> >      >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >      >>> Bridge: https://bluejeans.com/441850968
> >      >>>
> >      >>> Gluster-users mailing list
> >      >>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> >      >>> https://lists.gluster.org/mailman/listinfo/gluster-users
> >      > ________
> >      >
> >      >
> >      >
> >      > Community Meeting Calendar:
> >      >
> >      > Schedule -
> >      > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >      > Bridge: https://bluejeans.com/441850968
> >      >
> >      > Gluster-users mailing list
> >      > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> >      > https://lists.gluster.org/mailman/listinfo/gluster-users
> >     ________
> >
> >
> >
> >     Community Meeting Calendar:
> >
> >     Schedule -
> >     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >     Bridge: https://bluejeans.com/441850968
> >
> >     Gluster-users mailing list
> >     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> >     https://lists.gluster.org/mailman/listinfo/gluster-users
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200812/2c0b9712/attachment.html>

More information about the Gluster-users mailing list