[Gluster-users] Re. Gluster machines slowing down over time

[Dan Bretherton]

> *Tom Hall* thattommyhall at gmail.com 
> <mailto:gluster-users%40gluster.org?Subject=Re:%20Re%3A%20%5BGluster-users%5D%20Gluster%20machines%20slowing%20down%20over%20time&In-Reply-To=%3CCAMvQ07jForynyik2NYmYnKq-kAa5wHoZLwOU8HDtYXKChM2ZXQ%40mail.gmail.com%3E>
> /Wed Dec 12 12:47:09 UTC 2012/
>
>   * Previous message: [Gluster-users] Gluster machines slowing down
>     over time
>     <http://gluster.org/pipermail/gluster-users/2012-December/034974.html>
>   * Next message: [Gluster-users] glusterfs-3.4.0qa5 released
>     <http://gluster.org/pipermail/gluster-users/2012-December/034973.html>
>   * *Messages sorted by:* [ date ]
>     <http://gluster.org/pipermail/gluster-users/2012-December/date.html#34976>
>     [ thread ]
>     <http://gluster.org/pipermail/gluster-users/2012-December/thread.html#34976>
>     [ subject ]
>     <http://gluster.org/pipermail/gluster-users/2012-December/subject.html#34976>
>     [ author ]
>     <http://gluster.org/pipermail/gluster-users/2012-December/author.html#34976>
>
>
> ------------------------------------------------------------------------
> On Wed, Dec 12, 2012 at 10:41 AM, Brian Candler <B.Candler at pobox.com  <http://supercolony.gluster.org/mailman/listinfo/gluster-users>> wrote:
>
> >/  On Tue, Dec 11, 2012 at 04:13:15PM +0000, Tom Hall wrote:
> />/  >    I have 2 gluster servers in replicated mode on EC2 with ~4G RAM
> />/  >
> />/  >    CPU and RAM look fine but over time the system becomes sluggish,
> />/  >    particularly networking.
> />/  >
> />/  >    I notice when sshing into the machine takes ages and running remote
> />/  >    commands with capistrano takes longer and longer.
> />/
> />/  Do you have swap configured? Is the system going into swap?
> /
>
> No swap (default on that size EC2 instance)
>               total       used       free     shared    buffers     cached
> Mem:          3750       3731         19          0        274       2929
> -/+ buffers/cache:        526       3224
> Swap:            0          0          0
>
> Caching heavily but not using so much RAM
>
>
> >/
> /
>
> >/
> /Have a look at output of "free", "vmstat 5" (for a minute or two),
> >/  "iostat 5 -x" (ditto)
> />/
> />/
> /I'll get it next time we experience the issue.
> Seems to slow down over a week or so so it might be a while...
>
> Thanks guys!
> Tom
Dear All-
I too have been experiencing a gradual slowing of GlusterFS over time, 
accompanied by increases in CPU load and memory usage. However, the 
storage server CPU load does not seem to be related to the I/O load 
imposed by users.  The following three Ganglia plots for the past month 
illustrate the point.

http://www.nerc-essc.ac.uk/~dab/GlusterFS_Ganglia_CPU_Mem_MonthTo8thMarch2013.jpeg
http://www.nerc-essc.ac.uk/~dab/Rocks_Ganglia_CPU_Mem_MonthTo8thMarch2013.jpeg
http://www.nerc-essc.ac.uk/~dab/GlusterFS-remus_Ganglia_CPU_Mem_MonthTo8thMarch2013.jpeg

The first plot shows the average storage server CPU load and memory 
usage rising during the course of the past month, which became almost 
unmanageable and prompted me to restart all the servers a few days ago.

The second plot shows the CPU load and memory usage on my small Rocks 
compute cluster, which is responsible for most of the I/O load on the 
GlusterFS cluster.

The third chart shows CPU and memory usage on a typical storage server, 
which has been restarted a couple of times in the past month.  The first 
restart was at the beginning of week 7 on the plot, and the second was a 
couple days ago at the same time as the other servers.

There is no evidence that the I/O load on the storage cluster has been 
increasing over the past month, so why does the CPU load on the storage 
servers  gradually increase (along with memory usage) until they have to 
be restarted?  Restarting the all the servers can be very disruptive 
because it seems to take a long time for them all to start talking to 
each other properly again.  Once everything has settled down everyone 
breathes a sigh of relief and enjoys relatively fast and responsive I/O 
for a few days, before things start to slow down again.  It looks as if 
I'm going to have to start doing this routinely every weekend, but this 
isn't how I was planning to spend my weekends for the foreseeable 
future.  Are there any patches in the pipeline that are likely to 
address this issue?

-Dan

--
Dan Bretherton
ESSC Computer System Manager
Department of Meteorology
Harry Pitt Building, 3 Earley Gate
University of Reading
Reading, RG6 7BE (or RG6 6AL for postal service deliveries)
UK
Tel. +44 118 378 5205, Fax: +44 118 378 6413
-- 
## Please sponsor me to run in VSO's 30km Race to the Eye ##
##        http://www.justgiving.com/DanBretherton         ##

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130308/598da05e/attachment.html>