[Gluster-users] Perfomance issue on a 90+% full file system

[Ellison, Bob]

Dan,
Thanks very much for this (and to all that replied!).

I took my test system down to 80% full and the system went back to the expect performance rate. There must be some high(90%)/(80%)low water triggers in the code to tell me to buy more storage (which I have started to do :-).

Thanks again.
Bob

-----Original Message-----
From: Dan Mons [mailto:dmons at cuttingedge.com.au] 
Sent: Monday, October 06, 2014 6:41 PM
To: Ellison, Bob
Cc: gluster-users at gluster.org
Subject: Re: [Gluster-users] Perfomance issue on a 90+% full file system

Yup, pretty common for us.  Once we hit ~90% on either of our two production clusters (107 TB usable each), performance takes a beating.

I don't consider this a problem, per se.  Most file systems (clustered or otherwise) are the same.  I consider a high water mark for any production file system to be 80% (and I consider that vendor agnostic), at which time action should be taken to begin clean up.
That's good sysadminning 101.

-Dan

----------------
Dan Mons
Unbreaker of broken things
Cutting Edge
http://cuttingedge.com.au


On 7 October 2014 08:36, Ellison, Bob <bob.ellison at ccur.com> wrote:
> Hello,
>
> My glusterfs-3.4.2-1.el6 is having a performance issue. It was working 
> fine until the 100TB file system hit ~90% full. I was seeing around 
> 90Mb/s for the last 10 months. This then dropped to 40Mb/s. Since 
> nothing changed on the system, I focused on the transition to the 90% 
> full file system.  I also found that the 6 undelaying XFS files systems were pretty fragmented (~56%).
>
>
>
> We are using gluster to achieve a large flat file system. This is a 
> single server/node configuration, so no network issues are involved.
>
>
>
> As the problem is on a production system, I setup a smaller test 
> system. I monitored performance and was able to duplicate the problem 
> (90MB/s up to 90% full, then a drop off in performance thereafter). 
> The closer to 100% full, the lower the throughput.
>
>
>
> I then started deleting content from the test server. I was surprised 
> to find that the performance did NOT increase – it stayed the same. I 
> took the test system to file system 50% full but still saw 40Mb/s!
>
>
>
> The test I set up was designed to fragment the XFS partitions (to 
> mimic the production system state). I defragmented online 
> successfully, however performance did not increase.
>
>
>
> I am currently trying a rebalance across the 6 XFS partitions to see 
> if that helps.
>
>
>
> I was wondering if anyone remembers a problem like this? Is there a 
> chance that the rebalance will get me back to the normally seen 
> performance? Would upgrading gluster fix this?
>
>
>
> Thanks,
>
> Bob
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users