[Gluster-users] Perfomance issue on a 90+% full file system

[Dan Mons]

Yup, pretty common for us.  Once we hit ~90% on either of our two
production clusters (107 TB usable each), performance takes a beating.

I don't consider this a problem, per se.  Most file systems (clustered
or otherwise) are the same.  I consider a high water mark for any
production file system to be 80% (and I consider that vendor
agnostic), at which time action should be taken to begin clean up.
That's good sysadminning 101.

-Dan

----------------
Dan Mons
Unbreaker of broken things
Cutting Edge
http://cuttingedge.com.au


On 7 October 2014 08:36, Ellison, Bob <bob.ellison at ccur.com> wrote:
> Hello,
>
> My glusterfs-3.4.2-1.el6 is having a performance issue. It was working fine
> until the 100TB file system hit ~90% full. I was seeing around 90Mb/s for
> the last 10 months. This then dropped to 40Mb/s. Since nothing changed on
> the system, I focused on the transition to the 90% full file system.  I also
> found that the 6 undelaying XFS files systems were pretty fragmented (~56%).
>
>
>
> We are using gluster to achieve a large flat file system. This is a single
> server/node configuration, so no network issues are involved.
>
>
>
> As the problem is on a production system, I setup a smaller test system. I
> monitored performance and was able to duplicate the problem (90MB/s up to
> 90% full, then a drop off in performance thereafter). The closer to 100%
> full, the lower the throughput.
>
>
>
> I then started deleting content from the test server. I was surprised to
> find that the performance did NOT increase – it stayed the same. I took the
> test system to file system 50% full but still saw 40Mb/s!
>
>
>
> The test I set up was designed to fragment the XFS partitions (to mimic the
> production system state). I defragmented online successfully, however
> performance did not increase.
>
>
>
> I am currently trying a rebalance across the 6 XFS partitions to see if that
> helps.
>
>
>
> I was wondering if anyone remembers a problem like this? Is there a chance
> that the rebalance will get me back to the normally seen performance? Would
> upgrading gluster fix this?
>
>
>
> Thanks,
>
> Bob
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users