[Gluster-users] Perfomance issue on a 90+% full file system

[Ellison, Bob]

Hello,
My glusterfs-3.4.2-1.el6 is having a performance issue. It was working fine until the 100TB file system hit ~90% full. I was seeing around 90Mb/s for the last 10 months. This then dropped to 40Mb/s. Since nothing changed on the system, I focused on the transition to the 90% full file system.  I also found that the 6 undelaying XFS files systems were pretty fragmented (~56%).

We are using gluster to achieve a large flat file system. This is a single server/node configuration, so no network issues are involved.

As the problem is on a production system, I setup a smaller test system. I monitored performance and was able to duplicate the problem (90MB/s up to 90% full, then a drop off in performance thereafter). The closer to 100% full, the lower the throughput.

I then started deleting content from the test server. I was surprised to find that the performance did NOT increase - it stayed the same. I took the test system to file system 50% full but still saw 40Mb/s!

The test I set up was designed to fragment the XFS partitions (to mimic the production system state). I defragmented online successfully, however performance did not increase.

I am currently trying a rebalance across the 6 XFS partitions to see if that helps.

I was wondering if anyone remembers a problem like this? Is there a chance that the rebalance will get me back to the normally seen performance? Would upgrading gluster fix this?

Thanks,
Bob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141006/8acb1ff3/attachment.html>