[Gluster-users] Perfomance issue on a 90+% full file system

[Juan José Pavlik Salles]

Same here, we try to keep them under 80% too.

2014-10-06 19:40 GMT-03:00 Dan Mons <dmons at cuttingedge.com.au>:

> Yup, pretty common for us.  Once we hit ~90% on either of our two
> production clusters (107 TB usable each), performance takes a beating.
>
> I don't consider this a problem, per se.  Most file systems (clustered
> or otherwise) are the same.  I consider a high water mark for any
> production file system to be 80% (and I consider that vendor
> agnostic), at which time action should be taken to begin clean up.
> That's good sysadminning 101.
>
> -Dan
>
> ----------------
> Dan Mons
> Unbreaker of broken things
> Cutting Edge
> http://cuttingedge.com.au
>
>
> On 7 October 2014 08:36, Ellison, Bob <bob.ellison at ccur.com> wrote:
> > Hello,
> >
> > My glusterfs-3.4.2-1.el6 is having a performance issue. It was working
> fine
> > until the 100TB file system hit ~90% full. I was seeing around 90Mb/s for
> > the last 10 months. This then dropped to 40Mb/s. Since nothing changed on
> > the system, I focused on the transition to the 90% full file system.  I
> also
> > found that the 6 undelaying XFS files systems were pretty fragmented
> (~56%).
> >
> >
> >
> > We are using gluster to achieve a large flat file system. This is a
> single
> > server/node configuration, so no network issues are involved.
> >
> >
> >
> > As the problem is on a production system, I setup a smaller test system.
> I
> > monitored performance and was able to duplicate the problem (90MB/s up to
> > 90% full, then a drop off in performance thereafter). The closer to 100%
> > full, the lower the throughput.
> >
> >
> >
> > I then started deleting content from the test server. I was surprised to
> > find that the performance did NOT increase – it stayed the same. I took
> the
> > test system to file system 50% full but still saw 40Mb/s!
> >
> >
> >
> > The test I set up was designed to fragment the XFS partitions (to mimic
> the
> > production system state). I defragmented online successfully, however
> > performance did not increase.
> >
> >
> >
> > I am currently trying a rebalance across the 6 XFS partitions to see if
> that
> > helps.
> >
> >
> >
> > I was wondering if anyone remembers a problem like this? Is there a
> chance
> > that the rebalance will get me back to the normally seen performance?
> Would
> > upgrading gluster fix this?
> >
> >
> >
> > Thanks,
> >
> > Bob
> >
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users




-- 
Pavlik Salles Juan José
Blog - http://viviendolared.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141006/87dcec45/attachment.html>