[Gluster-users] glusterfsd process spinning

[Franco Broi]

dd read test on the disk shows 700MB/Sec, which is about normal for
these bricks.

On Sun, 2014-06-01 at 13:23 +0800, Franco Broi wrote: 
> The volume is almost completely idle now and the CPU for the brick
> process has returned to normal. I've included the profile and I think it
> shows the latency for the bad brick (data12) is unusually high, probably
> indicating the filesystem is at fault after all??
> 
> On Sun, 2014-06-01 at 01:01 -0400, Pranith Kumar Karampuri wrote: 
> > Franco,
> >     Could you do the following to get more information:
> > 
> > "gluster volume profile <volname> start"
> > 
> > Wait for some time, this will start gathering what operations are coming to all the bricks"
> > Now execute "gluster volume profile <volname> info" > /file/you/should/reply/to/this/mail/with
> > 
> > Then execute:
> > gluster volume profile <volname> stop
> > 
> > Lets see if this throws any light on the problem at hand
> > 
> > Pranith
> > ----- Original Message -----
> > > From: "Franco Broi" <franco.broi at iongeo.com>
> > > To: gluster-users at gluster.org
> > > Sent: Sunday, June 1, 2014 9:02:48 AM
> > > Subject: [Gluster-users] glusterfsd process spinning
> > > 
> > > Hi
> > > 
> > > I've been suffering from continual problems with my gluster filesystem
> > > slowing down due to what I thought was congestion on a single brick
> > > being caused by a problem with the underlying filesystem running slow
> > > but I've just noticed that the glusterfsd process for that particular
> > > brick is running at 100%+, even when the filesystem is almost idle.
> > > 
> > > I've done a couple of straces of the brick and another on the same
> > > server, does the high number of futex errors give any clues as to what
> > > might be wrong?
> > > 
> > > % time     seconds  usecs/call     calls    errors syscall
> > > ------ ----------- ----------- --------- --------- ----------------
> > > 45.58    0.027554           0    191665     20772 futex
> > > 28.26    0.017084           0    137133           readv
> > > 26.04    0.015743           0     66259           epoll_wait
> > >   0.13    0.000077           3        23           writev
> > >   0.00    0.000000           0         1           epoll_ctl
> > > ------ ----------- ----------- --------- --------- ----------------
> > > 100.00    0.060458                395081     20772 total
> > > 
> > > % time     seconds  usecs/call     calls    errors syscall
> > > ------ ----------- ----------- --------- --------- ----------------
> > > 99.25    0.334020         133      2516           epoll_wait
> > >   0.40    0.001347           0      4090        26 futex
> > >   0.35    0.001192           0      5064           readv
> > >   0.00    0.000000           0        20           writev
> > > ------ ----------- ----------- --------- --------- ----------------
> > > 100.00    0.336559                 11690        26 total
> > > 
> > > 
> > > 
> > > Cheers,
> > > 
> > > _______________________________________________
> > > Gluster-users mailing list
> > > Gluster-users at gluster.org
> > > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> > > 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users