[Gluster-users] glusterfsd process spinning

[Pranith Kumar Karampuri]

----- Original Message -----
> From: "Franco Broi" <franco.broi at iongeo.com>
> To: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> Cc: gluster-users at gluster.org
> Sent: Sunday, June 1, 2014 10:53:51 AM
> Subject: Re: [Gluster-users] glusterfsd process spinning
> 
> 
> The volume is almost completely idle now and the CPU for the brick
> process has returned to normal. I've included the profile and I think it
> shows the latency for the bad brick (data12) is unusually high, probably
> indicating the filesystem is at fault after all??

I am not sure if we can believe the outputs now that you say the brick returned to normal. Next time it is acting up, do the same procedure and post the result.

Pranith
> 
> On Sun, 2014-06-01 at 01:01 -0400, Pranith Kumar Karampuri wrote:
> > Franco,
> >     Could you do the following to get more information:
> > 
> > "gluster volume profile <volname> start"
> > 
> > Wait for some time, this will start gathering what operations are coming to
> > all the bricks"
> > Now execute "gluster volume profile <volname> info" >
> > /file/you/should/reply/to/this/mail/with
> > 
> > Then execute:
> > gluster volume profile <volname> stop
> > 
> > Lets see if this throws any light on the problem at hand
> > 
> > Pranith
> > ----- Original Message -----
> > > From: "Franco Broi" <franco.broi at iongeo.com>
> > > To: gluster-users at gluster.org
> > > Sent: Sunday, June 1, 2014 9:02:48 AM
> > > Subject: [Gluster-users] glusterfsd process spinning
> > > 
> > > Hi
> > > 
> > > I've been suffering from continual problems with my gluster filesystem
> > > slowing down due to what I thought was congestion on a single brick
> > > being caused by a problem with the underlying filesystem running slow
> > > but I've just noticed that the glusterfsd process for that particular
> > > brick is running at 100%+, even when the filesystem is almost idle.
> > > 
> > > I've done a couple of straces of the brick and another on the same
> > > server, does the high number of futex errors give any clues as to what
> > > might be wrong?
> > > 
> > > % time     seconds  usecs/call     calls    errors syscall
> > > ------ ----------- ----------- --------- --------- ----------------
> > > 45.58    0.027554           0    191665     20772 futex
> > > 28.26    0.017084           0    137133           readv
> > > 26.04    0.015743           0     66259           epoll_wait
> > >   0.13    0.000077           3        23           writev
> > >   0.00    0.000000           0         1           epoll_ctl
> > > ------ ----------- ----------- --------- --------- ----------------
> > > 100.00    0.060458                395081     20772 total
> > > 
> > > % time     seconds  usecs/call     calls    errors syscall
> > > ------ ----------- ----------- --------- --------- ----------------
> > > 99.25    0.334020         133      2516           epoll_wait
> > >   0.40    0.001347           0      4090        26 futex
> > >   0.35    0.001192           0      5064           readv
> > >   0.00    0.000000           0        20           writev
> > > ------ ----------- ----------- --------- --------- ----------------
> > > 100.00    0.336559                 11690        26 total
> > > 
> > > 
> > > 
> > > Cheers,
> > > 
> > > _______________________________________________
> > > Gluster-users mailing list
> > > Gluster-users at gluster.org
> > > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> > > 
> 
>