[Gluster-users] glusterfsd process spinning

[Franco Broi]

Hi Pranith

Here's a listing of the brick logs, looks very odd especially the size
of the log for data10.

[root at nas3 bricks]# ls -ltrh
total 2.6G
-rw------- 1 root root 381K May 13 12:15 data12-gvol.log-20140511
-rw------- 1 root root 430M May 13 12:15 data11-gvol.log-20140511
-rw------- 1 root root 328K May 13 12:15 data9-gvol.log-20140511
-rw------- 1 root root 2.0M May 13 12:15 data10-gvol.log-20140511
-rw------- 1 root root    0 May 18 03:43 data10-gvol.log-20140525
-rw------- 1 root root    0 May 18 03:43 data11-gvol.log-20140525
-rw------- 1 root root    0 May 18 03:43 data12-gvol.log-20140525
-rw------- 1 root root    0 May 18 03:43 data9-gvol.log-20140525
-rw------- 1 root root    0 May 25 03:19 data10-gvol.log-20140601
-rw------- 1 root root    0 May 25 03:19 data11-gvol.log-20140601
-rw------- 1 root root    0 May 25 03:19 data9-gvol.log-20140601
-rw------- 1 root root  98M May 26 03:04 data12-gvol.log-20140518
-rw------- 1 root root    0 Jun  1 03:37 data10-gvol.log
-rw------- 1 root root    0 Jun  1 03:37 data11-gvol.log
-rw------- 1 root root    0 Jun  1 03:37 data12-gvol.log
-rw------- 1 root root    0 Jun  1 03:37 data9-gvol.log
-rw------- 1 root root 1.8G Jun  2 16:35 data10-gvol.log-20140518
-rw------- 1 root root 279M Jun  2 16:35 data9-gvol.log-20140518
-rw------- 1 root root 328K Jun  2 16:35 data12-gvol.log-20140601
-rw------- 1 root root 8.3M Jun  2 16:35 data11-gvol.log-20140518

Too big to post everything.

Cheers,

On Sun, 2014-06-01 at 22:00 -0400, Pranith Kumar Karampuri wrote: 
> 
> ----- Original Message -----
> > From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> > To: "Franco Broi" <franco.broi at iongeo.com>
> > Cc: gluster-users at gluster.org
> > Sent: Monday, June 2, 2014 7:01:34 AM
> > Subject: Re: [Gluster-users] glusterfsd process spinning
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Franco Broi" <franco.broi at iongeo.com>
> > > To: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> > > Cc: gluster-users at gluster.org
> > > Sent: Sunday, June 1, 2014 10:53:51 AM
> > > Subject: Re: [Gluster-users] glusterfsd process spinning
> > > 
> > > 
> > > The volume is almost completely idle now and the CPU for the brick
> > > process has returned to normal. I've included the profile and I think it
> > > shows the latency for the bad brick (data12) is unusually high, probably
> > > indicating the filesystem is at fault after all??
> > 
> > I am not sure if we can believe the outputs now that you say the brick
> > returned to normal. Next time it is acting up, do the same procedure and
> > post the result.
> 
> On second thought may be its not a bad idea to inspect the log files of the bricks in nas3. Could you post them.
> 
> Pranith
> 
> > 
> > Pranith
> > > 
> > > On Sun, 2014-06-01 at 01:01 -0400, Pranith Kumar Karampuri wrote:
> > > > Franco,
> > > >     Could you do the following to get more information:
> > > > 
> > > > "gluster volume profile <volname> start"
> > > > 
> > > > Wait for some time, this will start gathering what operations are coming
> > > > to
> > > > all the bricks"
> > > > Now execute "gluster volume profile <volname> info" >
> > > > /file/you/should/reply/to/this/mail/with
> > > > 
> > > > Then execute:
> > > > gluster volume profile <volname> stop
> > > > 
> > > > Lets see if this throws any light on the problem at hand
> > > > 
> > > > Pranith
> > > > ----- Original Message -----
> > > > > From: "Franco Broi" <franco.broi at iongeo.com>
> > > > > To: gluster-users at gluster.org
> > > > > Sent: Sunday, June 1, 2014 9:02:48 AM
> > > > > Subject: [Gluster-users] glusterfsd process spinning
> > > > > 
> > > > > Hi
> > > > > 
> > > > > I've been suffering from continual problems with my gluster filesystem
> > > > > slowing down due to what I thought was congestion on a single brick
> > > > > being caused by a problem with the underlying filesystem running slow
> > > > > but I've just noticed that the glusterfsd process for that particular
> > > > > brick is running at 100%+, even when the filesystem is almost idle.
> > > > > 
> > > > > I've done a couple of straces of the brick and another on the same
> > > > > server, does the high number of futex errors give any clues as to what
> > > > > might be wrong?
> > > > > 
> > > > > % time     seconds  usecs/call     calls    errors syscall
> > > > > ------ ----------- ----------- --------- --------- ----------------
> > > > > 45.58    0.027554           0    191665     20772 futex
> > > > > 28.26    0.017084           0    137133           readv
> > > > > 26.04    0.015743           0     66259           epoll_wait
> > > > >   0.13    0.000077           3        23           writev
> > > > >   0.00    0.000000           0         1           epoll_ctl
> > > > > ------ ----------- ----------- --------- --------- ----------------
> > > > > 100.00    0.060458                395081     20772 total
> > > > > 
> > > > > % time     seconds  usecs/call     calls    errors syscall
> > > > > ------ ----------- ----------- --------- --------- ----------------
> > > > > 99.25    0.334020         133      2516           epoll_wait
> > > > >   0.40    0.001347           0      4090        26 futex
> > > > >   0.35    0.001192           0      5064           readv
> > > > >   0.00    0.000000           0        20           writev
> > > > > ------ ----------- ----------- --------- --------- ----------------
> > > > > 100.00    0.336559                 11690        26 total
> > > > > 
> > > > > 
> > > > > 
> > > > > Cheers,
> > > > > 
> > > > > _______________________________________________
> > > > > Gluster-users mailing list
> > > > > Gluster-users at gluster.org
> > > > > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> > > > > 
> > > 
> > > 
> >