[Gluster-users] glusterfsd process spinning
Pranith Kumar Karampuri
pkarampu at redhat.com
Wed Jun 4 01:58:53 UTC 2014
Franco,
Thanks for providing the logs. I just copied over the logs to my
machine. Most of the logs I see are related to "No such File or
Directory" I wonder what lead to this. Do you have any idea?
Pranith
On 06/02/2014 02:48 PM, Franco Broi wrote:
> Hi Pranith
>
> Here's a listing of the brick logs, looks very odd especially the size
> of the log for data10.
>
> [root at nas3 bricks]# ls -ltrh
> total 2.6G
> -rw------- 1 root root 381K May 13 12:15 data12-gvol.log-20140511
> -rw------- 1 root root 430M May 13 12:15 data11-gvol.log-20140511
> -rw------- 1 root root 328K May 13 12:15 data9-gvol.log-20140511
> -rw------- 1 root root 2.0M May 13 12:15 data10-gvol.log-20140511
> -rw------- 1 root root 0 May 18 03:43 data10-gvol.log-20140525
> -rw------- 1 root root 0 May 18 03:43 data11-gvol.log-20140525
> -rw------- 1 root root 0 May 18 03:43 data12-gvol.log-20140525
> -rw------- 1 root root 0 May 18 03:43 data9-gvol.log-20140525
> -rw------- 1 root root 0 May 25 03:19 data10-gvol.log-20140601
> -rw------- 1 root root 0 May 25 03:19 data11-gvol.log-20140601
> -rw------- 1 root root 0 May 25 03:19 data9-gvol.log-20140601
> -rw------- 1 root root 98M May 26 03:04 data12-gvol.log-20140518
> -rw------- 1 root root 0 Jun 1 03:37 data10-gvol.log
> -rw------- 1 root root 0 Jun 1 03:37 data11-gvol.log
> -rw------- 1 root root 0 Jun 1 03:37 data12-gvol.log
> -rw------- 1 root root 0 Jun 1 03:37 data9-gvol.log
> -rw------- 1 root root 1.8G Jun 2 16:35 data10-gvol.log-20140518
> -rw------- 1 root root 279M Jun 2 16:35 data9-gvol.log-20140518
> -rw------- 1 root root 328K Jun 2 16:35 data12-gvol.log-20140601
> -rw------- 1 root root 8.3M Jun 2 16:35 data11-gvol.log-20140518
>
> Too big to post everything.
>
> Cheers,
>
> On Sun, 2014-06-01 at 22:00 -0400, Pranith Kumar Karampuri wrote:
>> ----- Original Message -----
>>> From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
>>> To: "Franco Broi" <franco.broi at iongeo.com>
>>> Cc: gluster-users at gluster.org
>>> Sent: Monday, June 2, 2014 7:01:34 AM
>>> Subject: Re: [Gluster-users] glusterfsd process spinning
>>>
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Franco Broi" <franco.broi at iongeo.com>
>>>> To: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
>>>> Cc: gluster-users at gluster.org
>>>> Sent: Sunday, June 1, 2014 10:53:51 AM
>>>> Subject: Re: [Gluster-users] glusterfsd process spinning
>>>>
>>>>
>>>> The volume is almost completely idle now and the CPU for the brick
>>>> process has returned to normal. I've included the profile and I think it
>>>> shows the latency for the bad brick (data12) is unusually high, probably
>>>> indicating the filesystem is at fault after all??
>>> I am not sure if we can believe the outputs now that you say the brick
>>> returned to normal. Next time it is acting up, do the same procedure and
>>> post the result.
>> On second thought may be its not a bad idea to inspect the log files of the bricks in nas3. Could you post them.
>>
>> Pranith
>>
>>> Pranith
>>>> On Sun, 2014-06-01 at 01:01 -0400, Pranith Kumar Karampuri wrote:
>>>>> Franco,
>>>>> Could you do the following to get more information:
>>>>>
>>>>> "gluster volume profile <volname> start"
>>>>>
>>>>> Wait for some time, this will start gathering what operations are coming
>>>>> to
>>>>> all the bricks"
>>>>> Now execute "gluster volume profile <volname> info" >
>>>>> /file/you/should/reply/to/this/mail/with
>>>>>
>>>>> Then execute:
>>>>> gluster volume profile <volname> stop
>>>>>
>>>>> Lets see if this throws any light on the problem at hand
>>>>>
>>>>> Pranith
>>>>> ----- Original Message -----
>>>>>> From: "Franco Broi" <franco.broi at iongeo.com>
>>>>>> To: gluster-users at gluster.org
>>>>>> Sent: Sunday, June 1, 2014 9:02:48 AM
>>>>>> Subject: [Gluster-users] glusterfsd process spinning
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I've been suffering from continual problems with my gluster filesystem
>>>>>> slowing down due to what I thought was congestion on a single brick
>>>>>> being caused by a problem with the underlying filesystem running slow
>>>>>> but I've just noticed that the glusterfsd process for that particular
>>>>>> brick is running at 100%+, even when the filesystem is almost idle.
>>>>>>
>>>>>> I've done a couple of straces of the brick and another on the same
>>>>>> server, does the high number of futex errors give any clues as to what
>>>>>> might be wrong?
>>>>>>
>>>>>> % time seconds usecs/call calls errors syscall
>>>>>> ------ ----------- ----------- --------- --------- ----------------
>>>>>> 45.58 0.027554 0 191665 20772 futex
>>>>>> 28.26 0.017084 0 137133 readv
>>>>>> 26.04 0.015743 0 66259 epoll_wait
>>>>>> 0.13 0.000077 3 23 writev
>>>>>> 0.00 0.000000 0 1 epoll_ctl
>>>>>> ------ ----------- ----------- --------- --------- ----------------
>>>>>> 100.00 0.060458 395081 20772 total
>>>>>>
>>>>>> % time seconds usecs/call calls errors syscall
>>>>>> ------ ----------- ----------- --------- --------- ----------------
>>>>>> 99.25 0.334020 133 2516 epoll_wait
>>>>>> 0.40 0.001347 0 4090 26 futex
>>>>>> 0.35 0.001192 0 5064 readv
>>>>>> 0.00 0.000000 0 20 writev
>>>>>> ------ ----------- ----------- --------- --------- ----------------
>>>>>> 100.00 0.336559 11690 26 total
>>>>>>
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>>
>>>>
>
More information about the Gluster-users
mailing list