[Gluster-devel] [Gluster-users] heal hanging

Thu Jan 21 15:56:26 UTC 2016

I should mention that the problem is not currently occurring and there are
no heals (output appended). By restarting the gluster services, we can stop
the crawl, which lowers the load for a while. Subsequent crawls seem to
finish properly. For what it's worth, files/folders that show up in the
'volume info' output during a hung crawl don't seem to be anything out of
the ordinary.

Over the past four days, the typical time before the problem recurs after
suppressing it in this manner is an hour. Last night when we reached out to
you was the last time it happened and the load has been low since (a
relief).  David believes that recursively listing the files (ls -alR or
similar) from a client mount can force the issue to happen, but obviously
I'd rather not unless we have some precise thing we're looking for. Let me
know if you'd like me to attempt to drive the system unstable like that and
what I should look for. As it's a production system, I'd rather not leave
it in this state for long.

[root at gfs01a xattrop]# gluster volume heal homegfs info
Brick gfs01a.corvidtec.com:/data/brick01a/homegfs/
Number of entries: 0

Brick gfs01b.corvidtec.com:/data/brick01b/homegfs/
Number of entries: 0

Brick gfs01a.corvidtec.com:/data/brick02a/homegfs/
Number of entries: 0

Brick gfs01b.corvidtec.com:/data/brick02b/homegfs/
Number of entries: 0

Brick gfs02a.corvidtec.com:/data/brick01a/homegfs/
Number of entries: 0

Brick gfs02b.corvidtec.com:/data/brick01b/homegfs/
Number of entries: 0

Brick gfs02a.corvidtec.com:/data/brick02a/homegfs/
Number of entries: 0

Brick gfs02b.corvidtec.com:/data/brick02b/homegfs/
Number of entries: 0

On Thu, Jan 21, 2016 at 10:40 AM, Pranith Kumar Karampuri <
pkarampu at redhat.com> wrote:

>
>
> On 01/21/2016 08:25 PM, Glomski, Patrick wrote:
>
> Hello, Pranith. The typical behavior is that the %cpu on a glusterfsd
> process jumps to number of processor cores available (800% or 1200%,
> depending on the pair of nodes involved) and the load average on the
> machine goes very high (~20). The volume's heal statistics output shows
> that it is crawling one of the bricks and trying to heal, but this crawl
> hangs and never seems to finish.
>
>
> The number of files in the xattrop directory varies over time, so I ran a
> wc -l as you requested periodically for some time and then started
> including a datestamped list of the files that were in the xattrops
> directory on each brick to see which were persistent. All bricks had files
> in the xattrop folder, so all results are attached.
>
> Thanks this info is helpful. I don't see a lot of files. Could you give
> output of "gluster volume heal <volname> info"? Is there any directory in
> there which is LARGE?
>
> Pranith
>
>
> Please let me know if there is anything else I can provide.
>
> Patrick
>
>
> On Thu, Jan 21, 2016 at 12:01 AM, Pranith Kumar Karampuri <
> pkarampu at redhat.com> wrote:
>
>> hey,
>>        Which process is consuming so much cpu? I went through the logs
>> you gave me. I see that the following files are in gfid mismatch state:
>>
>> <066e4525-8f8b-43aa-b7a1-86bbcecc68b9/safebrowsing-backup>,
>> <1d48754b-b38c-403d-94e2-0f5c41d5f885/recovery.bak>,
>> <ddc92637-303a-4059-9c56-ab23b1bb6ae9/patch0008.cnvrg>,
>>
>> Could you give me the output of "ls <brick-path>/indices/xattrop | wc -l"
>> output on all the bricks which are acting this way? This will tell us the
>> number of pending self-heals on the system.
>>
>> Pranith
>>
>>
>> On 01/20/2016 09:26 PM, David Robinson wrote:
>>
>> resending with parsed logs...
>>
>>
>>
>>
>>
>> I am having issues with 3.6.6 where the load will spike up to 800% for
>> one of the glusterfsd processes and the users can no longer access the
>> system.  If I reboot the node, the heal will finish normally after a few
>> minutes and the system will be responsive, but a few hours later the issue
>> will start again.  It look like it is hanging in a heal and spinning up the
>> load on one of the bricks.  The heal gets stuck and says it is crawling and
>> never returns.  After a few minutes of the heal saying it is crawling, the
>> load spikes up and the mounts become unresponsive.
>>
>> Any suggestions on how to fix this?  It has us stopped cold as the user
>> can no longer access the systems when the load spikes... Logs attached.
>>
>> System setup info is:
>>
>> [root at gfs01a ~]# gluster volume info homegfs
>>
>> Volume Name: homegfs
>> Type: Distributed-Replicate
>> Volume ID: 1e32672a-f1b7-4b58-ba94-58c085e59071
>> Status: Started
>> Number of Bricks: 4 x 2 = 8
>> Transport-type: tcp
>> Bricks:
>> Brick1: gfsib01a.corvidtec.com:/data/brick01a/homegfs
>> Brick2: gfsib01b.corvidtec.com:/data/brick01b/homegfs
>> Brick3: gfsib01a.corvidtec.com:/data/brick02a/homegfs
>> Brick4: gfsib01b.corvidtec.com:/data/brick02b/homegfs
>> Brick5: gfsib02a.corvidtec.com:/data/brick01a/homegfs
>> Brick6: gfsib02b.corvidtec.com:/data/brick01b/homegfs
>> Brick7: gfsib02a.corvidtec.com:/data/brick02a/homegfs
>> Brick8: gfsib02b.corvidtec.com:/data/brick02b/homegfs
>> Options Reconfigured:
>> performance.io-thread-count: 32
>> performance.cache-size: 128MB
>> performance.write-behind-window-size: 128MB
>> server.allow-insecure: on
>> network.ping-timeout: 42
>> storage.owner-gid: 100
>> geo-replication.indexing: off
>> geo-replication.ignore-pid-check: on
>> changelog.changelog: off
>> changelog.fsync-interval: 3
>> changelog.rollover-time: 15
>> server.manage-gids: on
>> diagnostics.client-log-level: WARNING
>>
>> [root at gfs01a ~]# rpm -qa | grep gluster
>> gluster-nagios-common-0.1.1-0.el6.noarch
>> glusterfs-fuse-3.6.6-1.el6.x86_64
>> glusterfs-debuginfo-3.6.6-1.el6.x86_64
>> glusterfs-libs-3.6.6-1.el6.x86_64
>> glusterfs-geo-replication-3.6.6-1.el6.x86_64
>> glusterfs-api-3.6.6-1.el6.x86_64
>> glusterfs-devel-3.6.6-1.el6.x86_64
>> glusterfs-api-devel-3.6.6-1.el6.x86_64
>> glusterfs-3.6.6-1.el6.x86_64
>> glusterfs-cli-3.6.6-1.el6.x86_64
>> glusterfs-rdma-3.6.6-1.el6.x86_64
>> samba-vfs-glusterfs-4.1.11-2.el6.x86_64
>> glusterfs-server-3.6.6-1.el6.x86_64
>> glusterfs-extra-xlators-3.6.6-1.el6.x86_64
>>
>>
>>
>>
>>
>> _______________________________________________
>> Gluster-devel mailing listGluster-devel at gluster.orghttp://www.gluster.org/mailman/listinfo/gluster-devel
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160121/ac7484b2/attachment-0001.html>