[Gluster-users] One file incessant self heal
Homer Li
01jay.ly at gmail.com
Thu Jul 12 08:43:44 UTC 2012
Hi Pranith;
Thanks for your reply.
I checked md5sum too, it is different.
Here is my output:
# getfattr -d -m . -e hex
/export/data10/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
getfattr: Removing leading '/' from absolute path names
# file: export/data10/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
trusted.afr.gvol1-client-2=0x000003630000000000000000
trusted.afr.gvol1-client-3=0x000000010000000000000000
trusted.gfid=0xd9b0c35033ba4090ab08f91f30dd661f
trusted.glusterfs.quota.4111d3b4-7e06-483f-aae8-fbefe9e55843.contri=0x00000001c0d15000
# getfattr -d -m . -e hex
/export/data11/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
getfattr: Removing leading '/' from absolute path names
# file: export/data11/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
trusted.afr.gvol1-client-2=0x000000010000000000000000
trusted.afr.gvol1-client-3=0x000000010000000000000000
trusted.gfid=0xd9b0c35033ba4090ab08f91f30dd661f
trusted.glusterfs.quota.4111d3b4-7e06-483f-aae8-fbefe9e55843.contri=0x00000001c0d15000
#stat /export/data10/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
File: `/export/data10/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f'
Size: 7530086400 Blocks: 14706856 IO Block: 4096 regular file
Device: 6900h/26880d Inode: 81395728 Links: 2
Access: (0644/-rw-r--r--) Uid: ( 107/ UNKNOWN) Gid: ( 107/ UNKNOWN)
Access: 2012-06-21 09:58:59.242136421 +0800
Modify: 2012-07-10 13:42:04.381141510 +0800
Change: 2012-07-12 16:23:15.884163991 +0800
#stat /export/data11/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
File: `/export/data11/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f'
Size: 7530086400 Blocks: 14706856 IO Block: 4096 regular file
Device: 6910h/26896d Inode: 17956874 Links: 2
Access: (0644/-rw-r--r--) Uid: ( 107/ UNKNOWN) Gid: ( 107/ UNKNOWN)
Access: 2012-06-21 09:58:59.242136421 +0800
Modify: 2012-07-10 13:42:04.381141510 +0800
Change: 2012-07-12 16:23:15.885163872 +0800
#md5sum /export/data10/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
7fba61af476bf379c50f7429c89449ee
/export/data10/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
#md5sum /export/data11/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
9b08b7145c171afff863c4ae5884fa01
/export/data11/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
2012/7/12 Pranith Kumar Karampuri <pkarampu at redhat.com>:
> Homer,
> Could you give the output of
> getfattr -d -m . -e hex /export/data10/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
> getfattr -d -m . -e hex /export/data11/.glusterfs/d9/b0/d9b0c350-33ba-4090-ab08-f91f30dd661f
> and also 'stat' of these files.
>
> Pranith.
> ----- Original Message -----
> From: "Homer Li" <01jay.ly at gmail.com>
> To: "gluster-users" <gluster-users at gluster.org>
> Sent: Thursday, July 12, 2012 7:54:24 AM
> Subject: [Gluster-users] One file incessant self heal
>
> Hello ;
> I found many self-heal triggered log in every 10 minutes.
> Only one file , it 's gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f.
> Heal-failed and split-brain have not display anything.
> Does any problem in this file ?
>
>
> GlusterFS config:
>
> OS: 2.6.32-220.17.1.el6.x86_64 Scientific Linux release 6.2 (Carbon)
> # rpm -qa | grep glusterfs
> glusterfs-3.3.0-2.el6.x86_64
> glusterfs-devel-3.3.0-2.el6.x86_64
> glusterfs-fuse-3.3.0-2.el6.x86_64
> glusterfs-geo-replication-3.3.0-2.el6.x86_64
> glusterfs-rdma-3.3.0-2.el6.x86_64
> glusterfs-server-3.3.0-2.el6.x86_64
> glusterfs-debuginfo-3.3.0-2.el6.x86_64
>
> gluster> volume info
>
> Volume Name: gvol1
> Type: Distributed-Replicate
> Volume ID: a7d8ffdf-7296-404b-aeab-824ee853ec59
> Status: Started
> Number of Bricks: 2 x 2 = 4
> Transport-type: tcp
> Bricks:
> Brick1: 172.30.1.125:/export/data00
> Brick2: 172.30.1.125:/export/data01
> Brick3: 172.30.1.125:/export/data10
> Brick4: 172.30.1.125:/export/data11
> Options Reconfigured:
> features.limit-usage: /source:500GB
> features.quota: on
> performance.cache-refresh-timeout: 30
> performance.io-thread-count: 32
> nfs.disable: off
> cluster.min-free-disk: 5%
> performance.cache-size: 128MB
>
>
> gluster volume heal gvol1 info
> Heal operation on volume gvol1 has been successful
>
> Brick 172.30.1.125:/export/data00
> Number of entries: 0
>
> Brick 172.30.1.125:/export/data01
> Number of entries: 0
>
> Brick 172.30.1.125:/export/data10
> Number of entries: 1
> /fs126/Graphite-monitor_vdb.qcow2
>
> Brick 172.30.1.125:/export/data11
> Number of entries: 1
> /fs126/Graphite-monitor_vdb.qcow2
>
> # gluster volume heal gvol1 info heal-failed
> Heal operation on volume gvol1 has been successful
>
> Brick 172.30.1.125:/export/data00
> Number of entries: 0
>
> Brick 172.30.1.125:/export/data01
> Number of entries: 0
>
> Brick 172.30.1.125:/export/data10
> Number of entries: 0
>
> Brick 172.30.1.125:/export/data11
> Number of entries: 0
>
> gluster volume heal gvol1 info split-brain
> Heal operation on volume gvol1 has been successful
>
> Brick 172.30.1.125:/export/data00
> Number of entries: 0
>
> Brick 172.30.1.125:/export/data01
> Number of entries: 0
>
> Brick 172.30.1.125:/export/data10
> Number of entries: 0
>
> Brick 172.30.1.125:/export/data11
> Number of entries: 0
>
>
>
> Log detail:
> [2012-07-12 09:13:11.666417] I
> [afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0:
> Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on
> gvol1-client-0
> [2012-07-12 09:13:11.666998] W
> [client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote
> operation failed: No such file or directory
> [2012-07-12 09:13:11.667132] I
> [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
> background data self-heal triggered. path:
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
> pending operations
> [2012-07-12 09:13:11.667141] E
> [afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0:
> e6087bf7-ae55-441b-8f88-a7b17475caea: Failed to remove index on
> gvol1-client-0 - No such file or directory
> [2012-07-12 09:13:11.667589] I
> [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
> background data self-heal triggered. path:
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
> pending operations
> [2012-07-12 09:13:11.668581] I
> [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
> active sinks for performing self-heal on file
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:13:11.669322] I
> [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
> 0-gvol1-replicate-1: background data self-heal completed on
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:13:11.669775] I
> [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
> active sinks for performing self-heal on file
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:13:11.670408] I
> [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
> 0-gvol1-replicate-1: background data self-heal completed on
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:23:11.770994] I
> [afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0:
> Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on
> gvol1-client-0
> [2012-07-12 09:23:11.771358] W
> [client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote
> operation failed: No such file or directory
> [2012-07-12 09:23:11.771416] E
> [afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0:
> e6087bf7-ae55-441b-8f88-a7b17475caea: Failed to remove index on
> gvol1-client-0 - No such file or directory
> [2012-07-12 09:23:11.771898] I
> [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
> background data self-heal triggered. path:
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
> pending operations
> [2012-07-12 09:23:11.772059] I
> [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
> background data self-heal triggered. path:
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
> pending operations
> [2012-07-12 09:23:11.773074] I
> [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
> active sinks for performing self-heal on file
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:23:11.773686] I
> [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
> 0-gvol1-replicate-1: background data self-heal completed on
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:23:11.774094] I
> [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
> active sinks for performing self-heal on file
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:23:11.774652] I
> [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
> 0-gvol1-replicate-1: background data self-heal completed on
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:33:11.874474] I
> [afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0:
> Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on
> gvol1-client-0
> [2012-07-12 09:33:11.874919] W
> [client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote
> operation failed: No such file or directory
> [2012-07-12 09:33:11.874978] E
> [afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0:
> d9b0c350-33ba-4090-ab08-f91f30dd661f: Failed to remove index on
> gvol1-client-0 - No such file or directory
> [2012-07-12 09:33:11.875505] I
> [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
> background data self-heal triggered. path:
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
> pending operations
> [2012-07-12 09:33:11.875676] I
> [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
> background data self-heal triggered. path:
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
> pending operations
> [2012-07-12 09:33:11.876613] I
> [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
> active sinks for performing self-heal on file
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:33:11.877244] I
> [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
> 0-gvol1-replicate-1: background data self-heal completed on
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:33:11.877646] I
> [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
> active sinks for performing self-heal on file
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:33:11.878191] I
> [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
> 0-gvol1-replicate-1: background data self-heal completed on
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:43:11.971727] I
> [afr-self-heald.c:282:_remove_stale_index] 0-gvol1-replicate-0:
> Removing stale index for e6087bf7-ae55-441b-8f88-a7b17475caea on
> gvol1-client-0
> [2012-07-12 09:43:11.972004] W
> [client3_1-fops.c:592:client3_1_unlink_cbk] 0-gvol1-client-0: remote
> operation failed: No such file or directory
> [2012-07-12 09:43:11.972066] E
> [afr-self-heald.c:287:_remove_stale_index] 0-gvol1-replicate-0:
> e6087bf7-ae55-441b-8f88-a7b17475caea: Failed to remove index on
> gvol1-client-0 - No such file or directory
> [2012-07-12 09:43:11.972635] I
> [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
> background data self-heal triggered. path:
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
> pending operations
> [2012-07-12 09:43:11.972924] I
> [afr-common.c:1340:afr_launch_self_heal] 0-gvol1-replicate-1:
> background data self-heal triggered. path:
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>, reason: lookup detected
> pending operations
> [2012-07-12 09:43:11.973885] I
> [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
> active sinks for performing self-heal on file
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:43:11.974451] I
> [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
> 0-gvol1-replicate-1: background data self-heal completed on
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:43:11.974965] I
> [afr-self-heal-data.c:712:afr_sh_data_fix] 0-gvol1-replicate-1: no
> active sinks for performing self-heal on file
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
> [2012-07-12 09:43:11.975772] I
> [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk]
> 0-gvol1-replicate-1: background data self-heal completed on
> <gfid:d9b0c350-33ba-4090-ab08-f91f30dd661f>
>
> --
> Best Regards
> Homer Li
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
--
Best Regards
Homer Li
More information about the Gluster-users
mailing list