[Gluster-users] [Gluster-devel] Self Heal and dangling symlinks

[Pranith Kumar Karampuri]

Alexandre,
   Seems like there is an entry split-brain (same file/dir name but on one brick it is a file and on the other it is a directory) according to the following log:
> [2013-11-18 18:18:43.052446] W [afr-common.c:1411:afr_conflicting_iattrs]
> 0-gv0-replicate-0: /aa/aa/aa/aa: filetype differs on subvolumes (0, 1) 

Could you get us the output of "stat <brick-dir-path>/aa/aa/aa/aa/aa" and "getfattr -d -m. -e hex <brick-dir-path>/aa/aa/aa/aa/aa" on both the bricks.

Pranith
----- Original Message -----
> From: "Lalatendu Mohanty" <lmohanty at redhat.com>
> To: "Alexandre Fournier" <alexandre.fournier at ubisoft.com>, gluster-users at gluster.org, gluster-devel at nongnu.org
> Sent: Thursday, November 21, 2013 1:28:01 AM
> Subject: Re: [Gluster-devel] [Gluster-users] Self Heal and dangling symlinks
> 
> On 11/19/2013 10:49 PM, Alexandre Fournier wrote:
> 
> 
> 
> 
> 
> Hello,
> 
> 
> 
> We are experiencing strange behavior when writing file on the Gluster mount
> point. On some occasion, when writing to the Gluster Mount we have an Open
> Stream error. We’ve looked the gluster logs and found the following faulty
> entries :
> 
> 
> 
> [From /var/log/glusterfs/mnt-gv0.log]
> 
> 
> 
> [2013-11-18 18:18:43.052446] W [afr-common.c:1411:afr_conflicting_iattrs]
> 0-gv0-replicate-0: /aa/aa/aa/aa: filetype differs on subvolumes (0, 1)
> 
> [2013-11-18 18:18:43.052468] E
> [afr-self-heal-common.c:1409:afr_sh_common_lookup_cbk] 0-gv0-replicate-0:
> Conflicting entries for /aa/aa/aa/aa
> 
> [2013-11-18 18:18:43.052757] E
> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk]
> 0-gv0-replicate-0: background meta-data data entry missing-entry gfid
> self-heal
> 
> failed on /aa/aa/aa/aa/aa
> 
> [2013-11-18 18:18:43.052780] W [fuse-bridge.c:292:fuse_entry_cbk]
> 0-glusterfs-fuse: 439382194: LOOKUP() /aa/aa/aa/aa/aa => -1 (Input/output
> error)
> 
> 
> 
> We’ve looked at the log file etc-glusterfs-glusterd.vol.log but we found
> nothing related to this problem. Then, we’ve looked at the log From
> /var/log/glusterfs/bricks/mnt-data.log and we found 70 gig of logs of the
> same type :
> 
> 
> 
> [2013-11-19 17:13:32.269757] W [posix-handle.c:538:posix_handle_soft]
> 0-gv0-posix: symlink ../../ab/fe/abfeb61c-501d-4417-b8fb-0accdd57146f/cf ->
> /mnt/data/.glusterfs/ab/fe/abfeb61c-501d-4417-b8fb-0accdd57146f/cf failed
> (No such file or directory)
> 
> [2013-11-19 17:13:32.269978] W [posix-handle.c:538:posix_handle_soft]
> 0-gv0-posix: symlink ../../c7/8b/c78be78f-cc95-47b2-a27f-4217f1759b67/d2 ->
> /mnt/data/.glusterfs/c7/8b/c78be78f-cc95-47b2-a27f-4217f1759b67/d2 failed
> (No such file or directory)
> 
> [2013-11-19 17:13:32.270190] W [posix-handle.c:538:posix_handle_soft]
> 0-gv0-posix: symlink ../../5a/8f/5a8fa43c-4ccc-4d88-9122-a96bc8ffaebc/f2 ->
> /mnt/data/.glusterfs/5a/8f/5a8fa43c-4ccc-4d88-9122-a96bc8ffaebc/f2 failed
> (No such file or directory)
> 
> 
> 
> This looks like a bug, unless there is something wrong with the set-up. I
> have copied gluster-devel in this thread as I think they might help.
> 
> Just curious, is all your gluster nodes have equal time (i.e. ntp synced).
> 
> 
> 
> 
> 
> 
> And it does not stop logging. It seems that the self heal is not working
> properly when there are broking symlinks in the gluster. It is worth saying
> also that this log is only produce on a single node but the write fail on
> several node though. Also, we try to clean the symlinks manually but it
> always come back.
> 
> 
> 
> Is it possible to recover from broken symlinks?
> 
> 
> 
> Configuration :
> 
> Gluster Version : 3.3.2
> 
> Cluster setup : 4 X 2
> 
> OS : Ubuntu
> 
> On Fuse
> 
> 
> 
> Thanks,
> 
> 
> 
> Alexandre
> 
> 
> 
> 
> _______________________________________________
> Gluster-users mailing list Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
> 
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> https://lists.nongnu.org/mailman/listinfo/gluster-devel
>