[Gluster-users] Self Heal and dangling symlinks

[Lalatendu Mohanty]

On 11/19/2013 10:49 PM, Alexandre Fournier wrote:
>
> Hello,
>
> We are experiencing strange behavior when writing file on the Gluster 
> mount point.  On some occasion, when writing to the Gluster Mount we 
> have an Open Stream  error.  We've looked the gluster logs and  found 
> the following faulty entries :
>
> [From /var/log/glusterfs/mnt-gv0.log]
>
> [2013-11-18 18:18:43.052446] W 
> [afr-common.c:1411:afr_conflicting_iattrs] 0-gv0-replicate-0: 
> /aa/aa/aa/aa: filetype differs on subvolumes (0, 1)
>
> [2013-11-18 18:18:43.052468] E 
> [afr-self-heal-common.c:1409:afr_sh_common_lookup_cbk] 
> 0-gv0-replicate-0: Conflicting entries for /aa/aa/aa/aa
>
> [2013-11-18 18:18:43.052757] E 
> [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 
> 0-gv0-replicate-0: background  meta-data data entry missing-entry gfid 
> self-heal
>
> failed on /aa/aa/aa/aa/aa
>
> [2013-11-18 18:18:43.052780] W [fuse-bridge.c:292:fuse_entry_cbk] 
> 0-glusterfs-fuse: 439382194: LOOKUP() /aa/aa/aa/aa/aa => -1 
> (Input/output error)
>
> We've looked at the log file etc-glusterfs-glusterd.vol.log but we 
> found nothing related to this problem.  Then, we've looked at the log 
> From /var/log/glusterfs/bricks/mnt-data.log and we found 70 gig of 
> logs of the same type :
>
> [2013-11-19 17:13:32.269757] W [posix-handle.c:538:posix_handle_soft] 
> 0-gv0-posix: symlink 
> ../../ab/fe/abfeb61c-501d-4417-b8fb-0accdd57146f/cf -> 
> /mnt/data/.glusterfs/ab/fe/abfeb61c-501d-4417-b8fb-0accdd57146f/cf 
> failed (No such file or directory)
>
> [2013-11-19 17:13:32.269978] W [posix-handle.c:538:posix_handle_soft] 
> 0-gv0-posix: symlink 
> ../../c7/8b/c78be78f-cc95-47b2-a27f-4217f1759b67/d2 -> 
> /mnt/data/.glusterfs/c7/8b/c78be78f-cc95-47b2-a27f-4217f1759b67/d2 
> failed (No such file or directory)
>
> [2013-11-19 17:13:32.270190] W [posix-handle.c:538:posix_handle_soft] 
> 0-gv0-posix: symlink 
> ../../5a/8f/5a8fa43c-4ccc-4d88-9122-a96bc8ffaebc/f2 -> 
> /mnt/data/.glusterfs/5a/8f/5a8fa43c-4ccc-4d88-9122-a96bc8ffaebc/f2 
> failed (No such file or directory)
>

This looks like a bug, unless there is something wrong with the set-up. 
I have copied gluster-devel in this thread as I think they might help.

Just curious, is all your gluster nodes have equal time (i.e. ntp synced).

> And it does not stop logging.  It seems that the self heal is not 
> working properly when there are broking symlinks in the gluster.  It 
> is worth saying also that this log is only produce on a single node 
> but the write fail on several node though. Also, we try to clean the 
> symlinks manually but it always come back.
>
> Is it possible to recover from broken symlinks?
>
> Configuration :
>
> Gluster Version : 3.3.2
>
> Cluster setup : 4 X 2
>
> OS : Ubuntu
>
> On Fuse
>
> Thanks,
>
> Alexandre
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131121/9ff8ded4/attachment.html>