[Gluster-users] Self Heal and dangling symlinks

[Alexandre Fournier]

Hello,

We are experiencing strange behavior when writing file on the Gluster mount point.  On some occasion, when writing to the Gluster Mount we have an Open Stream  error.  We've looked the gluster logs and  found the following faulty entries :

[From /var/log/glusterfs/mnt-gv0.log]

[2013-11-18 18:18:43.052446] W [afr-common.c:1411:afr_conflicting_iattrs] 0-gv0-replicate-0: /aa/aa/aa/aa: filetype differs on subvolumes (0, 1)
[2013-11-18 18:18:43.052468] E [afr-self-heal-common.c:1409:afr_sh_common_lookup_cbk] 0-gv0-replicate-0: Conflicting entries for /aa/aa/aa/aa
[2013-11-18 18:18:43.052757] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-gv0-replicate-0: background  meta-data data entry missing-entry gfid self-heal
failed on /aa/aa/aa/aa/aa
[2013-11-18 18:18:43.052780] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 439382194: LOOKUP() /aa/aa/aa/aa/aa => -1 (Input/output error)

We've looked at the log file etc-glusterfs-glusterd.vol.log but we found nothing related to this problem.  Then, we've looked at the log From /var/log/glusterfs/bricks/mnt-data.log and we found 70 gig of logs of the same type :

[2013-11-19 17:13:32.269757] W [posix-handle.c:538:posix_handle_soft] 0-gv0-posix: symlink ../../ab/fe/abfeb61c-501d-4417-b8fb-0accdd57146f/cf -> /mnt/data/.glusterfs/ab/fe/abfeb61c-501d-4417-b8fb-0accdd57146f/cf failed (No such file or directory)
[2013-11-19 17:13:32.269978] W [posix-handle.c:538:posix_handle_soft] 0-gv0-posix: symlink ../../c7/8b/c78be78f-cc95-47b2-a27f-4217f1759b67/d2 -> /mnt/data/.glusterfs/c7/8b/c78be78f-cc95-47b2-a27f-4217f1759b67/d2 failed (No such file or directory)
[2013-11-19 17:13:32.270190] W [posix-handle.c:538:posix_handle_soft] 0-gv0-posix: symlink ../../5a/8f/5a8fa43c-4ccc-4d88-9122-a96bc8ffaebc/f2 -> /mnt/data/.glusterfs/5a/8f/5a8fa43c-4ccc-4d88-9122-a96bc8ffaebc/f2 failed (No such file or directory)

And it does not stop logging.  It seems that the self heal is not working properly when there are broking symlinks in the gluster.  It is worth saying also that this log is only produce on a single node but the write fail on several node though.  Also, we try to clean the symlinks manually but it always come back.

Is it possible to recover from broken symlinks?

Configuration :
Gluster Version : 3.3.2
Cluster setup : 4 X 2
OS : Ubuntu
On Fuse

Thanks,

Alexandre

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131119/df7c194d/attachment.html>