[Gluster-devel] Self heal failure

[Emmanuel Dreyfus]

Client:
$ cat /pfs/manu/netbsd/usr/src/tools/mktemp/Makefile
cat: /pfs/manu/netbsd/usr/src/tools/mktemp/Makefile: Input/output error

Client logs:
[2012-06-15 03:32:06.234208] W [afr-common.c:1226:afr_detect_self_heal_by_lookup_status] 0-pfs-replicate-0: split brain detected during lookup of /manu/netbsd/usr/src/tools/mktemp/Makefile.
[2012-06-15 03:32:06.234610] I [afr-common.c:1340:afr_launch_self_heal] 0-pfs-replicate-0: background  meta-data data gfid self-heal triggered. path: /manu/netbsd/usr/src/tools/mktemp/Makefile, reason: lookup detected pending operations
[2012-06-15 03:32:06.236294] I [afr-self-heal-common.c:1189:afr_sh_missing_entry_call_impunge_recreate] 0-pfs-replicate-0: no missing files - /manu/netbsd/usr/src/tools/mktemp/Makefile. proceeding to metadata check
[2012-06-15 03:32:06.237047] I [afr-self-heal-common.c:994:afr_sh_missing_entries_done] 0-pfs-replicate-0: split brain found, aborting selfheal of /manu/netbsd/usr/src/tools/mktemp/Makefile
[2012-06-15 03:32:06.237322] E [afr-self-heal-common.c:2156:afr_self_heal_completion_cbk] 0-pfs-replicate-0: background  meta-data data gfid self-heal failed on /manu/netbsd/usr/src/tools/mktemp/Makefile
[2012-06-15 03:32:06.238175] W [afr-open.c:213:afr_open] 0-pfs-replicate-0: failed to open as split brain seen, returning EIO
[2012-06-15 03:32:06.254921] W [fuse-bridge.c:713:fuse_fd_cbk] 0-glusterfs-fuse: 2434279: OPEN() /manu/netbsd/usr/src/tools/mktemp/Makefile => -1 (Input/output error)

Brick1:
-rw-r--r--  2 manu  manu  165 Dec  8  2002 /export/wd3a/manu/netbsd/usr/src/tools/mktemp/Makefile
trusted.gfid    6b db b7 73 cc e7 46 a8 9d fc 96 40 2c 6a fe e8
trusted.afr.pfs-client-0        00 00 00 00 00 00 00 03 00 00 00 00 00 

Brick2 (note 0 instead of 3 in trusted.afr.pfs-client-0)
-rw-r--r--  2 manu  manu  165 Dec  8  2002 /export/wd3a/manu/netbsd/usr/src/tools/mktemp/Makefile
trusted.gfid    6b db b7 73 cc e7 46 a8 9d fc 96 40 2c 6a fe e8
trusted.afr.pfs-client-0        00 00 00 00 00 00 00 00 00 00 00 00 00 

I delete the file on brick2 so that self heal can occur. The removed file is indeed created but I still get EIO. Here is client log on self heal:

[2012-06-15 03:37:07.717081] I [afr-common.c:1215:afr_detect_self_heal_by_lookup_status] 0-pfs-replicate-0: entries are missing in lookup of /manu/netbsd/usr/src/tools/mktemp/Makefile.
[2012-06-15 03:37:07.717459] I [afr-common.c:1340:afr_launch_self_heal] 0-pfs-replicate-0: background  meta-data data entry missing-entry gfid self-heal triggered. path: /manu/netbsd/usr/src/tools/mktemp/Makefile, reason: lookup detected pending operations
[2012-06-15 03:37:07.719497] E [afr-self-heal-common.c:1087:afr_sh_common_lookup_resp_handler] 0-pfs-replicate-0: path /manu/netbsd/usr/src/tools/mktemp/Makefile on subvolume pfs-client-0 => -1 (No such file or directory)
[2012-06-15 03:37:17.571513] I [afr-self-heal-common.c:994:afr_sh_missing_entries_done] 0-pfs-replicate-0: split brain found, aborting selfheal of /manu/netbsd/usr/src/tools/mktemp/Makefile
[2012-06-15 03:37:17.571926] E [afr-self-heal-common.c:2156:afr_self_heal_completion_cbk] 0-pfs-replicate-0: background  meta-data data entry missing-entry gfid self-heal failed on /manu/netbsd/usr/src/tools/mktemp/Makefile
[2012-06-15 03:37:17.572911] W [afr-open.c:213:afr_open] 0-pfs-replicate-0: failed to open as split brain seen, returning EIO
[2012-06-15 03:37:17.573365] W [fuse-bridge.c:713:fuse_fd_cbk] 0-glusterfs-fuse: 2434373: OPEN() /manu/netbsd/usr/src/tools/mktemp/Makefile => -1 (Input/output error)

Nothing in server log at that time.
-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu at netbsd.org