[Gluster-devel] split brain with all-zero pending matrix

[Pranith Kumar Karampuri]

Emmanuel,
   What do the extended attributes look like in the backend for this file on the two bricks.

Pranith.

----- Original Message -----
> From: "Emmanuel Dreyfus" <manu at netbsd.org>
> To: "Gluster Devel" <gluster-devel at nongnu.org>
> Sent: Friday, June 21, 2013 6:23:31 AM
> Subject: [Gluster-devel] split brain with all-zero pending matrix
> 
> On 3.4.0beta3, after using the volume for a while, I get split-brain errors
> with an unhelpful pending matrix.
> 
> [2013-06-20 08:44:00.665731] E
> [afr-self-heal-common.c:197:afr_sh_print_split_brain_log]
> 0-gfs34-replicate-1: Unable to self-heal contents of
> '/manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o' (possible
> split-brain). Please delete the file from all but the preferred subvolume.-
> Pending matrix:  [ [ 0 0 ] [ 0 0 ] ]
> [2013-06-20 08:44:00.666431] E
> [afr-self-heal-common.c:2212:afr_self_heal_completion_cbk]
> 0-gfs34-replicate-1: background  data self-heal failed on
> /manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o
> [2013-06-20 08:44:00.667201] W [afr-open.c:213:afr_open] 0-gfs34-replicate-1:
> failed to open as split brain seen, returning EIO
> [2013-06-20 08:44:00.668193] W [fuse-bridge.c:875:fuse_fd_cbk]
> 0-glusterfs-fuse: 8711927: OPEN()
> /manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o => -1 (Input/output
> error)
> 
> On the bricks (ls -l, md5 and first 32 bytes):
> brick0
> -rw-r--r--  2 manu  manu  7216 Jun 20 09:56
> /export/wd3a/manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o
> MD5 (/export/wd3a/manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o) =
> 24ee57aa8e2aeb6102ba170fb81bbf22
> 00000000  7f 45 4c 46 01 01 01 00  00 00 00 00 00 00 00 00
> |.ELF............|
> 00000010  01 00 03 00 01 00 00 00  00 00 00 00 00 00 00 00
> |................|
> brick1
> -rw-r--r--  2 manu  manu  7216 Jun 20 09:56
> /export/wd1a/manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o
> MD5 (/export/wd1a/manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o) =
> 24ee57aa8e2aeb6102ba170fb81bbf22
> 00000000  7f 45 4c 46 01 01 01 00  00 00 00 00 00 00 00 00
> |.ELF............|
> 00000010  01 00 03 00 01 00 00 00  00 00 00 00 00 00 00 00
> |................|
> brick2
> -rw-r--r--  2 manu  manu  6256 Jun 20 09:56
> /export/wd3a/manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o
> MD5 (/export/wd3a/manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o) =
> 58a4b8c5929cac2f799c60b3dd2acc2f
> brick3
> 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
> |................|
> 00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
> |................|
> -rw-r--r--  2 manu  manu  7216 Jun 20 09:56
> /export/wd1a/manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o
> MD5 (/export/wd1a/manu/netbsd/usr/src/tools/gcc/obj/build/gcc/cfgbuild.o) =
> 2bc383747432c49793a2cf1a2c0e8cfa
> 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
> |................|
> 00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
> |................|
> 
> That shows the offending copies without any doubt, but I still wonder how
> we come to that situation. There are a lot of such files, making the
> glusterfs
> volume just unusable at this point.
> 
> --
> Emmanuel Dreyfus
> http://hcpnet.free.fr/pubz
> manu at netbsd.org
> 
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> https://lists.nongnu.org/mailman/listinfo/gluster-devel
>