[Gluster-users] Possible split-brain

[Aaron Roberts]

Hi Jeff,
	Thanks for your quick response, I have included results of the command below.

> -----Original Message-----
> From: gluster-users-bounces at gluster.org [mailto:gluster-users-
> bounces at gluster.org] On Behalf Of Jeff Darcy
 
> > The platform is not currently running production data and I have been
> testing the redundancy of the setup (pulling cables etc.).  All my
> servers are now logging the following messages every 1 minute or so:
> >
> > [2010-11-11 14:18:49.636327] I [afr-common.c:672:afr_lookup_done]
> datastore-replicate-0: split brain detected during lookup of /.
> > [2010-11-11 14:18:49.636388] I [afr-common.c:716:afr_lookup_done]
> datastore-replicate-0: background  meta-data data self-heal triggered.
> path: /
> > [2010-11-11 14:18:49.636863] E [afr-self-heal-
> metadata.c:524:afr_sh_metadata_fix] datastore-replicate-0: Unable to
> self-heal permissions/ownership of '/' (possible split-brain). Please
> fix the file on all backend volumes
> 
> Can you do a "getfattr -d -e hex -m trusted.afr $path" on the path for
> each brick's root directory (server side)?  There seem to be a few
> different ways for the split-brain flag to be set, all having to do
> with
> the contents of these xattrs.  The solution might be to clear them, but
> it would be good to see what the values are and have someone closer to
> the AFR code than I am determine exactly which case we're in.

[root at agent001 ~]# getfattr -d -e hex -m trusted.afr /glusterfs/primary
getfattr: Removing leading '/' from absolute path names
# file: glusterfs/primary
trusted.afr.datastore-client-0=0x000000000000000000000000
trusted.afr.datastore-client-1=0x000000000000000200000000

[root at agent001 ~]# getfattr -d -e hex -m trusted.afr /glusterfs/secondary
getfattr: Removing leading '/' from absolute path names
# file: glusterfs/secondary
trusted.afr.datastore-client-4=0x000000000000000000000000
trusted.afr.datastore-client-5=0x000000000000000000000000
trusted.afr.datastore-client-6=0x000000000000000000000000
trusted.afr.datastore-client-7=0x000000000000000000000000
trusted.afr.datastore-client-8=0x000000000000000000000000
trusted.afr.datastore-client-9=0x000000000000000000000000

[root at agent002 ~]# getfattr -d -e hex -m trusted.afr /glusterfs/primary
getfattr: Removing leading '/' from absolute path names
# file: glusterfs/primary
trusted.afr.datastore-client-10=0x000000000000000000000000
trusted.afr.datastore-client-11=0x000000000000000000000000
trusted.afr.datastore-client-2=0x000000000000000000000000
trusted.afr.datastore-client-3=0x000000000000000000000000
trusted.afr.datastore-client-8=0x000000000000000000000000
trusted.afr.datastore-client-9=0x000000000000000200000000

[root at agent002 ~]# getfattr -d -e hex -m trusted.afr /glusterfs/secondary
getfattr: Removing leading '/' from absolute path names
# file: glusterfs/secondary
trusted.afr.datastore-client-10=0x000000000000000300000000
trusted.afr.datastore-client-11=0x000000000000000000000000
trusted.afr.datastore-client-6=0x000000000000000000000000
trusted.afr.datastore-client-7=0x000000000000000000000000
trusted.afr.datastore-client-8=0x000000000000000000000000
trusted.afr.datastore-client-9=0x000000000000000000000000

[root at agent003 ~]# getfattr -d -e hex -m trusted.afr /glusterfs/primary
getfattr: Removing leading '/' from absolute path names
# file: glusterfs/primary
trusted.afr.datastore-client-2=0x000000000000000000000000
trusted.afr.datastore-client-3=0x000000000000000200000001
trusted.afr.datastore-client-4=0x000000000000000000000000
trusted.afr.datastore-client-5=0x000000000000000000000000

[root at agent003 ~]# getfattr -d -e hex -m trusted.afr /glusterfs/secondary
getfattr: Removing leading '/' from absolute path names
# file: glusterfs/secondary
trusted.afr.datastore-client-0=0x000000000000000100000000
trusted.afr.datastore-client-1=0x000000000000000000000000

[root at agent004 ~]# getfattr -d -e hex -m trusted.afr /glusterfs/primary
getfattr: Removing leading '/' from absolute path names
# file: glusterfs/primary
trusted.afr.datastore-client-10=0x000000000000000300000000
trusted.afr.datastore-client-11=0x000000000000000000000000
trusted.afr.datastore-client-4=0x000000000000000000000000
trusted.afr.datastore-client-5=0x000000000000000500000000
trusted.afr.datastore-client-6=0x000000000000000000000000
trusted.afr.datastore-client-7=0x000000000000000000000000
trusted.afr.datastore-client-8=0x000000000000000000000000
trusted.afr.datastore-client-9=0x000000000000000000000000

[root at agent004 ~]# getfattr -d -e hex -m trusted.afr /glusterfs/secondary
getfattr: Removing leading '/' from absolute path names
# file: glusterfs/secondary
trusted.afr.datastore-client-10=0x000000000000000000000000
trusted.afr.datastore-client-11=0x000000000000000000000000
trusted.afr.datastore-client-2=0x000000000000000000000000
trusted.afr.datastore-client-3=0x000000000000000000000000
trusted.afr.datastore-client-8=0x000000000000000000000000
trusted.afr.datastore-client-9=0x000000000000000200000000