[Gluster-users] Unable to self-heal permissions/ownership of '/' (possible split-brain)

Fri Mar 4 01:35:23 UTC 2011

Hi everyone,

I have a problem about GlusterFS-3.1.2.

The following message occurred:
"Unable to self-heal permissions/ownership of '/' (possible split-brain)"

The volume seems to be working normally, but what kind of issue does this error cause?
Please let us know how to fix it and the reason about this error.

I found the same issue from the following, but I could not find the resolution.
<http://www.mail-archive.com/gluster-users@gluster.org/msg04442.html>
<http://gluster.org/pipermail/gluster-users/2010-November/005780.html>

I am seeing the following messages repeated in my volume log file:
[2011-03-01 18:12:54.862809] I [afr-common.c:672:afr_lookup_done] nas-d-volume-replicate-1: split brain detected during lookup of /.
[2011-03-01 18:12:54.862839] I [afr-common.c:716:afr_lookup_done] nas-d-volume-replicate-1: background  meta-data data self-heal triggered. path: /
[2011-03-01 18:12:54.862990] I [afr-common.c:672:afr_lookup_done] nas-d-volume-replicate-2: split brain detected during lookup of /.
[2011-03-01 18:12:54.863006] I [afr-common.c:716:afr_lookup_done] nas-d-volume-replicate-2: background  meta-data data self-heal triggered. path: /
[2011-03-01 18:12:54.863106] I [afr-common.c:672:afr_lookup_done] nas-d-volume-replicate-0: split brain detected during lookup of /.
[2011-03-01 18:12:54.863120] I [afr-common.c:716:afr_lookup_done] nas-d-volume-replicate-0: background  meta-data data self-heal triggered. path: /
[2011-03-01 18:12:54.863486] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] nas-d-volume-replicate-1: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2011-03-01 18:12:54.863599] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] nas-d-volume-replicate-2: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2011-03-01 18:12:54.863742] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] nas-d-volume-replicate-1: background  meta-data data self-heal completed on /
[2011-03-01 18:12:54.863804] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] nas-d-volume-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2011-03-01 18:12:54.863864] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] nas-d-volume-replicate-2: background  meta-data data self-heal completed on /
[2011-03-01 18:12:54.863987] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] nas-d-volume-replicate-0: background  meta-data data self-heal completed on /
[2011-03-01 18:12:55.880727] I [afr-common.c:672:afr_lookup_done] nas-d-volume-replicate-1: split brain detected during lookup of /.
[2011-03-01 18:12:55.880754] I [afr-common.c:716:afr_lookup_done] nas-d-volume-replicate-1: background  meta-data data self-heal triggered. path: /
[2011-03-01 18:12:55.880903] I [afr-common.c:672:afr_lookup_done] nas-d-volume-replicate-0: split brain detected during lookup of /.
[2011-03-01 18:12:55.880950] I [afr-common.c:716:afr_lookup_done] nas-d-volume-replicate-0: background  meta-data data self-heal triggered. path: /
[2011-03-01 18:12:55.881100] I [afr-common.c:672:afr_lookup_done] nas-d-volume-replicate-2: split brain detected during lookup of /.
[2011-03-01 18:12:55.881116] I [afr-common.c:716:afr_lookup_done] nas-d-volume-replicate-2: background  meta-data data self-heal triggered. path: /
[2011-03-01 18:12:55.881295] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] nas-d-volume-replicate-1: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2011-03-01 18:12:55.881543] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] nas-d-volume-replicate-1: background  meta-data data self-heal completed on /
[2011-03-01 18:12:55.881597] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] nas-d-volume-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2011-03-01 18:12:55.881784] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] nas-d-volume-replicate-0: background  meta-data data self-heal completed on /
[2011-03-01 18:12:55.881858] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] nas-d-volume-replicate-2: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2011-03-01 18:12:55.882045] I [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk] nas-d-volume-replicate-2: background  meta-data data self-heal completed on /

I am seeing the following server's information and volume status:
[root at nas-d01 ~]# gluster peer status
Number of Peers: 8

Hostname: nas-d07.linda.localdomain
Uuid: 7c314b62-5df9-4488-b517-2983eb09e895
State: Peer in Cluster (Connected)

Hostname: nas-d09.linda.localdomain
Uuid: edcba91f-65d1-4a42-a66a-67b18efb7534
State: Peer in Cluster (Connected)

Hostname: nas-d06.linda.localdomain
Uuid: 3e24be97-bc91-4462-8584-15d4df0099b1
State: Peer in Cluster (Connected)

Hostname: nas-d04.linda.localdomain
Uuid: ad46f49a-ddbf-4112-889f-d21628c8ff70
State: Peer in Cluster (Connected)

Hostname: nas-d03.linda.localdomain
Uuid: 9d22942b-14a1-4fa8-b1a4-513a3ddf7348
State: Peer in Cluster (Connected)

Hostname: nas-d05.linda.localdomain
Uuid: 6b419ddc-1f1a-41d0-b771-32b584fee3b2
State: Peer in Cluster (Connected)

Hostname: nas-d02.linda.localdomain
Uuid: f1194656-e5fc-4bd0-a784-c1f90cb4cff5
State: Peer in Cluster (Connected)

Hostname: nas-d08.linda.localdomain
Uuid: 1d4e8e0c-299f-4abe-82f0-d908bc8238a4
State: Peer in Cluster (Connected)

[root at nas-d01 ~]# gluster volume info

Volume Name: nas-d-volume
Type: Distributed-Replicate
Status: Started
Number of Bricks: 3 x 3 = 9
Transport-type: tcp
Bricks:
Brick1: nas-d01.linda.localdomain:/glusterfsd/data
Brick2: nas-d02.linda.localdomain:/glusterfsd/data
Brick3: nas-d03.linda.localdomain:/glusterfsd/data
Brick4: nas-d04.linda.localdomain:/glusterfsd/data
Brick5: nas-d05.linda.localdomain:/glusterfsd/data
Brick6: nas-d06.linda.localdomain:/glusterfsd/data
Brick7: nas-d07.linda.localdomain:/glusterfsd/data
Brick8: nas-d08.linda.localdomain:/glusterfsd/data
Brick9: nas-d09.linda.localdomain:/glusterfsd/data

[root at nas-d01 ~]# getfattr -d -e hex -m trusted.afr /glusterfsd/data/
getfattr: Removing leading '/' from absolute path names
# file: glusterfsd/data
trusted.afr.nas-d-volume-client-0=0x000000000000000000000000
trusted.afr.nas-d-volume-client-1=0x000000000100000000000000
trusted.afr.nas-d-volume-client-2=0x000000000000000000000000

[root at nas-d02 ~]# getfattr -d -e hex -m trusted.afr /glusterfsd/data/
getfattr: Removing leading '/' from absolute path names
# file: glusterfsd/data
trusted.afr.nas-d-volume-client-0=0x000000000200000000000000
trusted.afr.nas-d-volume-client-1=0x000000000000000000000000
trusted.afr.nas-d-volume-client-2=0x000000000100000000000000

[root at nas-d03 ~]# getfattr -d -e hex -m trusted.afr /glusterfsd/data/
getfattr: Removing leading '/' from absolute path names
# file: glusterfsd/data
trusted.afr.nas-d-volume-client-0=0x000000000000000000000000
trusted.afr.nas-d-volume-client-1=0x000000000200000000000000
trusted.afr.nas-d-volume-client-2=0x000000000000000000000000

[root at nas-d04 ~]# getfattr -d -e hex -m trusted.afr /glusterfsd/data/
getfattr: Removing leading '/' from absolute path names
# file: glusterfsd/data
trusted.afr.nas-d-volume-client-3=0x000000000000000000000000
trusted.afr.nas-d-volume-client-4=0x000000000000000000000000
trusted.afr.nas-d-volume-client-5=0x000000000000000000000000

[root at nas-d05 ~]# getfattr -d -e hex -m trusted.afr /glusterfsd/data/
getfattr: Removing leading '/' from absolute path names
# file: glusterfsd/data
trusted.afr.nas-d-volume-client-3=0x000000000000000000000000
trusted.afr.nas-d-volume-client-4=0x000000000000000000000000
trusted.afr.nas-d-volume-client-5=0x000000000200000000000000

[root at nas-d06 ~]# getfattr -d -e hex -m trusted.afr /glusterfsd/data/
getfattr: Removing leading '/' from absolute path names
# file: glusterfsd/data
trusted.afr.nas-d-volume-client-3=0x000000000100000000000000
trusted.afr.nas-d-volume-client-4=0x000000000100000000000000
trusted.afr.nas-d-volume-client-5=0x000000000000000000000000

[root at nas-d07 ~]# getfattr -d -e hex -m trusted.afr /glusterfsd/data/
getfattr: Removing leading '/' from absolute path names
# file: glusterfsd/data
trusted.afr.nas-d-volume-client-6=0x000000000000000000000000
trusted.afr.nas-d-volume-client-7=0x000000000100000000000000
trusted.afr.nas-d-volume-client-8=0x000000000100000000000000

[root at nas-d08 ~]# getfattr -d -e hex -m trusted.afr /glusterfsd/data/
getfattr: Removing leading '/' from absolute path names
# file: glusterfsd/data
trusted.afr.nas-d-volume-client-6=0x000000000200000100000000
trusted.afr.nas-d-volume-client-7=0x000000000000000000000000
trusted.afr.nas-d-volume-client-8=0x000000000200000000000000

[root at nas-d09 ~]#  getfattr -d -e hex -m trusted.afr /glusterfsd/data/
getfattr: Removing leading '/' from absolute path names
# file: glusterfsd/data
trusted.afr.nas-d-volume-client-6=0x000000000000000100000000
trusted.afr.nas-d-volume-client-7=0x000000000000000000000000
trusted.afr.nas-d-volume-client-8=0x000000000000000000000000