[Gluster-users] Split brain or what

[Jesper Led Lauridsen TS Infra server]

Hi,

I have a wired situation. Is this as it should be? or what is going on

# gluster --version
glusterfs 3.6.2 built on Jan 22 2015 12:58:11

There are 8 files reported in split-brain (1024 times), but they seem to be identical.
# gluster volume heal glu_rhevtst_dr2_data_01 info split-brain| awk '$3 ~ /^\// {print $3}'|sort|uniq -d|wc -l
8

Also when I run heal info without split-brain it all looks fine.
# gluster volume heal glu_rhevtst_dr2_data_01 info
-- snip --
Brick glustertst03.net.dr.dk:/bricks/brick1/glu_rhevtst_dr2_data_01/
Number of entries: 0

Brick glustertst04.net.dr.dk:/bricks/brick1/glu_rhevtst_dr2_data_01/
Number of entries: 0

Brick glustertst03.net.dr.dk:/bricks/brick2/glu_rhevtst_dr2_data_01/
Number of entries: 0

Brick glustertst04.net.dr.dk:/bricks/brick2/glu_rhevtst_dr2_data_01/
Number of entries: 0

The fattr and md5sum on the files are identical on both nodes (except for the contri=). Here is one sample
[root at glustertst03 ~]# getfattr -d -m . -e hex /bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/8e488152-be3c-42fb-909f-fd123a033a8e/7718978e-cd30-4916-966e-c9cd65e6f14c
getfattr: Removing leading '/' from absolute path names
# file: bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/8e488152-be3c-42fb-909f-fd123a033a8e/7718978e-cd30-4916-966e-c9cd65e6f14c
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.glu_rhevtst_dr2_data_01-client-0=0x000000000000000000000000
trusted.afr.glu_rhevtst_dr2_data_01-client-1=0x000000000000000000000000
trusted.gfid=0x4c717a67aed64ad4b78b9881af44be61
trusted.glusterfs.quota.9fc26a27-988b-4926-817e-d184d87f16bf.contri=0x000000077275f000
trusted.pgfid.9fc26a27-988b-4926-817e-d184d87f16bf=0x00000001

[root at glustertst03 ~]# md5sum /bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/8e488152-be3c-42fb-909f-fd123a033a8e/7718978e-cd30-4916-966e-c9cd65e6f14c
bd9b7b68cc9f80fb9e7782a4569afa2b  /bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/8e488152-be3c-42fb-909f-fd123a033a8e/7718978e-cd30-4916-966e-c9cd65e6f14c

[root at glustertst04 ~]# getfattr -d -m . -e hex /bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/8e488152-be3c-42fb-909f-fd123a033a8e/7718978e-cd30-4916-966e-c9cd65e6f14c
getfattr: Removing leading '/' from absolute path names
# file: bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/8e488152-be3c-42fb-909f-fd123a033a8e/7718978e-cd30-4916-966e-c9cd65e6f14c
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.glu_rhevtst_dr2_data_01-client-0=0x000000000000000000000000
trusted.afr.glu_rhevtst_dr2_data_01-client-1=0x000000000000000000000000
trusted.gfid=0x4c717a67aed64ad4b78b9881af44be61
trusted.glusterfs.quota.9fc26a27-988b-4926-817e-d184d87f16bf.contri=0x0000000772958000
trusted.pgfid.9fc26a27-988b-4926-817e-d184d87f16bf=0x00000001

[root at glustertst04 ~]# md5sum /bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/8e488152-be3c-42fb-909f-fd123a033a8e/7718978e-cd30-4916-966e-c9cd65e6f14c
bd9b7b68cc9f80fb9e7782a4569afa2b  /bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/8e488152-be3c-42fb-909f-fd123a033a8e/7718978e-cd30-4916-966e-c9cd65e6f14c

# gluster volume heal glu_rhevtst_dr2_data_01 info split-brain
  -- snip  --
Brick glustoretst04.net.dr.dk:/bricks/brick2/glu_rhevtst_dr2_data_01
Number of entries: 1024
at                    path on brick
-----------------------------------
2015-07-14 03:56:22  /6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/a2a61288-e39e-4765-a7a5-870b5a248e8d/e168cf91-8005-49da-9d11-b07f326898ba

Brick glustoretst03.net.dr.dk:/bricks/brick2/glu_rhevtst_dr2_data_01
Number of entries: 1024
at                    path on brick
-----------------------------------
2015-07-14 03:55:02  /6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/a2a61288-e39e-4765-a7a5-870b5a248e8d/e168cf91-8005-49da-9d11-b07f326898ba

Brick glustoretst04.net.dr.dk:/bricks/brick1/glu_rhevtst_dr2_data_01
Number of entries: 1024
at                    path on brick
-----------------------------------
2015-07-14 08:01:22 /6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/8e488152-be3c-42fb-909f-fd123a033a8e/7718978e-cd30-4916-966e-c9cd65e6f14c

Brick glustoretst03.net.dr.dk:/bricks/brick1/glu_rhevtst_dr2_data_01
Number of entries: 1024
at                    path on brick
-----------------------------------
2015-07-14 07:41:25 /6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/8e488152-be3c-42fb-909f-fd123a033a8e/7718978e-cd30-4916-966e-c9cd65e6f14c

# gluster volume info glu_rhevtst_dr2_data_01

Volume Name: glu_rhevtst_dr2_data_01
Type: Distributed-Replicate
Volume ID: 5f24fb9e-76cd-4610-85c5-4b4734349958
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: glustoretst03.net.dr.dk:/bricks/brick1/glu_rhevtst_dr2_data_01
Brick2: glustoretst04.net.dr.dk:/bricks/brick1/glu_rhevtst_dr2_data_01
Brick3: glustoretst03.net.dr.dk:/bricks/brick2/glu_rhevtst_dr2_data_01
Brick4: glustoretst04.net.dr.dk:/bricks/brick2/glu_rhevtst_dr2_data_01
Brick5: glustoretst01.net.dr.dk:/bricks/brick1/glu_rhevtst_dr2_data_01
Brick6: glustoretst02.net.dr.dk:/bricks/brick1/glu_rhevtst_dr2_data_01
Brick7: glustoretst01.net.dr.dk:/bricks/brick2/glu_rhevtst_dr2_data_01
Brick8: glustoretst02.net.dr.dk:/bricks/brick2/glu_rhevtst_dr2_data_01
Options Reconfigured:
nfs.disable: on
user.cifs: disable
auth.allow: xxxxxxx
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
cluster.quorum-type: none
cluster.server-quorum-type: server
storage.owner-uid: 36
storage.owner-gid: 36
features.quota-deem-statfs: on
features.quota: on
cluster.server-quorum-ratio: 50%

Thanks
Jesper



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150722/d5c94ede/attachment.html>