[Gluster-users] split brain on / just after installation

[Carl L Hoffman]

Hello - I was wondering if someone could please help me.

I've just setup Gluster 3.6 on two Ubuntu 14.04 hosts.  Gluster is setup to replicate two volumes (prod-volume, dev-volume) between the two hosts.  Replication is working fine.  The glustershd.log shows:

[2015-06-02 03:28:04.495162] E [afr-self-heal-common.c:197:afr_sh_print_split_brain_log] 0-prod-volume-replicate-0: Unable to self-heal contents of '<gfid:00000000-0000-0000-0000-000000000001>' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix:  [ [ 0 2 ] [ 2 0 ] ]

and the prod-volume logs shows:

[2015-06-02 02:54:28.286268] E [afr-self-heal-common.c:197:afr_sh_print_split_brain_log] 0-prod-volume-replicate-0: Unable to self-heal contents of '/' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix:  [ [ 0 2 ] [ 2 0 ] ]
[2015-06-02 02:54:28.287476] E [afr-self-heal-common.c:2212:afr_self_heal_completion_cbk] 0-prod-volume-replicate-0: background  meta-data self-heal failed on /

I've checked against https://github.com/gluster/glusterfs/blob/6c578c03f0d44913d264494de5df004544c96271/doc/features/heal-info-and-split-brain-resolution.md but I can't see any scenario that covers mine.  The output of bluster volume heal prod-volume info is:

Gathering Heal info on volume prod-volume has been successful

Brick server1:/export/prodvol/brick
Number of entries: 1
/

Brick server2
Number of entries: 1
/


and doesn't show anything in split-brain.

But the output of gluster volume heal prod-volume info split brain shows:

Gathering Heal info on volume prod-volume has been successful

Brick server1:/export/prodvol/brick
Number of entries: 6
at                    path on brick
-----------------------------------
2015-06-02 03:28:04 /
2015-06-02 03:18:04 /
2015-06-02 03:08:04 /
2015-06-02 02:58:04 /
2015-06-02 02:48:04 /
2015-06-02 02:48:04 /

Brick server2:/export/prodvol/brick
Number of entries: 5
at                    path on brick
-----------------------------------
2015-06-02 03:28:00 /
2015-06-02 03:18:00 /
2015-06-02 03:08:00 /
2015-06-02 02:58:00 /
2015-06-02 02:48:04 /


And the number continues to grow.  The count on server2 is always one behind server1.

Could someone please help?

Cheers,