[Gluster-users] split brain on / just after installation

[Ravishankar N]

On 06/02/2015 09:10 AM, Carl L Hoffman wrote:
> Hello - I was wondering if someone could please help me.
>
> I've just setup Gluster 3.6 on two Ubuntu 14.04 hosts.  Gluster is setup to replicate two volumes (prod-volume, dev-volume) between the two hosts.  Replication is working fine.  The glustershd.log shows:

Are you sure you are running gluster 3.6?  The 
'afr_sh_print_split_brain_log' message appears only in gluster 3.5 or lower.


>
> [2015-06-02 03:28:04.495162] E [afr-self-heal-common.c:197:afr_sh_print_split_brain_log] 0-prod-volume-replicate-0: Unable to self-heal contents of '<gfid:00000000-0000-0000-0000-000000000001>' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix:  [ [ 0 2 ] [ 2 0 ] ]
>
> and the prod-volume logs shows:
>
> [2015-06-02 02:54:28.286268] E [afr-self-heal-common.c:197:afr_sh_print_split_brain_log] 0-prod-volume-replicate-0: Unable to self-heal contents of '/' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix:  [ [ 0 2 ] [ 2 0 ] ]
> [2015-06-02 02:54:28.287476] E [afr-self-heal-common.c:2212:afr_self_heal_completion_cbk] 0-prod-volume-replicate-0: background  meta-data self-heal failed on /
>
> I've checked against https://github.com/gluster/glusterfs/blob/6c578c03f0d44913d264494de5df004544c96271/doc/features/heal-info-and-split-brain-resolution.md but I can't see any scenario that covers mine.  The output of bluster volume heal prod-volume info is:


Is the metadata same on both bricks on the root?  (Compare  `ls -ld 
/export/prodvol/brick`  and `getfattr -d -m . -e hex 
/export/prodvol/brick` on both servers to see if anything is mismatching).
-Ravi

>
> Gathering Heal info on volume prod-volume has been successful
>
> Brick server1:/export/prodvol/brick
> Number of entries: 1
> /
>
> Brick server2
> Number of entries: 1
> /
>
>
> and doesn't show anything in split-brain.
>
> But the output of gluster volume heal prod-volume info split brain shows:
>
> Gathering Heal info on volume prod-volume has been successful
>
> Brick server1:/export/prodvol/brick
> Number of entries: 6
> at                    path on brick
> -----------------------------------
> 2015-06-02 03:28:04 /
> 2015-06-02 03:18:04 /
> 2015-06-02 03:08:04 /
> 2015-06-02 02:58:04 /
> 2015-06-02 02:48:04 /
> 2015-06-02 02:48:04 /
>
> Brick server2:/export/prodvol/brick
> Number of entries: 5
> at                    path on brick
> -----------------------------------
> 2015-06-02 03:28:00 /
> 2015-06-02 03:18:00 /
> 2015-06-02 03:08:00 /
> 2015-06-02 02:58:00 /
> 2015-06-02 02:48:04 /
>
>
> And the number continues to grow.  The count on server2 is always one behind server1.
>
> Could someone please help?
>
> Cheers,
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users