[Gluster-users] Monitoring and solving split-brain

[Игорь Бирюлин]

Hello,
today in my 2 nodes replica set I've found split-brain. Command 'ls' start
told 'Input/output error'.
But command 'gluster v heal VOLNAME info split-brain' does not show problem
files:
# gluster v heal repofiles info split-brain
Brick dist-int-master03.xxx:/storage/gluster_brick_repofiles
Number of entries in split-brain: 0

Brick dist-int-master04.xxx:/storage/gluster_brick_repofiles
Number of entries in split-brain: 0
#
In output of 'gluster v heal VOLNAME info' I see problem files
(/xxx/keyrings/debian-keyring.gpg, /repos.json), but without split-brain
markers:
# gluster v heal repofiles info
Brick dist-int-master03.xxx:/storage/gluster_brick_repofiles
/xxx/keyrings/debian-keyring.gpg
<gfid:09ec49c9-911a-4b83-abe8-080fe79e7c69>
<gfid:35c51b11-a7fb-496d-9e88-6d5a54fda7da>
/repos.json
<gfid:4f5cb2b5-30e2-43b0-a935-cfc42af883bf>
<gfid:9d2fc354-37c0-47a7-b9f3-379504cba797>
<gfid:cd86a246-9fc4-47d2-bb4d-67566677f77a>
<gfid:b932eed0-07e9-45c5-943e-7478e9f654b4>
<gfid:28bf2ffe-948c-4c7d-bce6-966242338581>
<gfid:ee5659ae-1335-42c5-a852-790387b4213b>
<gfid:fdfb6b8c-3c04-435a-b8d3-8d8341b66409>
Number of entries: 11

Brick dist-int-master04.xxx:/storage/gluster_brick_repofiles
Number of entries: 0
#

I couldn't solve split-brain by new standard command:
# gluster v heal repofiles  split-brain bigger-file /repos.json
Lookup failed on /repos.json:Input/output error
Volume heal failed.
#

Additional info:
# gluster v info
 Volume Name: repofiles
 Type: Replicate
 Volume ID: 4b0e2a74-f1ca-4fe7-8518-23919e1b5fa0
 Status: Started
 Number of Bricks: 1 x 2 = 2
 Transport-type: tcp
 Bricks:
 Brick1: dist-int-master03.xxx:/storage/gluster_brick_repofiles
 Brick2: dist-int-master04.xxx:/storage/gluster_brick_repofiles
 Options Reconfigured:
 performance.readdir-ahead: on
 client.event-threads: 4
 server.event-threads: 4
 cluster.lookup-optimize: on
# cat /etc/issue
Ubuntu 14.04.3 LTS \n \l
# dpkg -l | grep glusterfs
ii  glusterfs-client
3.7.5-ubuntu1~trusty1                amd64        clustered file-system
(client package)
ii  glusterfs-common
3.7.5-ubuntu1~trusty1                amd64        GlusterFS common
libraries and translator modules
ii  glusterfs-server
3.7.5-ubuntu1~trusty1                amd64        clustered file-system
(server package)
#

I have 2 questions:
1. Why 'gluster v heal VOLNAME info split-brain' doesn't show actual
split-brain? Why in 'gluster v heal VOLNAME info' I doesn't see markers
like 'possible in split-brain'?
How I can monitor my gluster installation if these commands doesn't show
problems?
2. Why 'gluster volume heal VOLNAME split-brain bigger-file FILE' doesn't
solve split-brain? I understand that I can solve split-brain remove files
from brick but I thought to use this killer feature.

Best regards,
Igor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151014/82c6cf69/attachment.html>