[Gluster-users] Monitoring and solving split-brain

[Ravishankar N]

On 10/14/2015 07:02 PM, Игорь Бирюлин wrote:
> Hello,
> today in my 2 nodes replica set I've found split-brain. Command 'ls' 
> start told 'Input/output error'.

What does the mount log (/var/log/glusterfs/<path-to-mount>.log) say 
when you get this  error?

Can you run getfattr as root for the file from *both* bricks and share 
the result?
`getfattr -d -m . -e hex 
/storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg`

Thanks.
Ravi


> But command 'gluster v heal VOLNAME info split-brain' does not show 
> problem files:
> # gluster v heal repofiles info split-brain
> Brick dist-int-master03.xxx:/storage/gluster_brick_repofiles
> Number of entries in split-brain: 0
>
> Brick dist-int-master04.xxx:/storage/gluster_brick_repofiles
> Number of entries in split-brain: 0
> #
> In output of 'gluster v heal VOLNAME info' I see problem files 
> (/xxx/keyrings/debian-keyring.gpg, /repos.json), but without 
> split-brain markers:
> # gluster v heal repofiles info
> Brick dist-int-master03.xxx:/storage/gluster_brick_repofiles
> /xxx/keyrings/debian-keyring.gpg
> <gfid:09ec49c9-911a-4b83-abe8-080fe79e7c69>
> <gfid:35c51b11-a7fb-496d-9e88-6d5a54fda7da>
> /repos.json
> <gfid:4f5cb2b5-30e2-43b0-a935-cfc42af883bf>
> <gfid:9d2fc354-37c0-47a7-b9f3-379504cba797>
> <gfid:cd86a246-9fc4-47d2-bb4d-67566677f77a>
> <gfid:b932eed0-07e9-45c5-943e-7478e9f654b4>
> <gfid:28bf2ffe-948c-4c7d-bce6-966242338581>
> <gfid:ee5659ae-1335-42c5-a852-790387b4213b>
> <gfid:fdfb6b8c-3c04-435a-b8d3-8d8341b66409>
> Number of entries: 11
>
> Brick dist-int-master04.xxx:/storage/gluster_brick_repofiles
> Number of entries: 0
> #
>
> I couldn't solve split-brain by new standard command:
> # gluster v heal repofiles  split-brain bigger-file /repos.json
> Lookup failed on /repos.json:Input/output error
> Volume heal failed.
> #
>
> Additional info:
> # gluster v info
>  Volume Name: repofiles
>  Type: Replicate
>  Volume ID: 4b0e2a74-f1ca-4fe7-8518-23919e1b5fa0
>  Status: Started
>  Number of Bricks: 1 x 2 = 2
>  Transport-type: tcp
>  Bricks:
>  Brick1: dist-int-master03.xxx:/storage/gluster_brick_repofiles
>  Brick2: dist-int-master04.xxx:/storage/gluster_brick_repofiles
>  Options Reconfigured:
>  performance.readdir-ahead: on
>  client.event-threads: 4
>  server.event-threads: 4
>  cluster.lookup-optimize: on
> # cat /etc/issue
> Ubuntu 14.04.3 LTS \n \l
> # dpkg -l | grep glusterfs
> ii  glusterfs-client 3.7.5-ubuntu1~trusty1                amd64        
> clustered file-system (client package)
> ii  glusterfs-common 3.7.5-ubuntu1~trusty1                amd64        
> GlusterFS common libraries and translator modules
> ii  glusterfs-server 3.7.5-ubuntu1~trusty1                amd64        
> clustered file-system (server package)
> #
>
> I have 2 questions:
> 1. Why 'gluster v heal VOLNAME info split-brain' doesn't show actual 
> split-brain? Why in 'gluster v heal VOLNAME info' I doesn't see 
> markers like 'possible in split-brain'?
> How I can monitor my gluster installation if these commands doesn't 
> show problems?
> 2. Why 'gluster volume heal VOLNAME split-brain bigger-file FILE' 
> doesn't solve split-brain? I understand that I can solve split-brain 
> remove files from brick but I thought to use this killer feature.
>
> Best regards,
> Igor
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151014/ad4478a0/attachment.html>