[Gluster-users] Monitoring and solving split-brain

Ravishankar N ravishankar at redhat.com
Thu Oct 15 01:38:48 UTC 2015



On 10/14/2015 11:08 PM, Игорь Бирюлин wrote:
> Thanks for detailed description.
> Do you have a plans add resolution GFID split-brain by 'gluster volume 
> heal VOLNAME split-brain ...' ?
Not at the moment..
> What the main different between GFID split-brain and data split brain? 
> On nodes this file absolutely different by data content and size or it 
> isn't 'data' in glusterfs meaning?
>
GFID is unique to a file (something akin to an inode number) and is 
assigned when a file is created. Data split-brain occurs when the file 
with same gfid already exists on both bricks, but there's a difference 
in the file's content. (eg. one write succeeded only on brick1 and 
another write only on brick2).
gfid split-brain occurs when a file creation happens twice (say an 
application does an open() with O_CREAT) but succeeds only on one brick 
each time.
> Best regards,
> Igor
>
>
>
> 2015-10-14 20:13 GMT+03:00 Ravishankar N <ravishankar at redhat.com 
> <mailto:ravishankar at redhat.com>>:
>
>
>
>     On 10/14/2015 10:05 PM, Игорь Бирюлин wrote:
>>     Thanks for your replay.
>>
>>     If I do listing in mount point (/repo):
>>     # ls /repo/xxx/keyrings/debian-keyring.gpg
>>     ls: cannot access /repo/xxx/keyrings/debian-keyring.gpg:
>>     Input/output error
>>     #
>>     In log /var/log/glusterfs/repo.log I see:
>>     [2015-10-14 16:27:36.006815] W [MSGID: 108008]
>>     [afr-self-heal-name.c:359:afr_selfheal_name_gfid_mismatch_check]
>>     0-repofiles-replicate-0: GFID mismatch for
>>     <gfid:4a99bf9d-7423-47d9-a09d-fabaa333eccf>/debian-keyring.gpg
>>     69aaeee6-624b-400a-aa46-b5c6166c014c on repofiles-client-1 and
>>     b95ad06e-786a-44e5-ba71-af661982071f on repofiles-client-0
>
>     So the file has ended up in GFID split-brain (The trusted.gfid
>     value is different in both bricks as seen in your output below.),
>     which cannot be handled by the split-brain resolution commands.
>     These commands can only resolve data and metadata split-brain. I'm
>     afraid you'll manually need to delete one of the file and the
>     .glusterfs hardlink from the brick. Not sure why the
>     parent-directory was not listed in 'gluster v heal VOLNAME info
>     split-brain' output.
>
>>     [2015-10-14 16:27:36.008996] W [fuse-bridge.c:451:fuse_entry_cbk]
>>     0-glusterfs-fuse: 65961: LOOKUP()
>>     /xxx/keyrings/debian-keyring.gpg => -1 (Input/output error)
>>
>>     On first node getfattr return:
>>     # getfattr -d -m . -e hex
>>     /storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg
>>     getfattr: Removing leading '/' from absolute path names
>>     # file:
>>     storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg
>>     trusted.afr.dirty=0x000000000000000000000000
>>     trusted.afr.repofiles-client-1=0x000000020000000100000000
>>     trusted.bit-rot.version=0x020000000000000055fdf0910003b37b
>>     trusted.gfid=0xb95ad06e786a44e5ba71af661982071f
>>     # ls -l
>>     /storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg
>>     -rw-r--r-- 2 root root 3456271 Oct 13 19:00
>>     /storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg
>>     #
>>
>>     On second node getfattr return:
>>     # getfattr -d -m . -e hex
>>     /storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg
>>     getfattr: Removing leading '/' from absolute path names
>>     # file:
>>     storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg
>>     trusted.afr.dirty=0x000000000000000000000000
>>     trusted.afr.repofiles-client-0=0x000000000000000000000000
>>     trusted.bit-rot.version=0x020000000000000055f97b57000dc3c6
>>     trusted.gfid=0x69aaeee6624b400aaa46b5c6166c014c
>>     # ls -l
>>     /storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg
>>     -rw-r--r-- 2 root root 3450346 Oct  9 16:22
>>     /storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg
>>     #
>>
>>     Best regards,
>>     Igor
>>
>>
>>
>>
>>
>>
>>     2015-10-14 19:14 GMT+03:00 Ravishankar N <ravishankar at redhat.com
>>     <mailto:ravishankar at redhat.com>>:
>>
>>
>>
>>         On 10/14/2015 07:02 PM, Игорь Бирюлин wrote:
>>>         Hello,
>>>         today in my 2 nodes replica set I've found split-brain.
>>>         Command 'ls' start told 'Input/output error'.
>>
>>         What does the mount log
>>         (/var/log/glusterfs/<path-to-mount>.log) say when you get
>>         this  error?
>>
>>         Can you run getfattr as root for the file from *both* bricks
>>         and share the result?
>>         `getfattr -d -m . -e hex
>>         /storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg`
>>
>>         Thanks.
>>         Ravi
>>
>>
>>>         But command 'gluster v heal VOLNAME info split-brain' does
>>>         not show problem files:
>>>         # gluster v heal repofiles info split-brain
>>>         Brick dist-int-master03.xxx:/storage/gluster_brick_repofiles
>>>         Number of entries in split-brain: 0
>>>
>>>         Brick dist-int-master04.xxx:/storage/gluster_brick_repofiles
>>>         Number of entries in split-brain: 0
>>>         #
>>>         In output of 'gluster v heal VOLNAME info' I see problem
>>>         files (/xxx/keyrings/debian-keyring.gpg, /repos.json), but
>>>         without split-brain markers:
>>>         # gluster v heal repofiles info
>>>         Brick dist-int-master03.xxx:/storage/gluster_brick_repofiles
>>>         /xxx/keyrings/debian-keyring.gpg
>>>         <gfid:09ec49c9-911a-4b83-abe8-080fe79e7c69>
>>>         <gfid:35c51b11-a7fb-496d-9e88-6d5a54fda7da>
>>>         /repos.json
>>>         <gfid:4f5cb2b5-30e2-43b0-a935-cfc42af883bf>
>>>         <gfid:9d2fc354-37c0-47a7-b9f3-379504cba797>
>>>         <gfid:cd86a246-9fc4-47d2-bb4d-67566677f77a>
>>>         <gfid:b932eed0-07e9-45c5-943e-7478e9f654b4>
>>>         <gfid:28bf2ffe-948c-4c7d-bce6-966242338581>
>>>         <gfid:ee5659ae-1335-42c5-a852-790387b4213b>
>>>         <gfid:fdfb6b8c-3c04-435a-b8d3-8d8341b66409>
>>>         Number of entries: 11
>>>
>>>         Brick dist-int-master04.xxx:/storage/gluster_brick_repofiles
>>>         Number of entries: 0
>>>         #
>>>
>>>         I couldn't solve split-brain by new standard command:
>>>         # gluster v heal repofiles split-brain bigger-file /repos.json
>>>         Lookup failed on /repos.json:Input/output error
>>>         Volume heal failed.
>>>         #
>>>
>>>         Additional info:
>>>         # gluster v info
>>>          Volume Name: repofiles
>>>          Type: Replicate
>>>          Volume ID: 4b0e2a74-f1ca-4fe7-8518-23919e1b5fa0
>>>          Status: Started
>>>          Number of Bricks: 1 x 2 = 2
>>>          Transport-type: tcp
>>>          Bricks:
>>>          Brick1: dist-int-master03.xxx:/storage/gluster_brick_repofiles
>>>          Brick2: dist-int-master04.xxx:/storage/gluster_brick_repofiles
>>>          Options Reconfigured:
>>>          performance.readdir-ahead: on
>>>          client.event-threads: 4
>>>          server.event-threads: 4
>>>          cluster.lookup-optimize: on
>>>         # cat /etc/issue
>>>         Ubuntu 14.04.3 LTS \n \l
>>>         # dpkg -l | grep glusterfs
>>>         ii glusterfs-client 3.7.5-ubuntu1~trusty1 amd64       
>>>         clustered file-system (client package)
>>>         ii glusterfs-common 3.7.5-ubuntu1~trusty1 amd64       
>>>         GlusterFS common libraries and translator modules
>>>         ii glusterfs-server 3.7.5-ubuntu1~trusty1 amd64       
>>>         clustered file-system (server package)
>>>         #
>>>
>>>         I have 2 questions:
>>>         1. Why 'gluster v heal VOLNAME info split-brain' doesn't
>>>         show actual split-brain? Why in 'gluster v heal VOLNAME
>>>         info' I doesn't see markers like 'possible in split-brain'?
>>>         How I can monitor my gluster installation if these commands
>>>         doesn't show problems?
>>>         2. Why 'gluster volume heal VOLNAME split-brain bigger-file
>>>         FILE' doesn't solve split-brain? I understand that I can
>>>         solve split-brain remove files from brick but I thought to
>>>         use this killer feature.
>>>
>>>         Best regards,
>>>         Igor
>>>
>>>
>>>         _______________________________________________
>>>         Gluster-users mailing list
>>>         Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>         http://www.gluster.org/mailman/listinfo/gluster-users
>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151015/78fb744a/attachment.html>


More information about the Gluster-users mailing list