[Gluster-users] Split brain that is not split brain
Pranith Kumar Karampuri
pkarampu at redhat.com
Thu Sep 11 07:14:12 UTC 2014
On 09/11/2014 11:37 AM, Ilya Ivanov wrote:
> I don't understand why there's such a complicated process to recover
> when I can just look at both files, decide which one I need and delete
> another one.
If the file needs to be deleted the whole file needs to be copied which
is fine for small files but for big files like VM images it takes less
time if the file already exists and it syncs only the parts of files
that are different from the good copy.
One more reason is if the parent directory from which the file is
deleted from is the source then self-heal will delete the file from
other directory rather than creating it. SO instead of deleting the file
may be it is a better practise to make a copy of the file somewhere and
delete it. We shall update the document as well with this new
information. Thanks for the feedback. In 3.7 it is going to be
simplified. We are giving a command to fix the split-brains where the
user gets to choose the file and it will do the rest.
Pranith
>
>
> On Thu, Sep 11, 2014 at 7:56 AM, Pranith Kumar Karampuri
> <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote:
>
>
> On 09/11/2014 09:29 AM, Ilya Ivanov wrote:
>> Right... I deleted it and now all appears to be fine.
>>
>> Still, could you please elaborate on "gfid split-brain"?
> Could you go through
> https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md
> Let us know if you would like something to be more clearer and we
> can add that and improve the document.
>
> Pranith
>
>>
>>
>> On Thu, Sep 11, 2014 at 5:32 AM, Pranith Kumar Karampuri
>> <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote:
>>
>>
>> On 09/11/2014 12:16 AM, Ilya Ivanov wrote:
>>> Any insight?
>> Was the other file's gfid d3def9e1-c6d0-4b7d-a322-b5019305182e?
>> Could you check if this file exists in <brick>/.glusterfs/d3/de/
>> When a file is deleted this file also needs to be deleted if
>> there are no more hardlinks to the file
>>
>> Pranith
>>>
>>> On Tue, Sep 9, 2014 at 8:35 AM, Ilya Ivanov
>>> <bearwere at gmail.com <mailto:bearwere at gmail.com>> wrote:
>>>
>>> What's a gfid split-brain and how is it different from
>>> "normal" split-brain?
>>>
>>> I accessed the file with stat, but "heal info" still
>>> shows "Number of entries: 1"
>>>
>>> [root at gluster1 gluster]# getfattr -d -m. -e hex gv01/123
>>> # getfattr -d -m. -e hex gv01/123
>>> # file: gv01/123
>>> trusted.afr.gv01-client-0=0x000000000000000000000000
>>> trusted.afr.gv01-client-1=0x000000000000000000000000
>>> trusted.gfid=0x35f86f4561134ba0bd1b94ef70179d4d
>>>
>>> [root at gluster1 gluster]# getfattr -d -m. -e hex gv01
>>> # file: gv01
>>> trusted.afr.gv01-client-0=0x000000000000000000000000
>>> trusted.afr.gv01-client-1=0x000000000000000000000000
>>> trusted.gfid=0x00000000000000000000000000000001
>>> trusted.glusterfs.dht=0x000000010000000000000000ffffffff
>>> trusted.glusterfs.volume-id=0x31a2c4c486ca4344b838d2c2e6c716c1
>>>
>>>
>>>
>>> On Tue, Sep 9, 2014 at 8:19 AM, Pranith Kumar Karampuri
>>> <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote:
>>>
>>>
>>> On 09/09/2014 11:35 AM, Ilya Ivanov wrote:
>>>> Ahh, thank you, now I get it. I deleted it on one
>>>> node and it replicated to another one. Now I get
>>>> the following output:
>>>>
>>>> [root at gluster1 var]# gluster volume heal gv01 info
>>>> Brick gluster1:/home/gluster/gv01/
>>>> <gfid:d3def9e1-c6d0-4b7d-a322-b5019305182e>
>>>> Number of entries: 1
>>>>
>>>> Brick gluster2:/home/gluster/gv01/
>>>> Number of entries: 0
>>>>
>>>> Is it normal? Why the number of entries isn't reset
>>>> to 0?
>>> If you access the file using ls/stat etc, it will be
>>> fixed. But before that could you please post the
>>> output of 'getfattr -d -m. -e hex
>>> file/path/in/backend/brick' and 'getfattr -d -m. -e
>>> hex parent/dir/to/file/path/in/backend/brick'
>>>
>>> Pranith
>>>>
>>>>
>>>> And why wouldn't the file show up in split-brain
>>>> before, anyway?
>>> Gfid split-brains are not shown in
>>> heal-info-split-brain yet.
>>>
>>> Pranith
>>>>
>>>>
>>>> On Tue, Sep 9, 2014 at 7:46 AM, Pranith Kumar
>>>> Karampuri <pkarampu at redhat.com
>>>> <mailto:pkarampu at redhat.com>> wrote:
>>>>
>>>>
>>>> On 09/09/2014 01:54 AM, Ilya Ivanov wrote:
>>>>> Hello.
>>>>>
>>>>> I've Gluster 3.5.2 on Centos 6. A primitive
>>>>> replicated volume, as describe here
>>>>> <https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers>.
>>>>> I tried to simulate split-brain by temporarily
>>>>> disconnecting the nodes and creating a file
>>>>> with the same name and different contents.
>>>>> That worked.
>>>>>
>>>>> The question is, how do I fix it now? All the
>>>>> tutorials suggest deleting the file from one
>>>>> of the nodes. I can't do that, it reports
>>>>> "Input/output error". The file won't even show
>>>>> up in "gluster volume heal gv00 info
>>>>> split-brain". That shows 0 entries.
>>>> The deletion needs to happen on one of the
>>>> bricks, not from the mount point.
>>>>
>>>> Pranith
>>>>> I can see the file in "gluster volume heal
>>>>> gv00 info heal-failed", though.
>>>>>
>>>>>
>>>>> --
>>>>> Ilya.
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Ilya.
>>>
>>>
>>>
>>>
>>> --
>>> Ilya.
>>>
>>>
>>>
>>>
>>> --
>>> Ilya.
>>
>>
>>
>>
>> --
>> Ilya.
>
>
>
>
> --
> Ilya.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140911/bbb12858/attachment.html>
More information about the Gluster-users
mailing list