[Gluster-users] Split brain that is not split brain
Pranith Kumar Karampuri
pkarampu at redhat.com
Thu Sep 11 08:44:52 UTC 2014
On 09/11/2014 01:13 PM, Ilya Ivanov wrote:
> Makes some sense. Yes, I meant "make a backup and delete", rather than
> just delete.
>
> If I may suggest, putting that debug link somewhere more visible would
> be be good, too. I wouldn't find without your help.
Justin, where shall we put the doc?
Pranith
>
> Thank you for the assistance.
>
>
>
>
> On Thu, Sep 11, 2014 at 9:14 AM, Pranith Kumar Karampuri
> <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote:
>
>
> On 09/11/2014 11:37 AM, Ilya Ivanov wrote:
>> I don't understand why there's such a complicated process to
>> recover when I can just look at both files, decide which one I
>> need and delete another one.
> If the file needs to be deleted the whole file needs to be copied
> which is fine for small files but for big files like VM images it
> takes less time if the file already exists and it syncs only the
> parts of files that are different from the good copy.
> One more reason is if the parent directory from which the file is
> deleted from is the source then self-heal will delete the file
> from other directory rather than creating it. SO instead of
> deleting the file may be it is a better practise to make a copy of
> the file somewhere and delete it. We shall update the document as
> well with this new information. Thanks for the feedback. In 3.7 it
> is going to be simplified. We are giving a command to fix the
> split-brains where the user gets to choose the file and it will do
> the rest.
>
>
> Pranith
>>
>>
>> On Thu, Sep 11, 2014 at 7:56 AM, Pranith Kumar Karampuri
>> <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote:
>>
>>
>> On 09/11/2014 09:29 AM, Ilya Ivanov wrote:
>>> Right... I deleted it and now all appears to be fine.
>>>
>>> Still, could you please elaborate on "gfid split-brain"?
>> Could you go through
>> https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md
>> Let us know if you would like something to be more clearer
>> and we can add that and improve the document.
>>
>> Pranith
>>
>>>
>>>
>>> On Thu, Sep 11, 2014 at 5:32 AM, Pranith Kumar Karampuri
>>> <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote:
>>>
>>>
>>> On 09/11/2014 12:16 AM, Ilya Ivanov wrote:
>>>> Any insight?
>>> Was the other file's gfid
>>> d3def9e1-c6d0-4b7d-a322-b5019305182e?
>>> Could you check if this file exists in
>>> <brick>/.glusterfs/d3/de/
>>> When a file is deleted this file also needs to be
>>> deleted if there are no more hardlinks to the file
>>>
>>> Pranith
>>>>
>>>> On Tue, Sep 9, 2014 at 8:35 AM, Ilya Ivanov
>>>> <bearwere at gmail.com <mailto:bearwere at gmail.com>> wrote:
>>>>
>>>> What's a gfid split-brain and how is it different
>>>> from "normal" split-brain?
>>>>
>>>> I accessed the file with stat, but "heal info"
>>>> still shows "Number of entries: 1"
>>>>
>>>> [root at gluster1 gluster]# getfattr -d -m. -e hex
>>>> gv01/123
>>>> # getfattr -d -m. -e hex gv01/123
>>>> # file: gv01/123
>>>> trusted.afr.gv01-client-0=0x000000000000000000000000
>>>> trusted.afr.gv01-client-1=0x000000000000000000000000
>>>> trusted.gfid=0x35f86f4561134ba0bd1b94ef70179d4d
>>>>
>>>> [root at gluster1 gluster]# getfattr -d -m. -e hex gv01
>>>> # file: gv01
>>>> trusted.afr.gv01-client-0=0x000000000000000000000000
>>>> trusted.afr.gv01-client-1=0x000000000000000000000000
>>>> trusted.gfid=0x00000000000000000000000000000001
>>>> trusted.glusterfs.dht=0x000000010000000000000000ffffffff
>>>> trusted.glusterfs.volume-id=0x31a2c4c486ca4344b838d2c2e6c716c1
>>>>
>>>>
>>>>
>>>> On Tue, Sep 9, 2014 at 8:19 AM, Pranith Kumar
>>>> Karampuri <pkarampu at redhat.com
>>>> <mailto:pkarampu at redhat.com>> wrote:
>>>>
>>>>
>>>> On 09/09/2014 11:35 AM, Ilya Ivanov wrote:
>>>>> Ahh, thank you, now I get it. I deleted it on
>>>>> one node and it replicated to another one. Now
>>>>> I get the following output:
>>>>>
>>>>> [root at gluster1 var]# gluster volume heal gv01 info
>>>>> Brick gluster1:/home/gluster/gv01/
>>>>> <gfid:d3def9e1-c6d0-4b7d-a322-b5019305182e>
>>>>> Number of entries: 1
>>>>>
>>>>> Brick gluster2:/home/gluster/gv01/
>>>>> Number of entries: 0
>>>>>
>>>>> Is it normal? Why the number of entries isn't
>>>>> reset to 0?
>>>> If you access the file using ls/stat etc, it
>>>> will be fixed. But before that could you please
>>>> post the output of 'getfattr -d -m. -e hex
>>>> file/path/in/backend/brick' and 'getfattr -d
>>>> -m. -e hex
>>>> parent/dir/to/file/path/in/backend/brick'
>>>>
>>>> Pranith
>>>>>
>>>>>
>>>>> And why wouldn't the file show up in
>>>>> split-brain before, anyway?
>>>> Gfid split-brains are not shown in
>>>> heal-info-split-brain yet.
>>>>
>>>> Pranith
>>>>>
>>>>>
>>>>> On Tue, Sep 9, 2014 at 7:46 AM, Pranith Kumar
>>>>> Karampuri <pkarampu at redhat.com
>>>>> <mailto:pkarampu at redhat.com>> wrote:
>>>>>
>>>>>
>>>>> On 09/09/2014 01:54 AM, Ilya Ivanov wrote:
>>>>>> Hello.
>>>>>>
>>>>>> I've Gluster 3.5.2 on Centos 6. A
>>>>>> primitive replicated volume, as describe
>>>>>> here
>>>>>> <https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers>.
>>>>>> I tried to simulate split-brain by
>>>>>> temporarily disconnecting the nodes and
>>>>>> creating a file with the same name and
>>>>>> different contents. That worked.
>>>>>>
>>>>>> The question is, how do I fix it now? All
>>>>>> the tutorials suggest deleting the file
>>>>>> from one of the nodes. I can't do that,
>>>>>> it reports "Input/output error". The file
>>>>>> won't even show up in "gluster volume
>>>>>> heal gv00 info split-brain". That shows 0
>>>>>> entries.
>>>>> The deletion needs to happen on one of the
>>>>> bricks, not from the mount point.
>>>>>
>>>>> Pranith
>>>>>> I can see the file in "gluster volume
>>>>>> heal gv00 info heal-failed", though.
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ilya.
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Ilya.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Ilya.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Ilya.
>>>
>>>
>>>
>>>
>>> --
>>> Ilya.
>>
>>
>>
>>
>> --
>> Ilya.
>
>
>
>
> --
> Ilya.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140911/11642463/attachment.html>
More information about the Gluster-users
mailing list