[Gluster-users] Split brain that is not split brain

Thu Sep 11 07:43:42 UTC 2014

Makes some sense. Yes, I meant "make a backup and delete", rather than just
delete.

If I may suggest, putting that debug link somewhere more visible would be
be good, too. I wouldn't find without your help.

Thank you for the assistance.

On Thu, Sep 11, 2014 at 9:14 AM, Pranith Kumar Karampuri <
pkarampu at redhat.com> wrote:

>
> On 09/11/2014 11:37 AM, Ilya Ivanov wrote:
>
> I don't understand why there's such a complicated process to recover when
> I can just look at both files, decide which one I need and delete another
> one.
>
> If the file needs to be deleted the whole file needs to be copied which is
> fine for small files but for big files like VM images it takes less time if
> the file already exists and it syncs only the parts of files that are
> different from the good copy.
> One more reason is if the parent directory from which the file is deleted
> from is the source then self-heal will delete the file from other directory
> rather than creating it. SO instead of deleting the file may be it is a
> better practise to make a copy of the file somewhere and delete it. We
> shall update the document as well with this new information. Thanks for the
> feedback. In 3.7 it is going to be simplified. We are giving a command to
> fix the split-brains where the user gets to choose the file and it will do
> the rest.
>
>
> Pranith
>
>
>
> On Thu, Sep 11, 2014 at 7:56 AM, Pranith Kumar Karampuri <
> pkarampu at redhat.com> wrote:
>
>>
>> On 09/11/2014 09:29 AM, Ilya Ivanov wrote:
>>
>>  Right... I deleted it and now all appears to be fine.
>>
>>  Still, could you please elaborate on "gfid split-brain"?
>>
>>  Could you go through
>> https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md
>> Let us know if you would like something to be more clearer and we can add
>> that and improve the document.
>>
>> Pranith
>>
>>
>>
>> On Thu, Sep 11, 2014 at 5:32 AM, Pranith Kumar Karampuri <
>> pkarampu at redhat.com> wrote:
>>
>>>
>>> On 09/11/2014 12:16 AM, Ilya Ivanov wrote:
>>>
>>> Any insight?
>>>
>>> Was the other file's gfid d3def9e1-c6d0-4b7d-a322-b5019305182e?
>>> Could you check if this file exists in <brick>/.glusterfs/d3/de/
>>> When a file is deleted this file also needs to be deleted if there are
>>> no more hardlinks to the file
>>>
>>> Pranith
>>>
>>>
>>> On Tue, Sep 9, 2014 at 8:35 AM, Ilya Ivanov <bearwere at gmail.com> wrote:
>>>
>>>> What's a gfid split-brain and how is it different from "normal"
>>>> split-brain?
>>>>
>>>>  I accessed the file with stat, but "heal info" still shows "Number of
>>>> entries: 1"
>>>>
>>>> [root at gluster1 gluster]# getfattr -d -m. -e hex gv01/123
>>>> # getfattr -d -m. -e hex gv01/123
>>>> # file: gv01/123
>>>> trusted.afr.gv01-client-0=0x000000000000000000000000
>>>> trusted.afr.gv01-client-1=0x000000000000000000000000
>>>> trusted.gfid=0x35f86f4561134ba0bd1b94ef70179d4d
>>>>
>>>> [root at gluster1 gluster]# getfattr -d -m. -e hex gv01
>>>> # file: gv01
>>>> trusted.afr.gv01-client-0=0x000000000000000000000000
>>>> trusted.afr.gv01-client-1=0x000000000000000000000000
>>>> trusted.gfid=0x00000000000000000000000000000001
>>>> trusted.glusterfs.dht=0x000000010000000000000000ffffffff
>>>> trusted.glusterfs.volume-id=0x31a2c4c486ca4344b838d2c2e6c716c1
>>>>
>>>>
>>>>
>>>> On Tue, Sep 9, 2014 at 8:19 AM, Pranith Kumar Karampuri <
>>>> pkarampu at redhat.com> wrote:
>>>>
>>>>>
>>>>> On 09/09/2014 11:35 AM, Ilya Ivanov wrote:
>>>>>
>>>>>  Ahh, thank you, now I get it. I deleted it on one node and it
>>>>> replicated to another one. Now I get the following output:
>>>>>
>>>>> [root at gluster1 var]# gluster volume heal gv01 info
>>>>> Brick gluster1:/home/gluster/gv01/
>>>>> <gfid:d3def9e1-c6d0-4b7d-a322-b5019305182e>
>>>>> Number of entries: 1
>>>>>
>>>>> Brick gluster2:/home/gluster/gv01/
>>>>> Number of entries: 0
>>>>>
>>>>>  Is it normal? Why the number of entries isn't reset to 0?
>>>>>
>>>>>  If you access the file using ls/stat etc, it will be fixed. But
>>>>> before that could you please post the output of 'getfattr -d -m. -e hex
>>>>> file/path/in/backend/brick' and 'getfattr -d -m. -e hex
>>>>> parent/dir/to/file/path/in/backend/brick'
>>>>>
>>>>> Pranith
>>>>>
>>>>>
>>>>>
>>>>> And why wouldn't the file show up in split-brain before, anyway?
>>>>>
>>>>>  Gfid split-brains are not shown in heal-info-split-brain yet.
>>>>>
>>>>> Pranith
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Sep 9, 2014 at 7:46 AM, Pranith Kumar Karampuri <
>>>>> pkarampu at redhat.com> wrote:
>>>>>
>>>>>>
>>>>>> On 09/09/2014 01:54 AM, Ilya Ivanov wrote:
>>>>>>
>>>>>>  Hello.
>>>>>>
>>>>>>  I've Gluster 3.5.2 on Centos 6. A primitive replicated volume, as
>>>>>> describe here
>>>>>> <https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers>.
>>>>>> I tried to simulate split-brain by temporarily disconnecting the nodes and
>>>>>> creating a file with the same name and different contents. That worked.
>>>>>>
>>>>>>  The question is, how do I fix it now? All the tutorials suggest
>>>>>> deleting the file from one of the nodes. I can't do that, it reports
>>>>>> "Input/output error". The file won't even show up in "gluster volume heal
>>>>>> gv00 info split-brain". That shows 0 entries.
>>>>>>
>>>>>>  The deletion needs to happen on one of the bricks, not from the
>>>>>> mount point.
>>>>>>
>>>>>> Pranith
>>>>>>
>>>>>>  I can see the file in "gluster volume heal gv00 info heal-failed",
>>>>>> though.
>>>>>>
>>>>>>
>>>>>>  --
>>>>>> Ilya.
>>>>>>
>>>>>>
>>>>>>  _______________________________________________
>>>>>> Gluster-users mailing listGluster-users at gluster.orghttp://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Ilya.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>  --
>>>> Ilya.
>>>>
>>>
>>>
>>>
>>> --
>>> Ilya.
>>>
>>>
>>>
>>
>>
>> --
>> Ilya.
>>
>>
>>
>
>
> --
> Ilya.
>
>
>

-- 
Ilya.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140911/eb661eda/attachment.html>