[Gluster-users] ... i was able to produce a split brain...

Ml Ml mliebherr99 at googlemail.com
Thu Jan 29 08:34:28 UTC 2015 

Hm. okay. Then the split-brain seems to have been resolved somehow. thanks.

However a user friendly way of resolving split brains whould be great
(like Jeff suggested).

Thanks a lot so far.

Mario


On Thu, Jan 29, 2015 at 5:19 AM, Ravishankar N <ravishankar at redhat.com> wrote:
>
> On 01/28/2015 10:58 PM, Ml Ml wrote:
>>
>> "/1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids" is a binary file.
>>
>>
>> Here is the output of gluster volume info:
>>
>> --------------------------------------------------------------------------------------
>>
>>
>> [root at ovirt-node03 ~]# gluster volume info
>>
>> Volume Name: RaidVolB
>> Type: Replicate
>> Volume ID: e952fd41-45bf-42d9-b494-8e0195cb9756
>> Status: Started
>> Number of Bricks: 1 x 2 = 2
>> Transport-type: tcp
>> Bricks:
>> Brick1: ovirt-node03.example.local:/raidvol/volb/brick
>> Brick2: ovirt-node04.example.local:/raidvol/volb/brick
>> Options Reconfigured:
>> storage.owner-gid: 36
>> storage.owner-uid: 36
>> network.remote-dio: enable
>> cluster.eager-lock: enable
>> performance.stat-prefetch: off
>> performance.io-cache: off
>> performance.read-ahead: off
>> performance.quick-read: off
>> auth.allow: *
>> user.cifs: disable
>> nfs.disable: on
>>
>>
>>
>>
>> [root at ovirt-node04 ~]#  gluster volume info
>>
>> Volume Name: RaidVolB
>> Type: Replicate
>> Volume ID: e952fd41-45bf-42d9-b494-8e0195cb9756
>> Status: Started
>> Number of Bricks: 1 x 2 = 2
>> Transport-type: tcp
>> Bricks:
>> Brick1: ovirt-node03.example.local:/raidvol/volb/brick
>> Brick2: ovirt-node04.example.local:/raidvol/volb/brick
>> Options Reconfigured:
>> nfs.disable: on
>> user.cifs: disable
>> auth.allow: *
>> performance.quick-read: off
>> performance.read-ahead: off
>> performance.io-cache: off
>> performance.stat-prefetch: off
>> cluster.eager-lock: enable
>> network.remote-dio: enable
>> storage.owner-uid: 36
>> storage.owner-gid: 36
>>
>>
>> Here is the getfattr command in node03 and node 04:
>>
>> --------------------------------------------------------------------------------------
>>
>>
>>   getfattr -d -m . -e hex
>> /raidvol/volb/brick//1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids
>> getfattr: Removing leading '/' from absolute path names
>> # file:
>> raidvol/volb/brick//1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids
>> trusted.afr.RaidVolB-client-0=0x000000000000000000000000
>> trusted.afr.RaidVolB-client-1=0x000000000000000000000000
>> trusted.afr.dirty=0x000000000000000000000000
>> trusted.gfid=0x1c15d0cb1cca4627841c395f7b712f73
>>
>>
>>
>> [root at ovirt-node04 ~]# getfattr -d -m . -e hex
>> /raidvol/volb/brick//1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids
>> getfattr: Removing leading '/' from absolute path names
>> # file:
>> raidvol/volb/brick//1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids
>> trusted.afr.RaidVolB-client-0=0x000000000000000000000000
>> trusted.afr.RaidVolB-client-1=0x000000000000000000000000
>> trusted.afr.dirty=0x000000000000000000000000
>> trusted.gfid=0x1c15d0cb1cca4627841c395f7b712f73
>>
>
> These xattrs seem to indicate there is no split-brain for the file,
> heal-info also shows 0 entries on both bricks.
> Are you getting I/O error when you read
> "1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids" from the mount?
> If yes, is there a difference in file size on both nodes? How about the
> contents (check if md5sum is same)?
>
>> Am i supposed to run those commands on the mounted brick?:
>>
>> --------------------------------------------------------------------------------------
>> 127.0.0.1:RaidVolB on
>> /rhev/data-center/mnt/glusterSD/127.0.0.1:RaidVolB type fuse.glusterfs
>> (rw,default_permissions,allow_other,max_read=131072)
>>
>>
>> At the very beginning i thought i removed the file with "rm
>> /raidvol/volb/brick//1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids"
>> hoping gluster would then fix itself somehow :)
>> It was gone but it seems to be here again. Dunno if this is any help.
>>
>>
>> Here is gluster volume heal RaidVolB info on both nodes:
>>
>> --------------------------------------------------------------------------------------
>>
>> [root at ovirt-node03 ~]#  gluster volume heal RaidVolB info
>> Brick ovirt-node03.example.local:/raidvol/volb/brick/
>> Number of entries: 0
>>
>> Brick ovirt-node04.example.local:/raidvol/volb/brick/
>> Number of entries: 0
>>
>>
>> [root at ovirt-node04 ~]#  gluster volume heal RaidVolB info
>> Brick ovirt-node03.example.local:/raidvol/volb/brick/
>> Number of entries: 0
>>
>> Brick ovirt-node04.example.local:/raidvol/volb/brick/
>> Number of entries: 0
>>
>>
>> Thanks a lot,
>> Mario
>>
>>
>>
>> On Wed, Jan 28, 2015 at 4:57 PM, Ravishankar N <ravishankar at redhat.com>
>> wrote:
>>>
>>> On 01/28/2015 08:34 PM, Ml Ml wrote:
>>>>
>>>> Hello Ravi,
>>>>
>>>> thanks a lot for your reply.
>>>>
>>>> The Data on ovirt-node03 is the one which i want.
>>>>
>>>> Here are the infos collected by following the howto:
>>>>
>>>>
>>>> https://github.com/GlusterFS/glusterfs/blob/master/doc/debugging/split-brain.md
>>>>
>>>>
>>>>
>>>> [root at ovirt-node03 ~]# gluster volume heal RaidVolB info split-brain
>>>> Gathering list of split brain entries on volume RaidVolB has been
>>>> successful
>>>>
>>>> Brick ovirt-node03.example.local:/raidvol/volb/brick
>>>> Number of entries: 0
>>>>
>>>> Brick ovirt-node04.example.local:/raidvol/volb/brick
>>>> Number of entries: 14
>>>> at                    path on brick
>>>> -----------------------------------
>>>> 2015-01-27 17:33:00 <gfid:1c15d0cb-1cca-4627-841c-395f7b712f73>
>>>> 2015-01-27 17:34:01 <gfid:1c15d0cb-1cca-4627-841c-395f7b712f73>
>>>> 2015-01-27 17:35:04 <gfid:1c15d0cb-1cca-4627-841c-395f7b712f73>
>>>> 2015-01-27 17:36:05 <gfid:cd411b57-6078-4f3c-80d1-0ac1455186a6>/ids
>>>> 2015-01-27 17:37:06 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids
>>>> 2015-01-27 17:37:07 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids
>>>> 2015-01-27 17:38:08 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids
>>>> 2015-01-27 17:38:21 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids
>>>> 2015-01-27 17:39:22 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids
>>>> 2015-01-27 17:40:23 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids
>>>> 2015-01-27 17:41:24 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids
>>>> 2015-01-27 17:42:25 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids
>>>> 2015-01-27 17:43:26 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids
>>>> 2015-01-27 17:44:27 /1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids
>>>>
>>>> [root at ovirt-node03 ~]# gluster volume heal RaidVolB info
>>>> Brick ovirt-node03.example.local:/raidvol/volb/brick/
>>>> Number of entries: 0
>>>>
>>>> Brick ovirt-node04.example.local:/raidvol/volb/brick/
>>>> Number of entries: 0
>>>
>>>
>>> Hi Mario,
>>> Is "/1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids" a file or a
>>> directory?
>>> Whatever it is, it should be shown in the output of heal info /heal info
>>> split-brain command of both nodes. But I see it being listed only under
>>> node03.
>>> Also, heal info is showing zero entries for both nodes which is strange.
>>>
>>> Are node03 and node04 bricks of the same replica pair? Can you share
>>> 'gluster volume info` of RaidVolB?
>>> How did you infer that there is a split-brain? Does accessing the file(s)
>>> from the mount give input/output error?
>>>
>>>> [root at ovirt-node03 ~]# getfattr -d -m . -e hex
>>>> /raidvol/volb/brick/1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids
>>>> getfattr: Removing leading '/' from absolute path names
>>>> # file:
>>>> raidvol/volb/brick/1701d5ae-6a44-4374-8b29-61c699da870b/dom_md/ids
>>>> trusted.afr.RaidVolB-client-0=0x000000000000000000000000
>>>> trusted.afr.RaidVolB-client-1=0x000000000000000000000000
>>>> trusted.afr.dirty=0x000000000000000000000000
>>>> trusted.gfid=0x1c15d0cb1cca4627841c395f7b712f73
>>>
>>> What is the getfattr output of this file on the other brick? The afr
>>> specific xattrs being all zeros certainly don't indicate the possibility
>>> of
>>> a split-brain
>>>
>>>> The "Resetting the relevant changelogs to resolve the split-brain: "
>>>> part of the howto is now a little complictaed. Do i have a data or
>>>> meta split brain now?
>>>> I guess i have a data split brain in my case, right?
>>>>
>>>> What are my next setfattr commands nowin my case if i want to keep the
>>>> data from node03?
>>>>
>>>> Thanks a lot!
>>>>
>>>> Mario
>>>>
>>>>
>>>> On Wed, Jan 28, 2015 at 9:44 AM, Ravishankar N <ravishankar at redhat.com>
>>>> wrote:
>>>>>
>>>>> On 01/28/2015 02:02 PM, Ml Ml wrote:
>>>>>>
>>>>>> I want to either take the file from node03 or node04. i really don’t
>>>>>>>
>>>>>>> mind. Can i not just tell gluster that it should use one node as the
>>>>>>> „current“ one?
>>>>>
>>>>> Policy based split-brain resolution [1] which does just that, has been
>>>>> merged in master and should be available in glusterfs 3.7.
>>>>> For the moment, you would have to modify the xattrs on the one of the
>>>>> bricks
>>>>> and trigger heal. You can see
>>>>>
>>>>>
>>>>> https://github.com/GlusterFS/glusterfs/blob/master/doc/debugging/split-brain.md
>>>>> on how to do it.
>>>>>
>>>>> Hope this helps,
>>>>> Ravi
>>>>>
>>>>> [1] http://review.gluster.org/#/c/9377/
>>>>>
>

More information about the Gluster-users mailing list