[Gluster-users] How to clear split brain info

[Ravishankar N]

On 09/21/2015 05:08 PM, Davy Croonen wrote:
>> On 21 Sep 2015, at 12:44, Ravishankar N <ravishankar at redhat.com> wrote:
>>
>> s
>>
>> On 09/21/2015 03:48 PM, Davy Croonen wrote:
>>> Hmmm, strange, I went through all my bricks with every time the same result:
>>>
>>> -bash: cd: /mnt/public/brick1/.glusterfs/31/38/3138d605-25ec-4aa9-9069-5db2e4202db4: No such file or directory
>>>
>>> The directory /mnt/public/brick1/.glusterfs/31/38 does exist, and indeed there’s a symlink in there but not one with the referenced gfid.
>> That is strange..Does the same string ( "<gfid:3138d605-25ec-4aa9-9069-5db2e4202db4>/Doc1_LOUJA.htm - Is in split-brain.") appear on both bricks of the replica (assuming replica 2) when you run the heal info command?
>>
> We have indeed a replica 2 and yes exact the same gfid string shows up on both bricks.
>
>>> Any further suggestions? If I can get rid of the message it’s ok.
>> You could stat all the files on the mount (find /<mount-point> |xargs stat > /dev/null) and then run the heal info command again and see if you now get the absolute path instead of the gfid string.
>>
> I ran the stat command against every file on the volume but after that nothing was changed when running gluster volume heal public info.

I should have asked you earlier, but can you share the 
/var/log/glusterfs/glfsheal-<volname>.log on the node on which you are 
running the heal info command?

>
>>> Thanks in advance.
>>>
>>> Kind regards
>>> Davy
>>>
>>>> On 21 Sep 2015, at 11:59, Ravishankar N <ravishankar at redhat.com> wrote:
>>>>
>>>>
>>>>
>>>> On 09/21/2015 03:09 PM, Davy Croonen wrote:
>>>>> Ravi
>>>>>
>>>>> Thanks for your quick reply.
>>>>>
>>>>> I didn’t solve the split-brain on the file because I don’t know to which directory this gfid is referencing (due to the implementation of our application we have multiple directories containing the same files).
>>>> <gfid:3138d605-25ec-4aa9-9069-5db2e4202db4>/Doc1_LOUJA.htm - Is in split-brain.
>>>>
>>>> Directories are symlinked inside the .glusterfs folder on the bricks. You can `cd /mnt/public/brick1/.glusterfs/31/38/3138d605-25ec-4aa9-9069-5db2e4202db4` and get to the file 'Doc1_LOUJA.htm' and then resolve the split-brain.
>>>>
>>>> HTH,
>>>> Ravi
>>>>> Running the command “getfattr -m . -d -e hex <directory>” against every directory which could be the one didn’t show up the mentioned gfid. But there was one result that didn’t fit the picture, see results below:
>>>>>
>>>>> root at gfs01a-dcg:/mnt/public/brick1/provil/Timetable# getfattr -m . -d -e hex h9cnBmEx6j26sgidVKLaZhAqh
>>>>> # file: h9cnBmEx6j26sgidVKLaZhAqh
>>>>> trusted.afr.dirty=0x000000000000000000000000
>>>>> trusted.afr.public-client-0=0x000000000000000000000000
>>>>> trusted.afr.public-client-1=0x000000000000000000000000
>>>>> trusted.gfid=0xb560c2463f5b476fa95d318f6bb91356
>>>>> trusted.glusterfs.dht=0x00000001000000007ffe51f8ffffffff
>>>>>
>>>>> root at gfs01a-dcg:/mnt/public/brick1/provil/Timetable# getfattr -m . -d -e hex h9cnBmEx6j26sgidVKLaZhAqh1442579382189160
>>>>> # file: h9cnBmEx6j26sgidVKLaZhAqh1442579382189160
>>>>> trusted.afr.dirty=0x000000000000000000000000
>>>>> trusted.afr.public-client-0=0x000000000000000000000000
>>>>> trusted.afr.public-client-1=0x000000000000000000000000
>>>>> trusted.gfid=0x0367b477d5ea4b62b4da174b33622457
>>>>>
>>>>> On the other hand there is also the possibility that the directory doesn’t even exist anymore because every night there are running some cleanup scripts.
>>>>>
>>>>> By the way we are running gluster version 3.6.4.
>>>>>
>>>>> Davy
>>>>>
>>>>>> On 21 Sep 2015, at 11:16, Ravishankar N <ravishankar at redhat.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 09/21/2015 02:32 PM, Davy Croonen wrote:
>>>>>>> Hi all
>>>>>>>
>>>>>>> For, at the moment a unknown reason, the command "gluster volume heal public info” shows a lot of the following entries:
>>>>>>>
>>>>>>> <gfid:3138d605-25ec-4aa9-9069-5db2e4202db4>/Doc1_LOUJA.htm - Is in split-brain.
>>>>>>>
>>>>>>> The part after the / differs but the gfid is always the same, I suppose this gfid is referring to a directory.
>>>>>> Correct, it is the parent directory of the file in question.
>>>>>>>   Now considering the data this isn’t an issue the files can be or are already deleted, but is there a way to clear these split-brain entries?
>>>>>> Did you actually resolve the split-brain on the file ? If you do that, then the entry must disappear from the output of the heal info command.
>>>>>>> The command for restarting the self heal daemon “gluster v start public force”, didn’t solve the problem. Any other suggestions?
>>>>>> What version of gluster are you running?
>>>>>>
>>>>>> -Ravi
>>>>>>> Thanks in advance.
>>>>>>>
>>>>>>>
>>>>>>> Kind regards
>>>>>>> Davy
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users