[Gluster-users] Possible stale .glusterfs/indices/xattrop file?

Ravishankar N ravishankar at redhat.com
Mon Jul 31 07:25:04 UTC 2017 

On 07/31/2017 12:20 PM, mabi wrote:
> I did a find on this inode number and I could find the file but only 
> on node1 (nothing on node2 and the new arbiternode). Here is an ls 
> -lai of the file itself on node1:
Sorry I don't understand, isn't that (XFS) inode number specific to 
node2's brick? If you want to use the same command, maybe you should try 
`find /data/myvolume/brick -samefile 
/data/myvolume/brick/.glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397` 
on all 3 bricks.
>
> -rw-r--r-- 1 www-data www-data   32 Jun 19 17:42 fileKey
>
> As you can see it is a 32 bytes file and as you suggested I ran a 
> "stat" on this very same file through a glusterfs mount (using fuse) 
> but unfortunately nothing happened. The GFID is still being displayed 
> to be healed.  Just in case here is the output of the stat:
>
>   File: ‘fileKey’
>   Size: 32        Blocks: 1          IO Block: 131072 regular file
> Device: 1eh/30d Inode: 12086351742306673840  Links: 1
> Access: (0644/-rw-r--r--)  Uid: (   33/www-data)   Gid: ( 33/www-data)
> Access: 2017-06-19 17:42:35.339773495 +0200
> Modify: 2017-06-19 17:42:35.343773437 +0200
> Change: 2017-06-19 17:42:35.343773437 +0200
> Birth: -
>
Is this 'fileKey' on node1 having the same gfid (see getfattr output)? 
Looks like it is missing the hardlink inside .glusterfs folder since the 
link count is only 1.
Thanks,
Ravi
> What else can I do or try in order to fix this situation?
>
>
>
>
>> -------- Original Message --------
>> Subject: Re: [Gluster-users] Possible stale 
>> .glusterfs/indices/xattrop file?
>> Local Time: July 31, 2017 3:27 AM
>> UTC Time: July 31, 2017 1:27 AM
>> From: ravishankar at redhat.com
>> To: mabi <mabi at protonmail.ch>
>> Gluster Users <gluster-users at gluster.org>
>>
>>
>>
>>
>> On 07/30/2017 02:24 PM, mabi wrote:
>>> Hi Ravi,
>>>
>>> Thanks for your hints. Below you will find the answer to your questions.
>>>
>>> First I tried to start the healing process by running:
>>>
>>> gluster volume heal myvolume
>>>
>>> and then as you suggested watch the output of the glustershd.log 
>>> file but nothing appeared in that log file after running the above 
>>> command. I checked the files which need to be healing using the 
>>> "heal <volume> info" command and it still shows that very same GFID 
>>> on node2 to be healed. So nothing changed here.
>>>
>>> The file 
>>> /data/myvolume/brick/.glusterfs/indices/xattrop/29e0d13e-1217-41cc-9bda-1fbbf781c397 
>>> is only on node2 and not on my nod1 nor on my arbiternode. This file 
>>> seems to be a regular file and not a symlink. Here is the output of 
>>> the stat command on it from my node2:
>>>
>>>   File: 
>>> ‘/data/myvolume/brick/.glusterfs/indices/xattrop/29e0d13e-1217-41cc-9bda-1fbbf781c397’
>>>   Size: 0         Blocks: 1          IO Block: 512 regular empty file
>>> Device: 25h/37d Inode: 2798404     Links: 2
>>
>> Okay, link count of 2 means there is a hardlink somewhere on the 
>> brick. Try the find command again. I see that the inode number is 
>> 2798404, not the one you shared in your first mail. Once you find the 
>> path to the file, do a stat of the file from mount. This should 
>> create the entry in the other 2 bricks and do the heal. But FWIW, 
>> this seems to be a zero byte file.
>>
>> Regards,
>> Ravi
>>
>>> Access: (0000/----------)  Uid: (    0/    root)   Gid: (    0/    root)
>>> Access: 2017-04-28 22:51:15.215775269 +0200
>>> Modify: 2017-04-28 22:51:15.215775269 +0200
>>> Change: 2017-07-30 08:39:03.700872312 +0200
>>> Birth: -
>>>
>>> I hope this is enough info for a starter, else let me know if you 
>>> need any more info. I would be glad to resolve this weird file which 
>>> needs to be healed but can not.
>>>
>>> Best regards,
>>> Mabi
>>>
>>>
>>>
>>>> -------- Original Message --------
>>>> Subject: Re: [Gluster-users] Possible stale 
>>>> .glusterfs/indices/xattrop file?
>>>> Local Time: July 30, 2017 3:31 AM
>>>> UTC Time: July 30, 2017 1:31 AM
>>>> From: ravishankar at redhat.com
>>>> To: mabi <mabi at protonmail.ch>, Gluster Users 
>>>> <gluster-users at gluster.org>
>>>>
>>>>
>>>>
>>>>
>>>> On 07/29/2017 04:36 PM, mabi wrote:
>>>>> Hi,
>>>>>
>>>>> Sorry for mailing again but as mentioned in my previous mail, I 
>>>>> have added an arbiter node to my replica 2 volume and it seem to 
>>>>> have gone fine except for the fact that there is one single file 
>>>>> which needs healing and does not get healed as you can see here 
>>>>> from the output of a "heal info":
>>>>>
>>>>> Brick node1.domain.tld:/data/myvolume/brick
>>>>> Status: Connected
>>>>> Number of entries: 0
>>>>>
>>>>> Brick node2.domain.tld:/data/myvolume/brick
>>>>> <gfid:29e0d13e-1217-41cc-9bda-1fbbf781c397>
>>>>> Status: Connected
>>>>> Number of entries: 1
>>>>>
>>>>> Brick arbiternode.domain.tld:/srv/glusterfs/myvolume/brick
>>>>> Status: Connected
>>>>> Number of entries: 0
>>>>>
>>>>> On my node2 the respective .glusterfs/indices/xattrop directory 
>>>>> contains two files as you can see below:
>>>>>
>>>>> ls -lai /data/myvolume/brick/.glusterfs/indices/xattrop
>>>>> total 76180
>>>>>      10 drw------- 2 root root 4 Jul 29 12:15 .
>>>>>       9 drw------- 5 root root 5 Apr 28 22:15 ..
>>>>> 2798404 ---------- 2 root root 0 Apr 28 22:51 
>>>>> 29e0d13e-1217-41cc-9bda-1fbbf781c397
>>>>> 2798404 ---------- 2 root root 0 Apr 28 22:51 
>>>>> xattrop-6fa49ad5-71dd-4ec2-9246-7b302ab92d38
>>>>>
>>>>>
>>>>>
>>>>> I tried to find the real file on my brick where this xattrop file 
>>>>> points to using its inode number (command: find 
>>>>> /data/myvolume/brick/data -inum 8394642) but it does not find any 
>>>>> associated file.
>>>>>
>>>>> So my question here is, is it possible that this is a stale file 
>>>>> which just forgot to get deleted from the indices/xattrop file by 
>>>>> gluster for some unknown reason? If yes is it safe for me to 
>>>>> delete these two files? or what would be the correct process in 
>>>>> that case?
>>>> The 'xattrop-6fa...' is the base entry. gfids of files that need 
>>>> heal are hard linked to this entry, so nothing needs to be done for 
>>>> it. But you need to find out why '29e0d13...' is not healing. 
>>>> Launch the heal and observe the glustershd logs for errors. I 
>>>> suppose the inode number for 
>>>> .glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397 is what is 
>>>> 8394642.  Is .glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397 
>>>> a regular file or  symlink? Does it exist in the other 2 bricks? 
>>>> What is the link count (as seen from stat <file>)?
>>>> -Ravi
>>>>
>>>>>
>>>>> Thank you for your input.
>>>>> Mabi
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170731/bb3ecd4d/attachment.html>

More information about the Gluster-users mailing list