<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<br>
<div class="moz-cite-prefix">On 07/31/2017 02:00 PM, mabi wrote:<br>
</div>
<blockquote type="cite"
cite="mid:D9H_eIT2CaaT0nckDfm5ua-lWL_FaT23figxYiYFyOpf4OzG5KogksVZ5nKtOojN26Y-XEvG7qjvIV2t7AE2XGDHVqbxpCutsTq40zRd5yY=@protonmail.ch">
<div>To quickly resume my current situation:<br>
</div>
<div><br>
</div>
<div>on node2 I have found the following file xattrop/indices file
which matches the GFID of the "heal info" command (below is
there output of "ls -lai":<br>
</div>
<div><br>
</div>
<div>2798404 ---------- 2 root root 0 Apr 28 22:51
/data/myvolume/brick/.glusterfs/indices/xattrop/29e0d13e-1217-41cc-9bda-1fbbf781c397<br>
</div>
<div class="protonmail_signature_block
protonmail_signature_block-empty">
<div class="protonmail_signature_block-user
protonmail_signature_block-empty">
<div><br>
</div>
</div>
<div class="protonmail_signature_block-proton
protonmail_signature_block-empty"><br>
</div>
</div>
<div><br>
</div>
<div>As you can see this file has inode number 2798404, so I ran
the following command on all my nodes (node1, node2 and
arbiternode):<br>
</div>
</blockquote>
<br>
<br>
...which is what I was saying is incorrect. 2798404 is an XFS inode
number and is not common to the same file across nodes. So you will
get different results. Use the -samefile flag I shared earlier.<br>
-Ravi<br>
<br>
<blockquote type="cite"
cite="mid:D9H_eIT2CaaT0nckDfm5ua-lWL_FaT23figxYiYFyOpf4OzG5KogksVZ5nKtOojN26Y-XEvG7qjvIV2t7AE2XGDHVqbxpCutsTq40zRd5yY=@protonmail.ch">
<div><br>
</div>
<div>sudo find /data/myvolume/brick -inum 2798404 -ls <br>
</div>
<div><br>
</div>
<div>Here below are the results for all 3 nodes:<br>
</div>
<div><br>
</div>
<div>node1:<br>
</div>
<div><br>
</div>
<div>2798404 19 -rw-r--r-- 2 www-data www-data 32 Jun 19
17:42
/data/myvolume/brick/.glusterfs/e6/5b/e65b77e2-a4c4-4824-a7bb-58df969ce4b0<br>
</div>
<div>2798404 19 -rw-r--r-- 2 www-data www-data 32 Jun 19
17:42
/data/myvolume/brick/<REMOVED_DIRECTORIES_IN_BETWEEN>/fileKey<br>
</div>
<div><br>
</div>
<div>node2:<br>
</div>
<div><br>
</div>
<div>2798404 1 ---------- 2 root root 0 Apr 28
22:51
/data/myvolume/brick/.glusterfs/indices/xattrop/29e0d13e-1217-41cc-9bda-1fbbf781c397<br>
</div>
<div>2798404 1 ---------- 2 root root 0 Apr 28
22:51
/data/myvolume/brick/.glusterfs/indices/xattrop/xattrop-6fa49ad5-71dd-4ec2-9246-7b302ab92d38<br>
</div>
<div><br>
</div>
<div>arbirternode:<br>
</div>
<div><br>
</div>
<div>NOTHING<br>
</div>
<div><br>
</div>
<div>As you requested I have tried to run on node1 a getfattr on
the fileKey file by using the following command:<br>
</div>
<div><br>
</div>
<div>getfattr -m . -d -e hex fileKey<br>
</div>
<div><br>
</div>
<div>but there is no output. I am not familiar with the getfattr
command so maybe I am using the wrong parameters, could you help
me with that?<br>
</div>
<div><br>
</div>
<div><br>
</div>
<blockquote type="cite" class="protonmail_quote">
<div>-------- Original Message --------<br>
</div>
<div>Subject: Re: [Gluster-users] Possible stale
.glusterfs/indices/xattrop file?<br>
</div>
<div>Local Time: July 31, 2017 9:25 AM<br>
</div>
<div>UTC Time: July 31, 2017 7:25 AM<br>
</div>
<div>From: <a class="moz-txt-link-abbreviated" href="mailto:ravishankar@redhat.com">ravishankar@redhat.com</a><br>
</div>
<div>To: mabi <a class="moz-txt-link-rfc2396E" href="mailto:mabi@protonmail.ch"><mabi@protonmail.ch></a><br>
</div>
<div>Gluster Users <a class="moz-txt-link-rfc2396E" href="mailto:gluster-users@gluster.org"><gluster-users@gluster.org></a><br>
</div>
<div><br>
</div>
<div> On 07/31/2017 12:20 PM, mabi wrote:<br>
</div>
<div> <br>
</div>
<blockquote type="cite">
<div>I did a find on this inode number and I could find the
file but only on node1 (nothing on node2 and the new
arbiternode). Here is an ls -lai of the file itself on
node1:<br>
</div>
</blockquote>
<div>Sorry I don't understand, isn't that (XFS) inode number
specific to node2's brick? If you want to use the same
command, maybe you should try `find /data/myvolume/brick
-samefile
/data/myvolume/brick/.glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397`
on all 3 bricks.<br>
</div>
<div> <br>
</div>
<blockquote type="cite">
<div><br>
</div>
<div>-rw-r--r-- 1 www-data www-data 32 Jun 19 17:42 fileKey<br>
</div>
<div><br>
</div>
<div>As you can see it is a 32 bytes file and as you suggested
I ran a "stat" on this very same file through a glusterfs
mount (using fuse) but unfortunately nothing happened. The
GFID is still being displayed to be healed. Just in case
here is the output of the stat:<br>
</div>
<div><br>
</div>
<div> File: ‘fileKey’<br>
</div>
<div> Size: 32 Blocks: 1 IO Block: 131072
regular file<br>
</div>
<div>Device: 1eh/30d Inode: 12086351742306673840 Links: 1<br>
</div>
<div>Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid:
( 33/www-data)<br>
</div>
<div>Access: 2017-06-19 17:42:35.339773495 +0200<br>
</div>
<div>Modify: 2017-06-19 17:42:35.343773437 +0200<br>
</div>
<div>Change: 2017-06-19 17:42:35.343773437 +0200<br>
</div>
<div>Birth: -<br>
</div>
<div><br>
</div>
</blockquote>
<div>Is this 'fileKey' on node1 having the same gfid (see
getfattr output)? Looks like it is missing the hardlink inside
.glusterfs folder since the link count is only 1.<br>
</div>
<div> Thanks,<br>
</div>
<div> Ravi<br>
</div>
<div> <br>
</div>
<blockquote type="cite">
<div>What else can I do or try in order to fix this situation?<br>
</div>
<div><br>
</div>
<div class="protonmail_signature_block
protonmail_signature_block-empty">
<div class="protonmail_signature_block-user
protonmail_signature_block-empty">
<div><br>
</div>
</div>
<div class="protonmail_signature_block-proton
protonmail_signature_block-empty"><br>
</div>
</div>
<div><br>
</div>
<blockquote class="protonmail_quote" type="cite">
<div>-------- Original Message --------<br>
</div>
<div>Subject: Re: [Gluster-users] Possible stale
.glusterfs/indices/xattrop file?<br>
</div>
<div>Local Time: July 31, 2017 3:27 AM<br>
</div>
<div>UTC Time: July 31, 2017 1:27 AM<br>
</div>
<div>From: <a rel="noreferrer nofollow noopener"
class="moz-txt-link-abbreviated"
href="mailto:ravishankar@redhat.com"
moz-do-not-send="true">ravishankar@redhat.com</a><br>
</div>
<div>To: mabi <a rel="noreferrer nofollow noopener"
class="moz-txt-link-rfc2396E"
href="mailto:mabi@protonmail.ch" moz-do-not-send="true"><mabi@protonmail.ch></a><br>
</div>
<div>Gluster Users <a rel="noreferrer nofollow noopener"
class="moz-txt-link-rfc2396E"
href="mailto:gluster-users@gluster.org"
moz-do-not-send="true"><gluster-users@gluster.org></a><br>
</div>
<div><br>
</div>
<div><br>
</div>
<p><br>
</p>
<div><br>
</div>
<div class="moz-cite-prefix">On 07/30/2017 02:24 PM, mabi
wrote:<br>
</div>
<blockquote type="cite">
<div>Hi Ravi,<br>
</div>
<div><br>
</div>
<div>Thanks for your hints. Below you will find the answer
to your questions.<br>
</div>
<div><br>
</div>
<div>First I tried to start the healing process by
running:<br>
</div>
<div><br>
</div>
<div>gluster volume heal myvolume<br>
</div>
<div><br>
</div>
<div>and then as you suggested watch the output of the
glustershd.log file but nothing appeared in that log
file after running the above command. I checked the
files which need to be healing using the "heal
<volume> info" command and it still shows that
very same GFID on node2 to be healed. So nothing changed
here.<br>
</div>
<div><br>
</div>
<div>The file
/data/myvolume/brick/.glusterfs/indices/xattrop/29e0d13e-1217-41cc-9bda-1fbbf781c397
is only on node2 and not on my nod1 nor on my
arbiternode. This file seems to be a regular file and
not a symlink. Here is the output of the stat command on
it from my node2:<br>
</div>
<div><br>
</div>
<div> File:
‘/data/myvolume/brick/.glusterfs/indices/xattrop/29e0d13e-1217-41cc-9bda-1fbbf781c397’<br>
</div>
<div> Size: 0 Blocks: 1 IO Block: 512
regular empty file<br>
</div>
<div>Device: 25h/37d Inode: 2798404 Links: 2<br>
</div>
</blockquote>
<div><br>
</div>
<div>Okay, link count of 2 means there is a hardlink
somewhere on the brick. Try the find command again. I see
that the inode number is 2798404, not the one you shared
in your first mail. Once you find the path to the file, do
a stat of the file from mount. This should create the
entry in the other 2 bricks and do the heal. But FWIW,
this seems to be a zero byte file.<br>
</div>
<div> <br>
</div>
<div>Regards,<br>
</div>
<div>Ravi<br>
</div>
<div><br>
</div>
<blockquote type="cite">
<div>Access: (0000/----------) Uid: ( 0/ root)
Gid: ( 0/ root)<br>
</div>
<div>Access: 2017-04-28 22:51:15.215775269 +0200<br>
</div>
<div>Modify: 2017-04-28 22:51:15.215775269 +0200<br>
</div>
<div>Change: 2017-07-30 08:39:03.700872312 +0200<br>
</div>
<div>Birth: -<br>
</div>
<div><br>
</div>
<div>I hope this is enough info for a starter, else let me
know if you need any more info. I would be glad to
resolve this weird file which needs to be healed but can
not.<br>
</div>
<div><br>
</div>
<div>Best regards,<br>
</div>
<div>Mabi<br>
</div>
<div class="protonmail_signature_block
protonmail_signature_block-empty">
<div class="protonmail_signature_block-user
protonmail_signature_block-empty">
<div><br>
</div>
</div>
<div class="protonmail_signature_block-proton
protonmail_signature_block-empty"><br>
</div>
</div>
<div><br>
</div>
<blockquote type="cite" class="protonmail_quote">
<div>-------- Original Message --------<br>
</div>
<div>Subject: Re: [Gluster-users] Possible stale
.glusterfs/indices/xattrop file?<br>
</div>
<div>Local Time: July 30, 2017 3:31 AM<br>
</div>
<div>UTC Time: July 30, 2017 1:31 AM<br>
</div>
<div>From: <a href="mailto:ravishankar@redhat.com"
class="moz-txt-link-abbreviated" rel="noreferrer
nofollow noopener" moz-do-not-send="true">ravishankar@redhat.com</a><br>
</div>
<div>To: mabi <a href="mailto:mabi@protonmail.ch"
class="moz-txt-link-rfc2396E" rel="noreferrer
nofollow noopener" moz-do-not-send="true"><mabi@protonmail.ch></a>,
Gluster Users <a
href="mailto:gluster-users@gluster.org"
class="moz-txt-link-rfc2396E" rel="noreferrer
nofollow noopener" moz-do-not-send="true"><gluster-users@gluster.org></a><br>
</div>
<div><br>
</div>
<div><br>
</div>
<p><br>
</p>
<div><br>
</div>
<div class="moz-cite-prefix">On 07/29/2017 04:36 PM,
mabi wrote:<br>
</div>
<blockquote type="cite">
<div>Hi,<br>
</div>
<div><br>
</div>
<div>Sorry for mailing again but as mentioned in my
previous mail, I have added an arbiter node to my
replica 2 volume and it seem to have gone fine
except for the fact that there is one single file
which needs healing and does not get healed as you
can see here from the output of a "heal info":<br>
</div>
<div><br>
</div>
<div>Brick node1.domain.tld:/data/myvolume/brick<br>
</div>
<div>Status: Connected<br>
</div>
<div>Number of entries: 0<br>
</div>
<div><br>
</div>
<div>Brick node2.domain.tld:/data/myvolume/brick<br>
</div>
<div><gfid:29e0d13e-1217-41cc-9bda-1fbbf781c397><br>
</div>
<div>Status: Connected<br>
</div>
<div>Number of entries: 1<br>
</div>
<div><br>
</div>
<div>Brick
arbiternode.domain.tld:/srv/glusterfs/myvolume/brick<br>
</div>
<div>Status: Connected<br>
</div>
<div>Number of entries: 0<br>
</div>
<div><br>
</div>
<div>On my node2 the respective
.glusterfs/indices/xattrop directory contains two
files as you can see below:<br>
</div>
<div><br>
</div>
<div>ls -lai
/data/myvolume/brick/.glusterfs/indices/xattrop<br>
</div>
<div>total 76180<br>
</div>
<div> 10 drw------- 2 root root 4 Jul 29 12:15 .<br>
</div>
<div> 9 drw------- 5 root root 5 Apr 28 22:15 ..<br>
</div>
<div>2798404 ---------- 2 root root 0 Apr 28 22:51
29e0d13e-1217-41cc-9bda-1fbbf781c397<br>
</div>
<div>2798404 ---------- 2 root root 0 Apr 28 22:51
xattrop-6fa49ad5-71dd-4ec2-9246-7b302ab92d38<br>
</div>
<div class="protonmail_signature_block
protonmail_signature_block-empty">
<div class="protonmail_signature_block-user
protonmail_signature_block-empty">
<div><br>
</div>
</div>
<div class="protonmail_signature_block-proton
protonmail_signature_block-empty"><br>
</div>
</div>
<div><br>
</div>
<div>I tried to find the real file on my brick where
this xattrop file points to using its inode number
(command: find /data/myvolume/brick/data -inum
8394642) but it does not find any associated file.<br>
</div>
<div><br>
</div>
<div>So my question here is, is it possible that this
is a stale file which just forgot to get deleted
from the indices/xattrop file by gluster for some
unknown reason? If yes is it safe for me to delete
these two files? or what would be the correct
process in that case?<br>
</div>
</blockquote>
<div>The 'xattrop-6fa...' is the base entry. gfids of
files that need heal are hard linked to this entry, so
nothing needs to be done for it. But you need to find
out why '29e0d13...' is not healing. Launch the heal
and observe the glustershd logs for errors. I suppose
the inode number for
.glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397
is what is 8394642. Is
.glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397
a regular file or symlink? Does it exist in the other
2 bricks? What is the link count (as seen from stat
<file>)?<br>
</div>
<div>-Ravi<br>
</div>
<div><br>
</div>
<blockquote type="cite">
<div><br>
</div>
<div>Thank you for your input.<br>
</div>
<div>Mabi<br>
</div>
<div><br>
</div>
<div><br>
</div>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a rel="noreferrer nofollow noopener" class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" moz-do-not-send="true">Gluster-users@gluster.org</a>
<a rel="noreferrer nofollow noopener" class="moz-txt-link-freetext" href="http://lists.gluster.org/mailman/listinfo/gluster-users" moz-do-not-send="true">http://lists.gluster.org/mailman/listinfo/gluster-users</a>
</pre>
</blockquote>
</blockquote>
<div><br>
</div>
</blockquote>
</blockquote>
<div><br>
</div>
</blockquote>
</blockquote>
<div><br>
</div>
</blockquote>
<br>
</body>
</html>