<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p><br>
    </p>
    <br>
    <div class="moz-cite-prefix">On 07/31/2017 02:33 PM, mabi wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:Q3hvjwOnep83vQICaLYZnHGqB_9S545FYY-yU8JCWEQYuEqZnJZNcInyMb3lmeGOW-DoOVYIclg1umyJ6RAICwPyDCbV8ZNrv8H7TqBw5mo=@protonmail.ch">
      <div>Now I understand what you mean the the "-samefile" parameter
        of "find". As requested I have now run the following command on
        all 3 nodes with the ouput of all 3 nodes below:<br>
      </div>
      <div><br>
      </div>
      <div>sudo find /data/myvolume/brick -samefile
/data/myvolume/brick/.glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397
        -ls<br>
      </div>
      <div><br>
      </div>
      <div>node1:<br>
      </div>
      <div>8404683    0 lrwxrwxrwx   1 root     root           66 Jul 27
        15:43
/data/myvolume/brick/.glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397
        -&gt;
        ../../fe/c0/fec0e4f4-38d2-4e2e-b5db-fdc0b9b54810/OC_DEFAULT_MODULE<br>
      </div>
      <div><br>
      </div>
      <div>node2:<br>
      </div>
      <div>8394638    0 lrwxrwxrwx   1 root     root           66 Jul 27
        15:43
/data/myvolume/brick/.glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397
        -&gt;
        ../../fe/c0/fec0e4f4-38d2-4e2e-b5db-fdc0b9b54810/OC_DEFAULT_MODULE<br>
      </div>
      <div class="protonmail_signature_block
        protonmail_signature_block-empty">
        <div class="protonmail_signature_block-user
          protonmail_signature_block-empty">
          <div><br>
          </div>
        </div>
        <div class="protonmail_signature_block-proton
          protonmail_signature_block-empty"><br>
        </div>
      </div>
      <div><br>
      </div>
      <div>arbiternode:<br>
      </div>
      <div>find:
'/data/myvolume/brick/.glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397':
        No such file or directory<br>
      </div>
    </blockquote>
    <br>
    <br>
    Right, so the file OC_DEFAULT_MODULE is missing in this brick It's
    parent directory has gfid fec0e4f4-38d2-4e2e-b5db-fdc0b9b54810. <br>
    Goal is to do a stat of this file from the fuse mount. If you know
    the complete path to this file, good. Otherwise you can use this
    script [1] to find the path to the parent dir corresponding to the
    gfid fec0e4f4-38d2-4e2e-b5db-fdc0b9b54810 like so:<br>
    `./gfid-to-dirname.sh  /data/myvolume/brick
    fec0e4f4-38d2-4e2e-b5db-fdc0b9b54810`<br>
    <br>
    [1]
<a class="moz-txt-link-freetext" href="https://github.com/gluster/glusterfs/blob/master/extras/gfid-to-dirname.sh">https://github.com/gluster/glusterfs/blob/master/extras/gfid-to-dirname.sh</a><br>
    <br>
    Try to stat the file from a new (temporary) fuse mount to avoid any
    caching effects.<br>
    -Ravi<br>
    <br>
    <blockquote type="cite"
cite="mid:Q3hvjwOnep83vQICaLYZnHGqB_9S545FYY-yU8JCWEQYuEqZnJZNcInyMb3lmeGOW-DoOVYIclg1umyJ6RAICwPyDCbV8ZNrv8H7TqBw5mo=@protonmail.ch">
      <div><br>
      </div>
      <div>Hope that helps.<br>
      </div>
      <div><br>
      </div>
      <blockquote class="protonmail_quote" type="cite">
        <div>-------- Original Message --------<br>
        </div>
        <div>Subject: Re: [Gluster-users] Possible stale
          .glusterfs/indices/xattrop file?<br>
        </div>
        <div>Local Time: July 31, 2017 10:55 AM<br>
        </div>
        <div>UTC Time: July 31, 2017 8:55 AM<br>
        </div>
        <div>From: <a class="moz-txt-link-abbreviated" href="mailto:ravishankar@redhat.com">ravishankar@redhat.com</a><br>
        </div>
        <div>To: mabi <a class="moz-txt-link-rfc2396E" href="mailto:mabi@protonmail.ch">&lt;mabi@protonmail.ch&gt;</a><br>
        </div>
        <div>Gluster Users <a class="moz-txt-link-rfc2396E" href="mailto:gluster-users@gluster.org">&lt;gluster-users@gluster.org&gt;</a><br>
        </div>
        <div><br>
        </div>
        <div> <br>
        </div>
        <p><br>
        </p>
        <div><br>
        </div>
        <div class="moz-cite-prefix">On 07/31/2017 02:00 PM, mabi wrote:<br>
        </div>
        <blockquote type="cite">
          <div>To quickly resume my current situation:<br>
          </div>
          <div><br>
          </div>
          <div>on node2 I have found the following file xattrop/indices
            file which matches the GFID of the "heal info" command
            (below is there output of "ls -lai":<br>
          </div>
          <div><br>
          </div>
          <div>2798404 ---------- 2 root root 0 Apr 28 22:51
/data/myvolume/brick/.glusterfs/indices/xattrop/29e0d13e-1217-41cc-9bda-1fbbf781c397<br>
          </div>
          <div class="protonmail_signature_block
            protonmail_signature_block-empty">
            <div class="protonmail_signature_block-user
              protonmail_signature_block-empty">
              <div><br>
              </div>
            </div>
            <div class="protonmail_signature_block-proton
              protonmail_signature_block-empty"><br>
            </div>
          </div>
          <div><br>
          </div>
          <div>As you can see this file has inode number 2798404, so I
            ran the following command on all my nodes (node1, node2 and
            arbiternode):<br>
          </div>
        </blockquote>
        <div><br>
        </div>
        <div><br>
        </div>
        <div>...which is what I was saying is incorrect. 2798404 is an
          XFS inode number and is not common to the same file across
          nodes. So you will get different results. Use the -samefile
          flag I shared earlier.<br>
        </div>
        <div> -Ravi<br>
        </div>
        <div> <br>
        </div>
        <div> <br>
        </div>
        <blockquote type="cite">
          <div><br>
          </div>
          <div>sudo find /data/myvolume/brick -inum 2798404 -ls <br>
          </div>
          <div><br>
          </div>
          <div>Here below are the results for all 3 nodes:<br>
          </div>
          <div><br>
          </div>
          <div>node1:<br>
          </div>
          <div><br>
          </div>
          <div>2798404   19 -rw-r--r--   2 www-data www-data       32
            Jun 19 17:42
/data/myvolume/brick/.glusterfs/e6/5b/e65b77e2-a4c4-4824-a7bb-58df969ce4b0<br>
          </div>
          <div>2798404   19 -rw-r--r--   2 www-data www-data       32
            Jun 19 17:42
            /data/myvolume/brick/&lt;REMOVED_DIRECTORIES_IN_BETWEEN&gt;/fileKey<br>
          </div>
          <div><br>
          </div>
          <div>node2:<br>
          </div>
          <div><br>
          </div>
          <div>2798404    1 ----------   2 root     root            0
            Apr 28 22:51
/data/myvolume/brick/.glusterfs/indices/xattrop/29e0d13e-1217-41cc-9bda-1fbbf781c397<br>
          </div>
          <div>2798404    1 ----------   2 root     root            0
            Apr 28 22:51
/data/myvolume/brick/.glusterfs/indices/xattrop/xattrop-6fa49ad5-71dd-4ec2-9246-7b302ab92d38<br>
          </div>
          <div><br>
          </div>
          <div>arbirternode:<br>
          </div>
          <div><br>
          </div>
          <div>NOTHING<br>
          </div>
          <div><br>
          </div>
          <div>As you requested I have tried to run on node1 a getfattr
            on the fileKey file by using the following command:<br>
          </div>
          <div><br>
          </div>
          <div>getfattr -m . -d -e hex fileKey<br>
          </div>
          <div><br>
          </div>
          <div>but there is no output. I am not familiar with the
            getfattr command so maybe I am using the wrong parameters,
            could you help me with that?<br>
          </div>
          <div><br>
          </div>
          <div><br>
          </div>
          <blockquote class="protonmail_quote" type="cite">
            <div>-------- Original Message --------<br>
            </div>
            <div>Subject: Re: [Gluster-users] Possible stale
              .glusterfs/indices/xattrop file?<br>
            </div>
            <div>Local Time: July 31, 2017 9:25 AM<br>
            </div>
            <div>UTC Time: July 31, 2017 7:25 AM<br>
            </div>
            <div>From: <a href="mailto:ravishankar@redhat.com"
                class="moz-txt-link-abbreviated" rel="noreferrer
                nofollow noopener" moz-do-not-send="true">ravishankar@redhat.com</a><br>
            </div>
            <div>To: mabi <a href="mailto:mabi@protonmail.ch"
                class="moz-txt-link-rfc2396E" rel="noreferrer nofollow
                noopener" moz-do-not-send="true">&lt;mabi@protonmail.ch&gt;</a><br>
            </div>
            <div>Gluster Users <a
                href="mailto:gluster-users@gluster.org"
                class="moz-txt-link-rfc2396E" rel="noreferrer nofollow
                noopener" moz-do-not-send="true">&lt;gluster-users@gluster.org&gt;</a><br>
            </div>
            <div><br>
            </div>
            <div>On 07/31/2017 12:20 PM, mabi wrote:<br>
            </div>
            <div><br>
            </div>
            <blockquote type="cite">
              <div>I did a find on this inode number and I could find
                the file but only on node1 (nothing on node2 and the new
                arbiternode). Here is an ls -lai of the file itself on
                node1:<br>
              </div>
            </blockquote>
            <div>Sorry I don't understand, isn't that (XFS) inode number
              specific to node2's brick? If you want to use the same
              command, maybe you should try `find /data/myvolume/brick
              -samefile
/data/myvolume/brick/.glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397`
              on all 3 bricks.<br>
            </div>
            <div><br>
            </div>
            <blockquote type="cite">
              <div><br>
              </div>
              <div>-rw-r--r-- 1 www-data www-data   32 Jun 19 17:42
                fileKey<br>
              </div>
              <div><br>
              </div>
              <div>As you can see it is a 32 bytes file and as you
                suggested I ran a "stat" on this very same file through
                a glusterfs mount (using fuse) but unfortunately nothing
                happened. The GFID is still being displayed to be
                healed.  Just in case here is the output of the stat:<br>
              </div>
              <div><br>
              </div>
              <div>  File: ‘fileKey’<br>
              </div>
              <div>  Size: 32        Blocks: 1          IO Block: 131072
                regular file<br>
              </div>
              <div>Device: 1eh/30d Inode: 12086351742306673840  Links: 1<br>
              </div>
              <div>Access: (0644/-rw-r--r--)  Uid: (   33/www-data)  
                Gid: (   33/www-data)<br>
              </div>
              <div>Access: 2017-06-19 17:42:35.339773495 +0200<br>
              </div>
              <div>Modify: 2017-06-19 17:42:35.343773437 +0200<br>
              </div>
              <div>Change: 2017-06-19 17:42:35.343773437 +0200<br>
              </div>
              <div>Birth: -<br>
              </div>
              <div><br>
              </div>
            </blockquote>
            <div>Is this 'fileKey' on node1 having the same gfid (see
              getfattr output)? Looks like it is missing the hardlink
              inside .glusterfs folder since the link count is only 1.<br>
            </div>
            <div>Thanks,<br>
            </div>
            <div>Ravi<br>
            </div>
            <div><br>
            </div>
            <blockquote type="cite">
              <div>What else can I do or try in order to fix this
                situation?<br>
              </div>
              <div><br>
              </div>
              <div class="protonmail_signature_block
                protonmail_signature_block-empty">
                <div class="protonmail_signature_block-user
                  protonmail_signature_block-empty">
                  <div><br>
                  </div>
                </div>
                <div class="protonmail_signature_block-proton
                  protonmail_signature_block-empty"><br>
                </div>
              </div>
              <div><br>
              </div>
              <blockquote type="cite" class="protonmail_quote">
                <div>-------- Original Message --------<br>
                </div>
                <div>Subject: Re: [Gluster-users] Possible stale
                  .glusterfs/indices/xattrop file?<br>
                </div>
                <div>Local Time: July 31, 2017 3:27 AM<br>
                </div>
                <div>UTC Time: July 31, 2017 1:27 AM<br>
                </div>
                <div>From: <a href="mailto:ravishankar@redhat.com"
                    class="moz-txt-link-abbreviated" rel="noreferrer
                    nofollow noopener" moz-do-not-send="true">ravishankar@redhat.com</a><br>
                </div>
                <div>To: mabi <a href="mailto:mabi@protonmail.ch"
                    class="moz-txt-link-rfc2396E" rel="noreferrer
                    nofollow noopener" moz-do-not-send="true">&lt;mabi@protonmail.ch&gt;</a><br>
                </div>
                <div>Gluster Users <a
                    href="mailto:gluster-users@gluster.org"
                    class="moz-txt-link-rfc2396E" rel="noreferrer
                    nofollow noopener" moz-do-not-send="true">&lt;gluster-users@gluster.org&gt;</a><br>
                </div>
                <div><br>
                </div>
                <div><br>
                </div>
                <p><br>
                </p>
                <div><br>
                </div>
                <div class="moz-cite-prefix">On 07/30/2017 02:24 PM,
                  mabi wrote:<br>
                </div>
                <blockquote type="cite">
                  <div>Hi Ravi,<br>
                  </div>
                  <div><br>
                  </div>
                  <div>Thanks for your hints. Below you will find the
                    answer to your questions.<br>
                  </div>
                  <div><br>
                  </div>
                  <div>First I tried to start the healing process by
                    running:<br>
                  </div>
                  <div><br>
                  </div>
                  <div>gluster volume heal myvolume<br>
                  </div>
                  <div><br>
                  </div>
                  <div>and then as you suggested watch the output of the
                    glustershd.log file but nothing appeared in that log
                    file after running the above command. I checked the
                    files which need to be healing using the "heal
                    &lt;volume&gt; info" command and it still shows that
                    very same GFID on node2 to be healed. So nothing
                    changed here.<br>
                  </div>
                  <div><br>
                  </div>
                  <div>The file
/data/myvolume/brick/.glusterfs/indices/xattrop/29e0d13e-1217-41cc-9bda-1fbbf781c397
                    is only on node2 and not on my nod1 nor on my
                    arbiternode. This file seems to be a regular file
                    and not a symlink. Here is the output of the stat
                    command on it from my node2:<br>
                  </div>
                  <div><br>
                  </div>
                  <div>  File:
‘/data/myvolume/brick/.glusterfs/indices/xattrop/29e0d13e-1217-41cc-9bda-1fbbf781c397’<br>
                  </div>
                  <div>  Size: 0         Blocks: 1          IO Block:
                    512    regular empty file<br>
                  </div>
                  <div>Device: 25h/37d Inode: 2798404     Links: 2<br>
                  </div>
                </blockquote>
                <div><br>
                </div>
                <div>Okay, link count of 2 means there is a hardlink
                  somewhere on the brick. Try the find command again. I
                  see that the inode number is 2798404, not the one you
                  shared in your first mail. Once you find the path to
                  the file, do a stat of the file from mount. This
                  should create the entry in the other 2 bricks and do
                  the heal. But FWIW, this seems to be a zero byte file.<br>
                </div>
                <div> <br>
                </div>
                <div>Regards,<br>
                </div>
                <div>Ravi<br>
                </div>
                <div><br>
                </div>
                <blockquote type="cite">
                  <div>Access: (0000/----------)  Uid: (    0/   
                    root)   Gid: (    0/    root)<br>
                  </div>
                  <div>Access: 2017-04-28 22:51:15.215775269 +0200<br>
                  </div>
                  <div>Modify: 2017-04-28 22:51:15.215775269 +0200<br>
                  </div>
                  <div>Change: 2017-07-30 08:39:03.700872312 +0200<br>
                  </div>
                  <div>Birth: -<br>
                  </div>
                  <div><br>
                  </div>
                  <div>I hope this is enough info for a starter, else
                    let me know if you need any more info. I would be
                    glad to resolve this weird file which needs to be
                    healed but can not.<br>
                  </div>
                  <div><br>
                  </div>
                  <div>Best regards,<br>
                  </div>
                  <div>Mabi<br>
                  </div>
                  <div class="protonmail_signature_block
                    protonmail_signature_block-empty">
                    <div class="protonmail_signature_block-user
                      protonmail_signature_block-empty">
                      <div><br>
                      </div>
                    </div>
                    <div class="protonmail_signature_block-proton
                      protonmail_signature_block-empty"><br>
                    </div>
                  </div>
                  <div><br>
                  </div>
                  <blockquote class="protonmail_quote" type="cite">
                    <div>-------- Original Message --------<br>
                    </div>
                    <div>Subject: Re: [Gluster-users] Possible stale
                      .glusterfs/indices/xattrop file?<br>
                    </div>
                    <div>Local Time: July 30, 2017 3:31 AM<br>
                    </div>
                    <div>UTC Time: July 30, 2017 1:31 AM<br>
                    </div>
                    <div>From: <a rel="noreferrer nofollow noopener"
                        class="moz-txt-link-abbreviated"
                        href="mailto:ravishankar@redhat.com"
                        moz-do-not-send="true">ravishankar@redhat.com</a><br>
                    </div>
                    <div>To: mabi <a rel="noreferrer nofollow noopener"
                        class="moz-txt-link-rfc2396E"
                        href="mailto:mabi@protonmail.ch"
                        moz-do-not-send="true">&lt;mabi@protonmail.ch&gt;</a>,
                      Gluster Users <a rel="noreferrer nofollow
                        noopener" class="moz-txt-link-rfc2396E"
                        href="mailto:gluster-users@gluster.org"
                        moz-do-not-send="true">&lt;gluster-users@gluster.org&gt;</a><br>
                    </div>
                    <div><br>
                    </div>
                    <div><br>
                    </div>
                    <p><br>
                    </p>
                    <div><br>
                    </div>
                    <div class="moz-cite-prefix">On 07/29/2017 04:36 PM,
                      mabi wrote:<br>
                    </div>
                    <blockquote type="cite">
                      <div>Hi,<br>
                      </div>
                      <div><br>
                      </div>
                      <div>Sorry for mailing again but as mentioned in
                        my previous mail, I have added an arbiter node
                        to my replica 2 volume and it seem to have gone
                        fine except for the fact that there is one
                        single file which needs healing and does not get
                        healed as you can see here from the output of a
                        "heal info":<br>
                      </div>
                      <div><br>
                      </div>
                      <div>Brick node1.domain.tld:/data/myvolume/brick<br>
                      </div>
                      <div>Status: Connected<br>
                      </div>
                      <div>Number of entries: 0<br>
                      </div>
                      <div><br>
                      </div>
                      <div>Brick node2.domain.tld:/data/myvolume/brick<br>
                      </div>
                      <div>&lt;gfid:29e0d13e-1217-41cc-9bda-1fbbf781c397&gt;<br>
                      </div>
                      <div>Status: Connected<br>
                      </div>
                      <div>Number of entries: 1<br>
                      </div>
                      <div><br>
                      </div>
                      <div>Brick
                        arbiternode.domain.tld:/srv/glusterfs/myvolume/brick<br>
                      </div>
                      <div>Status: Connected<br>
                      </div>
                      <div>Number of entries: 0<br>
                      </div>
                      <div><br>
                      </div>
                      <div>On my node2 the respective
                        .glusterfs/indices/xattrop directory contains
                        two files as you can see below:<br>
                      </div>
                      <div><br>
                      </div>
                      <div>ls -lai 
                        /data/myvolume/brick/.glusterfs/indices/xattrop<br>
                      </div>
                      <div>total 76180<br>
                      </div>
                      <div>     10 drw------- 2 root root 4 Jul 29 12:15
                        .<br>
                      </div>
                      <div>      9 drw------- 5 root root 5 Apr 28 22:15
                        ..<br>
                      </div>
                      <div>2798404 ---------- 2 root root 0 Apr 28 22:51
                        29e0d13e-1217-41cc-9bda-1fbbf781c397<br>
                      </div>
                      <div>2798404 ---------- 2 root root 0 Apr 28 22:51
                        xattrop-6fa49ad5-71dd-4ec2-9246-7b302ab92d38<br>
                      </div>
                      <div class="protonmail_signature_block
                        protonmail_signature_block-empty">
                        <div class="protonmail_signature_block-user
                          protonmail_signature_block-empty">
                          <div><br>
                          </div>
                        </div>
                        <div class="protonmail_signature_block-proton
                          protonmail_signature_block-empty"><br>
                        </div>
                      </div>
                      <div><br>
                      </div>
                      <div>I tried to find the real file on my brick
                        where this xattrop file points to using its
                        inode number (command: find
                        /data/myvolume/brick/data -inum 8394642) but it
                        does not find any associated file.<br>
                      </div>
                      <div><br>
                      </div>
                      <div>So my question here is, is it possible that
                        this is a stale file which just forgot to get
                        deleted from the indices/xattrop file by gluster
                        for some unknown reason? If yes is it safe for
                        me to delete these two files? or what would be
                        the correct process in that case?<br>
                      </div>
                    </blockquote>
                    <div>The 'xattrop-6fa...' is the base entry. gfids
                      of files that need heal are hard linked to this
                      entry, so nothing needs to be done for it. But you
                      need to find out why '29e0d13...' is not healing.
                      Launch the heal and observe the glustershd logs
                      for errors. I suppose the inode number for
                      .glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397
                      is what is 8394642.  Is
                      .glusterfs/29/e0/29e0d13e-1217-41cc-9bda-1fbbf781c397
                      a regular file or  symlink? Does it exist in the
                      other 2 bricks? What is the link count (as seen
                      from stat &lt;file&gt;)?<br>
                    </div>
                    <div>-Ravi<br>
                    </div>
                    <div><br>
                    </div>
                    <blockquote type="cite">
                      <div><br>
                      </div>
                      <div>Thank you for your input.<br>
                      </div>
                      <div>Mabi<br>
                      </div>
                      <div><br>
                      </div>
                      <div><br>
                      </div>
                      <pre wrap="">_______________________________________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" class="moz-txt-link-abbreviated" rel="noreferrer nofollow noopener" moz-do-not-send="true">Gluster-users@gluster.org</a>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" class="moz-txt-link-freetext" rel="noreferrer nofollow noopener" moz-do-not-send="true">http://lists.gluster.org/mailman/listinfo/gluster-users</a>



</pre>
                    </blockquote>
                  </blockquote>
                  <div><br>
                  </div>
                </blockquote>
              </blockquote>
              <div><br>
              </div>
            </blockquote>
          </blockquote>
          <div><br>
          </div>
        </blockquote>
      </blockquote>
      <div><br>
      </div>
    </blockquote>
    <br>
  </body>
</html>