<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p><br>
    </p>
    <br>
    <div class="moz-cite-prefix">On 08/22/2017 02:30 PM, mabi wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:ADPyWM3z0Ka7a0CMQ9r_8-mAuP-p6hXyJ9y9dcrdaAuiCpd23i69jtyYrllqin6FVhS65LPqGKt-8qvMvx_QvVHAoXYmiS5CFUwUoKZAsiQ=@protonmail.ch">
      <div>Thanks for the additional hints, I have the following 2
        questions first:<br>
      </div>
      <div><br>
      </div>
      <div>- In order to launch the index heal is the following command
        correct:<br>
      </div>
      <div>gluster volume heal myvolume<br>
      </div>
      <div><br>
      </div>
    </blockquote>
    Yes<br>
    <blockquote type="cite"
cite="mid:ADPyWM3z0Ka7a0CMQ9r_8-mAuP-p6hXyJ9y9dcrdaAuiCpd23i69jtyYrllqin6FVhS65LPqGKt-8qvMvx_QvVHAoXYmiS5CFUwUoKZAsiQ=@protonmail.ch">
      <div>- If I run a "volume start force" will it have any short
        disruptions on my clients which mount the volume through FUSE?
        If yes, how long? This is a production system that's why I am
        asking.<br>
      </div>
      <div><br>
      </div>
      <div class="protonmail_signature_block
        protonmail_signature_block-empty">
        <div class="protonmail_signature_block-user
          protonmail_signature_block-empty">
          <div><br>
          </div>
        </div>
      </div>
    </blockquote>
    No. You can actually create a test volume on  your personal linux
    box to try these kinds of things without needing multiple machines.
    This is how we develop and test our patches :)<br>
    'gluster volume create testvol replica 3
    /home/mabi/bricks/brick{1..3} force` and so on. <br>
    <br>
    HTH,<br>
    Ravi <br>
    <br>
    <blockquote type="cite"
cite="mid:ADPyWM3z0Ka7a0CMQ9r_8-mAuP-p6hXyJ9y9dcrdaAuiCpd23i69jtyYrllqin6FVhS65LPqGKt-8qvMvx_QvVHAoXYmiS5CFUwUoKZAsiQ=@protonmail.ch">
      <div><br>
      </div>
      <blockquote class="protonmail_quote" type="cite">
        <div>-------- Original Message --------<br>
        </div>
        <div>Subject: Re: [Gluster-users] self-heal not working<br>
        </div>
        <div>Local Time: August 22, 2017 6:26 AM<br>
        </div>
        <div>UTC Time: August 22, 2017 4:26 AM<br>
        </div>
        <div>From: <a class="moz-txt-link-abbreviated" href="mailto:ravishankar@redhat.com">ravishankar@redhat.com</a><br>
        </div>
        <div>To: mabi <a class="moz-txt-link-rfc2396E" href="mailto:mabi@protonmail.ch">&lt;mabi@protonmail.ch&gt;</a>, Ben Turner
          <a class="moz-txt-link-rfc2396E" href="mailto:bturner@redhat.com">&lt;bturner@redhat.com&gt;</a><br>
        </div>
        <div>Gluster Users <a class="moz-txt-link-rfc2396E" href="mailto:gluster-users@gluster.org">&lt;gluster-users@gluster.org&gt;</a><br>
        </div>
        <div><br>
        </div>
        <div> <br>
        </div>
        <p>Explore the following:<br>
        </p>
        <p>- Launch index heal and look at the glustershd logs of all
          bricks for possible errors<br>
        </p>
        <p>- See if the glustershd in each node is connected to all
          bricks.<br>
        </p>
        <p>- If not try to restart shd by `volume start force`<br>
        </p>
        <p>- Launch index heal again and try.<br>
        </p>
        <div>- Try debugging the shd log by setting client-log-level to
          DEBUG temporarily.<br>
        </div>
        <div> <br>
        </div>
        <div> <br>
        </div>
        <div class="moz-cite-prefix">On 08/22/2017 03:19 AM, mabi wrote:<br>
        </div>
        <blockquote type="cite">
          <div>Sure, it doesn't look like a split brain based on the
            output:<br>
          </div>
          <div><br>
          </div>
          <div>Brick node1.domain.tld:/data/myvolume/brick<br>
          </div>
          <div>Status: Connected<br>
          </div>
          <div>Number of entries in split-brain: 0<br>
          </div>
          <div><br>
          </div>
          <div>Brick node2.domain.tld:/data/myvolume/brick<br>
          </div>
          <div>Status: Connected<br>
          </div>
          <div>Number of entries in split-brain: 0<br>
          </div>
          <div><br>
          </div>
          <div>Brick node3.domain.tld:/srv/glusterfs/myvolume/brick<br>
          </div>
          <div>Status: Connected<br>
          </div>
          <div>Number of entries in split-brain: 0<br>
          </div>
          <div><br>
          </div>
          <div class="protonmail_signature_block
            protonmail_signature_block-empty">
            <div class="protonmail_signature_block-user
              protonmail_signature_block-empty">
              <div><br>
              </div>
            </div>
            <div class="protonmail_signature_block-proton
              protonmail_signature_block-empty"><br>
            </div>
          </div>
          <div><br>
          </div>
          <blockquote type="cite" class="protonmail_quote">
            <div>-------- Original Message --------<br>
            </div>
            <div>Subject: Re: [Gluster-users] self-heal not working<br>
            </div>
            <div>Local Time: August 21, 2017 11:35 PM<br>
            </div>
            <div>UTC Time: August 21, 2017 9:35 PM<br>
            </div>
            <div>From: <a href="mailto:bturner@redhat.com"
                class="moz-txt-link-abbreviated" moz-do-not-send="true">bturner@redhat.com</a><br>
            </div>
            <div>To: mabi <a href="mailto:mabi@protonmail.ch"
                class="moz-txt-link-rfc2396E" moz-do-not-send="true">&lt;mabi@protonmail.ch&gt;</a><br>
            </div>
            <div>Gluster Users <a
                href="mailto:gluster-users@gluster.org"
                class="moz-txt-link-rfc2396E" moz-do-not-send="true">&lt;gluster-users@gluster.org&gt;</a><br>
            </div>
            <div><br>
            </div>
            <div>Can you also provide:<br>
            </div>
            <div><br>
            </div>
            <div>gluster v heal &lt;my vol&gt; info split-brain<br>
            </div>
            <div><br>
            </div>
            <div>If it is split brain just delete the incorrect file
              from the brick and run heal again. I haven"t tried this
              with arbiter but I assume the process is the same.<br>
            </div>
            <div><br>
            </div>
            <div>-b<br>
            </div>
            <div><br>
            </div>
            <div>----- Original Message -----<br>
            </div>
            <div>&gt; From: "mabi" <a href="mailto:mabi@protonmail.ch"
                class="moz-txt-link-rfc2396E" moz-do-not-send="true">&lt;mabi@protonmail.ch&gt;</a><br>
            </div>
            <div>&gt; To: "Ben Turner" <a
                href="mailto:bturner@redhat.com"
                class="moz-txt-link-rfc2396E" moz-do-not-send="true">&lt;bturner@redhat.com&gt;</a><br>
            </div>
            <div>&gt; Cc: "Gluster Users" <a
                href="mailto:gluster-users@gluster.org"
                class="moz-txt-link-rfc2396E" moz-do-not-send="true">&lt;gluster-users@gluster.org&gt;</a><br>
            </div>
            <div>&gt; Sent: Monday, August 21, 2017 4:55:59 PM<br>
            </div>
            <div>&gt; Subject: Re: [Gluster-users] self-heal not working<br>
            </div>
            <div>&gt; <br>
            </div>
            <div>&gt; Hi Ben,<br>
            </div>
            <div>&gt; <br>
            </div>
            <div>&gt; So it is really a 0 kBytes file everywhere (all
              nodes including the arbiter<br>
            </div>
            <div>&gt; and from the client).<br>
            </div>
            <div>&gt; Here below you will find the output you requested.
              Hopefully that will help<br>
            </div>
            <div>&gt; to find out why this specific file is not
              healing... Let me know if you need<br>
            </div>
            <div>&gt; any more information. Btw node3 is my arbiter
              node.<br>
            </div>
            <div>&gt; <br>
            </div>
            <div>&gt; NODE1:<br>
            </div>
            <div>&gt; <br>
            </div>
            <div>&gt; STAT:<br>
            </div>
            <div>&gt; File:<br>
            </div>
            <div>&gt;
‘/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png’<br>
            </div>
            <div>&gt; Size: 0 Blocks: 38 IO Block: 131072 regular empty
              file<br>
            </div>
            <div>&gt; Device: 24h/36d Inode: 10033884 Links: 2<br>
            </div>
            <div>&gt; Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid:
              ( 33/www-data)<br>
            </div>
            <div>&gt; Access: 2017-08-14 17:04:55.530681000 +0200<br>
            </div>
            <div>&gt; Modify: 2017-08-14 17:11:46.407404779 +0200<br>
            </div>
            <div>&gt; Change: 2017-08-14 17:11:46.407404779 +0200<br>
            </div>
            <div>&gt; Birth: -<br>
            </div>
            <div>&gt; <br>
            </div>
            <div>&gt; GETFATTR:<br>
            </div>
            <div>&gt; trusted.afr.dirty=0sAAAAAQAAAAAAAAAA<br>
            </div>
            <div>&gt; trusted.bit-rot.version=0sAgAAAAAAAABZhuknAAlJAg==<br>
            </div>
            <div>&gt; trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==<br>
            </div>
            <div>&gt;
trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOyo=<br>
            </div>
            <div>&gt; <br>
            </div>
            <div>&gt; NODE2:<br>
            </div>
            <div>&gt; <br>
            </div>
            <div>&gt; STAT:<br>
            </div>
            <div>&gt; File:<br>
            </div>
            <div>&gt;
‘/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png’<br>
            </div>
            <div>&gt; Size: 0 Blocks: 38 IO Block: 131072 regular empty
              file<br>
            </div>
            <div>&gt; Device: 26h/38d Inode: 10031330 Links: 2<br>
            </div>
            <div>&gt; Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid:
              ( 33/www-data)<br>
            </div>
            <div>&gt; Access: 2017-08-14 17:04:55.530681000 +0200<br>
            </div>
            <div>&gt; Modify: 2017-08-14 17:11:46.403704181 +0200<br>
            </div>
            <div>&gt; Change: 2017-08-14 17:11:46.403704181 +0200<br>
            </div>
            <div>&gt; Birth: -<br>
            </div>
            <div>&gt; <br>
            </div>
            <div>&gt; GETFATTR:<br>
            </div>
            <div>&gt; trusted.afr.dirty=0sAAAAAQAAAAAAAAAA<br>
            </div>
            <div>&gt; trusted.bit-rot.version=0sAgAAAAAAAABZhu6wAA8Hpw==<br>
            </div>
            <div>&gt; trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==<br>
            </div>
            <div>&gt;
trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOVE=<br>
            </div>
            <div>&gt; <br>
            </div>
            <div>&gt; NODE3:<br>
            </div>
            <div>&gt; STAT:<br>
            </div>
            <div>&gt; File:<br>
            </div>
            <div>&gt;
/srv/glusterfs/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png<br>
            </div>
            <div>&gt; Size: 0 Blocks: 0 IO Block: 4096 regular empty
              file<br>
            </div>
            <div>&gt; Device: ca11h/51729d Inode: 405208959 Links: 2<br>
            </div>
            <div>&gt; Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid:
              ( 33/www-data)<br>
            </div>
            <div>&gt; Access: 2017-08-14 17:04:55.530681000 +0200<br>
            </div>
            <div>&gt; Modify: 2017-08-14 17:04:55.530681000 +0200<br>
            </div>
            <div>&gt; Change: 2017-08-14 17:11:46.604380051 +0200<br>
            </div>
            <div>&gt; Birth: -<br>
            </div>
            <div>&gt; <br>
            </div>
            <div>&gt; GETFATTR:<br>
            </div>
            <div>&gt; trusted.afr.dirty=0sAAAAAQAAAAAAAAAA<br>
            </div>
            <div>&gt; trusted.bit-rot.version=0sAgAAAAAAAABZe6ejAAKPAg==<br>
            </div>
            <div>&gt; trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==<br>
            </div>
            <div>&gt;
trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOc4=<br>
            </div>
            <div>&gt; <br>
            </div>
            <div>&gt; CLIENT GLUSTER MOUNT:<br>
            </div>
            <div>&gt; STAT:<br>
            </div>
            <div>&gt; File:<br>
            </div>
            <div>&gt;
              "/mnt/myvolume/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png"<br>
            </div>
            <div>&gt; Size: 0 Blocks: 0 IO Block: 131072 regular empty
              file<br>
            </div>
            <div>&gt; Device: 1eh/30d Inode: 11897049013408443114 Links:
              1<br>
            </div>
            <div>&gt; Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid:
              ( 33/www-data)<br>
            </div>
            <div>&gt; Access: 2017-08-14 17:04:55.530681000 +0200<br>
            </div>
            <div>&gt; Modify: 2017-08-14 17:11:46.407404779 +0200<br>
            </div>
            <div>&gt; Change: 2017-08-14 17:11:46.407404779 +0200<br>
            </div>
            <div>&gt; Birth: -<br>
            </div>
            <div>&gt; <br>
            </div>
            <div>&gt; &gt; -------- Original Message --------<br>
            </div>
            <div>&gt; &gt; Subject: Re: [Gluster-users] self-heal not
              working<br>
            </div>
            <div>&gt; &gt; Local Time: August 21, 2017 9:34 PM<br>
            </div>
            <div>&gt; &gt; UTC Time: August 21, 2017 7:34 PM<br>
            </div>
            <div>&gt; &gt; From: <a href="mailto:bturner@redhat.com"
                class="moz-txt-link-abbreviated" moz-do-not-send="true">bturner@redhat.com</a><br>
            </div>
            <div>&gt; &gt; To: mabi <a href="mailto:mabi@protonmail.ch"
                class="moz-txt-link-rfc2396E" moz-do-not-send="true">&lt;mabi@protonmail.ch&gt;</a><br>
            </div>
            <div>&gt; &gt; Gluster Users <a
                href="mailto:gluster-users@gluster.org"
                class="moz-txt-link-rfc2396E" moz-do-not-send="true">&lt;gluster-users@gluster.org&gt;</a><br>
            </div>
            <div>&gt; &gt;<br>
            </div>
            <div>&gt; &gt; ----- Original Message -----<br>
            </div>
            <div>&gt; &gt;&gt; From: "mabi" <a
                href="mailto:mabi@protonmail.ch"
                class="moz-txt-link-rfc2396E" moz-do-not-send="true">&lt;mabi@protonmail.ch&gt;</a><br>
            </div>
            <div>&gt; &gt;&gt; To: "Gluster Users" <a
                href="mailto:gluster-users@gluster.org"
                class="moz-txt-link-rfc2396E" moz-do-not-send="true">&lt;gluster-users@gluster.org&gt;</a><br>
            </div>
            <div>&gt; &gt;&gt; Sent: Monday, August 21, 2017 9:28:24 AM<br>
            </div>
            <div>&gt; &gt;&gt; Subject: [Gluster-users] self-heal not
              working<br>
            </div>
            <div>&gt; &gt;&gt;<br>
            </div>
            <div>&gt; &gt;&gt; Hi,<br>
            </div>
            <div>&gt; &gt;&gt;<br>
            </div>
            <div>&gt; &gt;&gt; I have a replicat 2 with arbiter
              GlusterFS 3.8.11 cluster and there is<br>
            </div>
            <div>&gt; &gt;&gt; currently one file listed to be healed as
              you can see below but never gets<br>
            </div>
            <div>&gt; &gt;&gt; healed by the self-heal daemon:<br>
            </div>
            <div>&gt; &gt;&gt;<br>
            </div>
            <div>&gt; &gt;&gt; Brick
              node1.domain.tld:/data/myvolume/brick<br>
            </div>
            <div>&gt; &gt;&gt;
              /data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png<br>
            </div>
            <div>&gt; &gt;&gt; Status: Connected<br>
            </div>
            <div>&gt; &gt;&gt; Number of entries: 1<br>
            </div>
            <div>&gt; &gt;&gt;<br>
            </div>
            <div>&gt; &gt;&gt; Brick
              node2.domain.tld:/data/myvolume/brick<br>
            </div>
            <div>&gt; &gt;&gt;
              /data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png<br>
            </div>
            <div>&gt; &gt;&gt; Status: Connected<br>
            </div>
            <div>&gt; &gt;&gt; Number of entries: 1<br>
            </div>
            <div>&gt; &gt;&gt;<br>
            </div>
            <div>&gt; &gt;&gt; Brick
              node3.domain.tld:/srv/glusterfs/myvolume/brick<br>
            </div>
            <div>&gt; &gt;&gt;
              /data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png<br>
            </div>
            <div>&gt; &gt;&gt; Status: Connected<br>
            </div>
            <div>&gt; &gt;&gt; Number of entries: 1<br>
            </div>
            <div>&gt; &gt;&gt;<br>
            </div>
            <div>&gt; &gt;&gt; As once recommended on this mailing list
              I have mounted that glusterfs<br>
            </div>
            <div>&gt; &gt;&gt; volume<br>
            </div>
            <div>&gt; &gt;&gt; temporarily through fuse/glusterfs and
              ran a "stat" on that file which is<br>
            </div>
            <div>&gt; &gt;&gt; listed above but nothing happened.<br>
            </div>
            <div>&gt; &gt;&gt;<br>
            </div>
            <div>&gt; &gt;&gt; The file itself is available on all 3
              nodes/bricks but on the last node it<br>
            </div>
            <div>&gt; &gt;&gt; has a different date. By the way this
              file is 0 kBytes big. Is that maybe<br>
            </div>
            <div>&gt; &gt;&gt; the reason why the self-heal does not
              work?<br>
            </div>
            <div>&gt; &gt;<br>
            </div>
            <div>&gt; &gt; Is the file actually 0 bytes or is it just 0
              bytes on the arbiter(0 bytes<br>
            </div>
            <div>&gt; &gt; are expected on the arbiter, it just stores
              metadata)? Can you send us the<br>
            </div>
            <div>&gt; &gt; output from stat on all 3 nodes:<br>
            </div>
            <div>&gt; &gt;<br>
            </div>
            <div>&gt; &gt; $ stat &lt;file on back end brick&gt;<br>
            </div>
            <div>&gt; &gt; $ getfattr -d -m - &lt;file on back end
              brick&gt;<br>
            </div>
            <div>&gt; &gt; $ stat &lt;file from gluster mount&gt;<br>
            </div>
            <div>&gt; &gt;<br>
            </div>
            <div>&gt; &gt; Lets see what things look like on the back
              end, it should tell us why<br>
            </div>
            <div>&gt; &gt; healing is failing.<br>
            </div>
            <div>&gt; &gt;<br>
            </div>
            <div>&gt; &gt; -b<br>
            </div>
            <div>&gt; &gt;<br>
            </div>
            <div>&gt; &gt;&gt;<br>
            </div>
            <div>&gt; &gt;&gt; And how can I now make this file to heal?<br>
            </div>
            <div>&gt; &gt;&gt;<br>
            </div>
            <div>&gt; &gt;&gt; Thanks,<br>
            </div>
            <div>&gt; &gt;&gt; Mabi<br>
            </div>
            <div>&gt; &gt;&gt;<br>
            </div>
            <div>&gt; &gt;&gt;<br>
            </div>
            <div>&gt; &gt;&gt;<br>
            </div>
            <div>&gt; &gt;&gt;<br>
            </div>
            <div>&gt; &gt;&gt;
              _______________________________________________<br>
            </div>
            <div>&gt; &gt;&gt; Gluster-users mailing list<br>
            </div>
            <div>&gt; &gt;&gt; <a
                href="mailto:Gluster-users@gluster.org"
                class="moz-txt-link-abbreviated" moz-do-not-send="true">Gluster-users@gluster.org</a><br>
            </div>
            <div>&gt; &gt;&gt; <a
                href="http://lists.gluster.org/mailman/listinfo/gluster-users"
                class="moz-txt-link-freetext" moz-do-not-send="true">http://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
            </div>
          </blockquote>
          <div><br>
          </div>
          <div><br>
          </div>
          <div><br>
          </div>
          <pre wrap="">_______________________________________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" class="moz-txt-link-abbreviated" moz-do-not-send="true">Gluster-users@gluster.org</a>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" class="moz-txt-link-freetext" moz-do-not-send="true">http://lists.gluster.org/mailman/listinfo/gluster-users</a>
</pre>
        </blockquote>
      </blockquote>
      <div><br>
      </div>
    </blockquote>
    <br>
  </body>
</html>