<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p>Healing time set to 120 seconds for now.</p>
    <p>Just to make sure I understand I need to take the result of the
      gluster volume heal projects info and put it in a file. Then try
      and find each guid listed in that file in the .glusterfs directory
      for each brick listed in the output as having unhealed files and
      delete that file - if it exists.  If it doesn't exist don't worry
      about it. <br>
    </p>
    <p>So these bricks have unhealed entries listed<br>
    </p>
    <p>/srv/gfs01/Projects/.glusterfs - 85 files<br>
    </p>
    <p>/srv/gfs05/Projects/.glusterfs  - 58854 files<br>
    </p>
    <p>/srv/gfs06/Projects/.glusterfs- 58854 files</p>
    <p>Script time!<br>
    </p>
    <div class="moz-cite-prefix">On 12/31/18 4:39 AM, Davide Obbi wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAH2S_oTB4=M7N+5sP8SH5njzFghaMmyB7HO1s10cwgjkULqymw@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">cluster.quorum-type auto<br>
        cluster.quorum-count (null)<br>
        cluster.server-quorum-type off<br>
        cluster.server-quorum-ratio 0<br>
        cluster.quorum-reads                    no<br>
        <br>
        Where exacty do I remove the gfid entries from - the .glusterfs
        <br>
        <div> directory? --&gt; yes can't remember exactly where but try
          to do a find in the brick paths with the gfid  it should
          return something<br>
        </div>
        <br>
        Where do I put the cluster.heal-timeout option - which file?
        --&gt; gluster volume set volumename option value <br>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr">On Mon, Dec 31, 2018 at 10:34 AM Brett Holcomb
          &lt;<a href="mailto:biholcomb@l1049h.com"
            moz-do-not-send="true">biholcomb@l1049h.com</a>&gt; wrote:<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px&#xA;
          0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">That
          is probably the case as a lot of files were deleted some time
          ago.<br>
          <br>
          I'm on version 5.2 but was on 3.12 until about a week ago.<br>
          <br>
          Here is the quorum info.  I'm running a distributed replicated
          volumes <br>
          in 2 x 3 = 6<br>
          <br>
          cluster.quorum-type auto<br>
          cluster.quorum-count (null)<br>
          cluster.server-quorum-type off<br>
          cluster.server-quorum-ratio 0<br>
          cluster.quorum-reads                    no<br>
          <br>
          Where exacty do I remove the gfid entries from - the
          .glusterfs <br>
          directory?  Do I just delete all the directories can files
          under this <br>
          directory?<br>
          <br>
          Where do I put the cluster.heal-timeout option - which file?<br>
          <br>
          I think you've hit on the cause of the issue.  Thinking back
          we've had <br>
          some extended power outages and due to a misconfiguration in
          the swap <br>
          file device name a couple of the nodes did not come up and I
          didn't <br>
          catch it for a while so maybe the deletes occured then.<br>
          <br>
          Thank you.<br>
          <br>
          On 12/31/18 2:58 AM, Davide Obbi wrote:<br>
          &gt; if the long GFID does not correspond to any file it could
          mean the <br>
          &gt; file has been deleted by the client mounting the volume.
          I think this <br>
          &gt; is caused when the delete was issued and the number of
          active bricks <br>
          &gt; were not reaching quorum majority or a second brick was
          taken down <br>
          &gt; while another was down or did not finish the selfheal,
          the latter more <br>
          &gt; likely.<br>
          &gt; It would be interesting to see:<br>
          &gt; - what version of glusterfs you running, it happened to
          me with 3.12<br>
          &gt; - volume quorum rules: "gluster volume get vol all | grep
          quorum"<br>
          &gt;<br>
          &gt; To clean it up if i remember correctly it should be
          possible to delete <br>
          &gt; the gfid entries from the brick mounts on the glusterfs
          server nodes <br>
          &gt; reporting the files to heal.<br>
          &gt;<br>
          &gt; As a side note you might want to consider changing the
          selfheal <br>
          &gt; timeout to more agressive schedule in
          cluster.heal-timeout option<br>
          _______________________________________________<br>
          Gluster-users mailing list<br>
          <a href="mailto:Gluster-users@gluster.org" target="_blank"
            moz-do-not-send="true">Gluster-users@gluster.org</a><br>
          <a
            href="https://lists.gluster.org/mailman/listinfo/gluster-users"
            rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
      </div>
      <br clear="all">
      <br>
      -- <br>
      <div dir="ltr" class="gmail_signature"><span
          style="display:block;font-size:11pt;font-family:Century&#xA;
          Gothic;color:rgb(0,53,128)">
          <div
style="color:rgb(0,53,128);font-family:Arial,Helvetica,sans-serif;font-weight:bold;font-size:15px">
            <div>Davide Obbi</div>
            <div
              style="font-weight:normal;font-size:13px;color:rgb(0,174,239)">System
              Administrator<br>
              <br>
            </div>
            <div
              style="font-weight:normal;font-size:13px;color:rgb(102,102,102)">Booking.com
              B.V.<br>
              Vijzelstraat 66-80 Amsterdam 1017HL Netherlands</div>
            <div
              style="font-weight:normal;font-size:13px;color:rgb(102,102,102)"><span
                style="color:rgb(0,174,239)">Direct </span>+31207031558<br>
            </div>
            <div
              style="font-weight:normal;font-size:13px;color:rgb(102,102,102)">
              <div
                style="font-weight:bold;font-size:16px;color:rgb(0,53,128)"><a
                  href="https://www.booking.com/"
style="color:rgb(0,127,255);background-image:initial;background-position:initial;background-repeat:initial"
                  target="_blank" moz-do-not-send="true"><img
                    src="https://bstatic.com/static/img/siglogo.jpg"
                    alt="Booking.com" title="Booking.com"
                    moz-do-not-send="true"></a></div>
              <div><span style="font-size:11px">Empowering people to
                  experience the world since 1996</span></div>
              <div><span style="font-size:11px">43 languages, 214+
                  offices worldwide, 141,000+ global destinations, 29
                  million reported listings <br>
                </span><span style="font-size:11px">Subsidiary of
                  Booking Holdings Inc. (NASDAQ: BKNG)</span></div>
            </div>
          </div>
        </span></div>
    </blockquote>
  </body>
</html>