<div dir="ltr">OK, thank you. I&#39;ll try that.<div><br></div><div>The reason I was confused about its status is these things in the doc:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">How To Test<br>TBD.<br>Documentation<br>TBD<br>Status<br>Design complete. Implementation done. The only thing pending is the compounding of two fops in shd code.</blockquote></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><br>Sincerely,<br>Artem<br><br>--<br>Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK Mirror</a><span style="font-size:12.8000001907349px">, Illogical Robot LLC</span></div><div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a> | <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br></div></div></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Tue, Apr 17, 2018 at 11:49 PM, Ravishankar N <span dir="ltr">&lt;<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div text="#000000" bgcolor="#FFFFFF">
    <p><br>
    </p>
    <br>
    <div class="m_229972322048706121moz-cite-prefix">On 04/18/2018 11:59 AM, Artem
      Russakovskii wrote:<br>
    </div>
    <blockquote type="cite">
      <div dir="ltr">Btw, I&#39;ve now noticed at least 5 variations in
        toggling binary option values. Are they all interchangeable, or
        will using the wrong value not work in some cases?
        <div><br>
        </div>
        <div>yes/no</div>
        <div>true/false</div>
        <div>True/False</div>
        <div>on/off</div>
        <div>enable/disable</div>
        <div><br>
        </div>
        <div>It&#39;s quite a confusing/inconsistent practice, especially
          given that many options will accept any value without erroring
          out/validation.</div>
      </div>
    </blockquote>
    <br>
    All these options are okay.<br>
    <blockquote type="cite">
      <div dir="ltr">
        <div class="gmail_extra"><br clear="all">
          <div>
            <div class="m_229972322048706121m_-5940135115956573066gmail_signature" data-smartmail="gmail_signature">
              <div dir="ltr">
                <div>
                  <div dir="ltr">
                    <div>
                      <div dir="ltr">
                        <div>
                          <div dir="ltr">
                            <div>
                              <div dir="ltr"><br>
                                Sincerely,<br>
                                Artem<br>
                                <br>
                                --<br>
                                Founder, <a href="http://www.androidpolice.com" target="_blank">Android
                                  Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
                                  Mirror</a><span style="font-size:12.8000001907349px">,
                                  Illogical Robot LLC</span></div>
                              <div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
                                | <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a>
                                | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </div>
          <br>
          <div class="gmail_quote">On Tue, Apr 17, 2018 at 11:22 PM,
            Artem Russakovskii <span dir="ltr">&lt;<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>&gt;</span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div dir="ltr">Thanks for the link. Looking at the status
                of that doc, it isn&#39;t quite ready yet, and there&#39;s no
                mention of the option.</div>
            </blockquote>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    No, this is a completed feature available since 3.8 IIRC. You can
    use it safely. There is a difference in how to enable it though.
    Instead of using &#39;gluster volume set ...&#39;, you need to use &#39;gluster
    volume heal &lt;volname&gt; granular-entry-heal enable&#39; to turn it
    on. If there are no pending heals, it will run successfully.
    Otherwise you need to wait until heals are over (i.e. heal info
    shows zero entries). Just follow what the CLI says and you should be
    fine.<br>
    <br>
    -Ravi<br>
    <blockquote type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">
            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div dir="ltr">
                <div><br>
                </div>
                <div>Does it mean that whatever is ready now in 4.0.1 is
                  incomplete but can be enabled via
                  granular-entry-heal=on, and when it is complete, it&#39;ll
                  become the default and the flag will simply go away?
                  <div><br>
                  </div>
                  <div>Is there any risk enabling the option now in
                    4.0.1?</div>
                </div>
              </div>
              <div class="gmail_extra"><span><br clear="all">
                  <div>
                    <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205gmail_signature" data-smartmail="gmail_signature">
                      <div dir="ltr">
                        <div>
                          <div dir="ltr">
                            <div>
                              <div dir="ltr">
                                <div>
                                  <div dir="ltr">
                                    <div>
                                      <div dir="ltr"><br>
                                        Sincerely,<br>
                                        Artem<br>
                                        <br>
                                        --<br>
                                        Founder, <a href="http://www.androidpolice.com" target="_blank">Android
                                          Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
                                          Mirror</a><span style="font-size:12.8000001907349px">,
                                          Illogical Robot LLC</span></div>
                                      <div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
                                        | <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a>
                                        | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                  <br>
                </span>
                <div>
                  <div class="m_229972322048706121m_-5940135115956573066h5">
                    <div class="gmail_quote">On Tue, Apr 17, 2018 at
                      11:16 PM, Ravishankar N <span dir="ltr">&lt;<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>&gt;</span>
                      wrote:<br>
                      <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                        <div text="#000000" bgcolor="#FFFFFF"><span>
                            <p><br>
                            </p>
                            <br>
                            <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473moz-cite-prefix">On
                              04/18/2018 10:35 AM, Artem Russakovskii
                              wrote:<br>
                            </div>
                            <blockquote type="cite">
                              <div dir="ltr">Hi Ravi,
                                <div><br>
                                </div>
                                <div>Could you please expand on how
                                  these would help?</div>
                                <div><br>
                                </div>
                                <div>By forcing full here, we move the
                                  logic from the CPU to network, thus
                                  decreasing CPU utilization, is that
                                  right?</div>
                              </div>
                            </blockquote>
                          </span> Yes, &#39;diff&#39; employs the rchecksum FOP
                          which does a sha256  checksum which can
                          consume CPU. So yes it is sort of shifting the
                          load from CPU to the network. But if your
                          average file size is small, it would make
                          sense to copy the entire file instead of
                          computing checksums.<span><br>
                            <br>
                            <blockquote type="cite">
                              <div dir="ltr">
                                <div> This is assuming the CPU and disk
                                  utilization are caused by the differ
                                  and not by lstat and other calls or
                                  something.</div>
                                <div>
                                  <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Option:
                                    cluster.data-self-heal-algorit<wbr>hm<br>
                                    Default Value: (null)<br>
                                    Description: Select between &quot;full&quot;,
                                    &quot;diff&quot;. The &quot;full&quot; algorithm copies
                                    the entire file from source to sink.
                                    The &quot;diff&quot; algorithm copies to sink
                                    only those blocks whose checksums
                                    don&#39;t match with those of source. If
                                    no option is configured the option
                                    is chosen dynamically as follows: If
                                    the file does not exist on one of
                                    the sinks or empty file exists or if
                                    the source file size is about the
                                    same as page size the entire file
                                    will be read and written i.e &quot;full&quot;
                                    algo, otherwise &quot;diff&quot; algo is
                                    chosen.</blockquote>
                                </div>
                                <div><br>
                                </div>
                                <div>I really have no idea what this
                                  means and how/why it would help. Any
                                  more info on this option?</div>
                              </div>
                            </blockquote>
                            <br>
                          </span><a class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473moz-txt-link-freetext" href="https://github.com/gluster/glusterfs-specs/blob/master/done/GlusterFS%203.8/granular-entry-self-healing.md" target="_blank">https://github.com/gluster/glu<wbr>sterfs-specs/blob/master/done/<wbr>GlusterFS%203.8/granular-entry<wbr>-self-healing.md</a>
                          should help.<br>
                          Regards,<br>
                          Ravi
                          <div>
                            <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205h5"><br>
                              <br>
                              <blockquote type="cite">
                                <div dir="ltr">
                                  <div>
                                    <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Option:
                                      cluster.granular-entry-heal<br>
                                      Default Value: no<br>
                                      Description: If this option is
                                      enabled, self-heal will resort to
                                      granular way of recording
                                      changelogs and doing entry
                                      self-heal.</blockquote>
                                  </div>
                                  <div><br>
                                  </div>
                                  <div>Thank you.</div>
                                </div>
                                <div class="gmail_extra"><br clear="all">
                                  <div>
                                    <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473gmail_signature" data-smartmail="gmail_signature">
                                      <div dir="ltr">
                                        <div>
                                          <div dir="ltr">
                                            <div>
                                              <div dir="ltr">
                                                <div>
                                                  <div dir="ltr">
                                                    <div>
                                                      <div dir="ltr"><br>
                                                        Sincerely,<br>
                                                        Artem<br>
                                                        <br>
                                                        --<br>
                                                        Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
                                                          Mirror</a><span style="font-size:12.8000001907349px">, Illogical Robot LLC</span></div>
                                                      <div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
                                                        | <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
                                                      </div>
                                                    </div>
                                                  </div>
                                                </div>
                                              </div>
                                            </div>
                                          </div>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                  <br>
                                  <div class="gmail_quote">On Tue, Apr
                                    17, 2018 at 9:58 PM, Ravishankar N <span dir="ltr">&lt;<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>&gt;</span>
                                    wrote:<br>
                                    <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                      <div text="#000000" bgcolor="#FFFFFF">
                                        <div>
                                          <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473h5">
                                            <br>
                                            <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854moz-cite-prefix">On
                                              04/18/2018 10:14 AM, Artem
                                              Russakovskii wrote:<br>
                                            </div>
                                            <blockquote type="cite">
                                              <div dir="ltr">Following
                                                up here on a related and
                                                very serious for us
                                                issue.
                                                <div><br>
                                                </div>
                                                <div>I took down one of
                                                  the 4 replicate
                                                  gluster servers for
                                                  maintenance today.
                                                  There are 2 gluster
                                                  volumes totaling about
                                                  600GB. Not that much
                                                  data. After the server
                                                  comes back online, it
                                                  starts auto healing
                                                  and pretty much all
                                                  operations on gluster
                                                  freeze for many
                                                  minutes.</div>
                                                <div><br>
                                                </div>
                                                <div>For example, I was
                                                  trying to run an ls
                                                  -alrt in a folder with
                                                  7300 files, and it
                                                  took a good 15-20
                                                  minutes before
                                                  returning.</div>
                                                <div><br>
                                                </div>
                                                <div>During this time, I
                                                  can see iostat show
                                                  100% utilization on
                                                  the brick, heal status
                                                  takes many minutes to
                                                  return, glusterfsd
                                                  uses up tons of CPU (I
                                                  saw it spike to 600%).
                                                  gluster already has
                                                  massive performance
                                                  issues for me, but
                                                  healing after a 4-hour
                                                  downtime is on another
                                                  level of bad perf.</div>
                                                <div><br>
                                                </div>
                                                <div>For example, this
                                                  command took many
                                                  minutes to run:</div>
                                                <div><br>
                                                </div>
                                                <div>
                                                  <div>gluster volume
                                                    heal
                                                    androidpolice_data3
                                                    info summary</div>
                                                  <div>Brick
                                                    nexus2:/mnt/nexus2_block4/andr<wbr>oidpolice_data3</div>
                                                  <div>Status: Connected</div>
                                                  <div>Total Number of
                                                    entries: 91</div>
                                                  <div>Number of entries
                                                    in heal pending: 90</div>
                                                  <div>Number of entries
                                                    in split-brain: 0</div>
                                                  <div>Number of entries
                                                    possibly healing: 1</div>
                                                  <div><br>
                                                  </div>
                                                  <div>Brick
                                                    forge:/mnt/forge_block4/androi<wbr>dpolice_data3</div>
                                                  <div>Status: Connected</div>
                                                  <div>Total Number of
                                                    entries: 87</div>
                                                  <div>Number of entries
                                                    in heal pending: 86</div>
                                                  <div>Number of entries
                                                    in split-brain: 0</div>
                                                  <div>Number of entries
                                                    possibly healing: 1</div>
                                                  <div><br>
                                                  </div>
                                                  <div>Brick
                                                    hive:/mnt/hive_block4/androidp<wbr>olice_data3</div>
                                                  <div>Status: Connected</div>
                                                  <div>Total Number of
                                                    entries: 87</div>
                                                  <div>Number of entries
                                                    in heal pending: 86</div>
                                                  <div>Number of entries
                                                    in split-brain: 0</div>
                                                  <div>Number of entries
                                                    possibly healing: 1</div>
                                                  <div><br>
                                                  </div>
                                                  <div>Brick
                                                    citadel:/mnt/citadel_block4/an<wbr>droidpolice_data3</div>
                                                  <div>Status: Connected</div>
                                                  <div>Total Number of
                                                    entries: 0</div>
                                                  <div>Number of entries
                                                    in heal pending: 0</div>
                                                  <div>Number of entries
                                                    in split-brain: 0</div>
                                                  <div>Number of entries
                                                    possibly healing: 0</div>
                                                </div>
                                                <div><br>
                                                </div>
                                                <div><br>
                                                </div>
                                                <div>Statistics showed a
                                                  diminishing number of
                                                  failed heals:</div>
                                                <div>...</div>
                                                <div>
                                                  <div>Ending time of
                                                    crawl: Tue Apr 17
                                                    21:13:08 2018</div>
                                                  <div><br>
                                                  </div>
                                                  <div>Type of crawl:
                                                    INDEX</div>
                                                  <div>No. of entries
                                                    healed: 2</div>
                                                  <div>No. of entries in
                                                    split-brain: 0</div>
                                                  <div>No. of heal
                                                    failed entries: 102</div>
                                                  <div><br>
                                                  </div>
                                                  <div>Starting time of
                                                    crawl: Tue Apr 17
                                                    21:13:09 2018</div>
                                                  <div><br>
                                                  </div>
                                                  <div>Ending time of
                                                    crawl: Tue Apr 17
                                                    21:14:30 2018</div>
                                                  <div><br>
                                                  </div>
                                                  <div>Type of crawl:
                                                    INDEX</div>
                                                  <div>No. of entries
                                                    healed: 4</div>
                                                  <div>No. of entries in
                                                    split-brain: 0</div>
                                                  <div>No. of heal
                                                    failed entries: 91</div>
                                                  <div><br>
                                                  </div>
                                                  <div>Starting time of
                                                    crawl: Tue Apr 17
                                                    21:14:31 2018</div>
                                                  <div><br>
                                                  </div>
                                                  <div>Ending time of
                                                    crawl: Tue Apr 17
                                                    21:15:34 2018</div>
                                                  <div><br>
                                                  </div>
                                                  <div>Type of crawl:
                                                    INDEX</div>
                                                  <div>No. of entries
                                                    healed: 0</div>
                                                  <div>No. of entries in
                                                    split-brain: 0</div>
                                                  <div>No. of heal
                                                    failed entries: 88</div>
                                                </div>
                                                <div>...</div>
                                                <div><br>
                                                </div>
                                                <div>Eventually,
                                                  everything heals and
                                                  goes back to at least
                                                  where the roof isn&#39;t
                                                  on fire anymore.</div>
                                                <div><br>
                                                </div>
                                                <div>The server stats
                                                  and volume options
                                                  were given in one of
                                                  the previous replies
                                                  to this thread.</div>
                                                <div><br>
                                                </div>
                                                <div>Any ideas or things
                                                  I could run and show
                                                  the output of to help
                                                  diagnose? I&#39;m also
                                                  very open to working
                                                  with someone on the
                                                  team on a live
                                                  debugging session if
                                                  there&#39;s interest.</div>
                                              </div>
                                            </blockquote>
                                            <br>
                                          </div>
                                        </div>
                                        It is likely that self-heal is
                                        causing the CPU spike due to the
                                        flood of lookups/ locks and
                                        checksum fops that the
                                        self-heal-daemon sends to the
                                        bricks.<br>
                                        There&#39;s a script to control
                                        shd&#39;s cpu usage using cgroups.
                                        That should help in regulating
                                        self-heal traffic: <a class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854moz-txt-link-freetext" href="https://review.gluster.org/#/c/18404/" target="_blank">https://review.gluster.org/#/c<wbr>/18404/</a>
                                        (see extras/control-cpu-load.sh)<br>
                                        Other self-heal related volume
                                        options that you could change
                                        are setting
                                        &#39;cluster.data-self-heal-algori<wbr>thm&#39;
                                        to &#39;full&#39; and
                                        &#39;granular-entry-heal&#39; to
                                        &#39;enable&#39;.  `gluster volume set
                                        help` should give you more
                                        information about these options.<br>
                                        Thanks,<br>
                                        Ravi
                                        <div>
                                          <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473h5"><br>
                                            <br>
                                            <blockquote type="cite">
                                              <div dir="ltr">
                                                <div><br>
                                                </div>
                                                <div>Thank you.</div>
                                              </div>
                                              <div class="gmail_extra"><br clear="all">
                                                <div>
                                                  <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854gmail_signature" data-smartmail="gmail_signature">
                                                    <div dir="ltr">
                                                      <div>
                                                        <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr"><br>
                                                          Sincerely,<br>
                                                          Artem<br>
                                                          <br>
                                                          --<br>
                                                          Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
                                                          Mirror</a><span style="font-size:12.8000001907349px">, Illogical Robot LLC</span></div>
                                                          <div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
                                                          | <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                        </div>
                                                      </div>
                                                    </div>
                                                  </div>
                                                </div>
                                                <br>
                                                <div class="gmail_quote">On
                                                  Tue, Apr 10, 2018 at
                                                  9:56 AM, Artem
                                                  Russakovskii <span dir="ltr">&lt;<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>&gt;</span>
                                                  wrote:<br>
                                                  <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                    <div dir="ltr">Hi
                                                      Vlad,
                                                      <div><br>
                                                      </div>
                                                      <div>I actually
                                                        saw that post
                                                        already and even
                                                        asked a question
                                                        4 days ago (<a href="https://serverfault.com/questions/517775/glusterfs-direct-i-o-mode#comment1172497_540917" target="_blank">https://serverfault.com/quest<wbr>ions/517775/glusterfs-direct-i<wbr>-o-mode#comment1172497_540917</a>)<wbr>.
                                                        The accepted
                                                        answer also
                                                        seems to go
                                                        against your
                                                        suggestion to
                                                        enable
                                                        direct-io-mode
                                                        as it says it
                                                        should be
                                                        disabled for
                                                        better
                                                        performance when
                                                        used just for
                                                        file accesses.</div>
                                                      <div><br>
                                                      </div>
                                                      <div>It&#39;d be great
                                                        if someone from
                                                        the Gluster team
                                                        chimed in about
                                                        this thread.</div>
                                                    </div>
                                                    <div class="gmail_extra"><span><br clear="all">
                                                        <div>
                                                          <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420gmail_signature" data-smartmail="gmail_signature">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr"><br>
                                                          Sincerely,<br>
                                                          Artem<br>
                                                          <br>
                                                          --<br>
                                                          Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
                                                          Mirror</a><span style="font-size:12.8000001907349px">, Illogical Robot LLC</span></div>
                                                          <div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
                                                          | <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                        </div>
                                                        <br>
                                                      </span>
                                                      <div>
                                                        <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854h5">
                                                          <div class="gmail_quote">On
                                                          Tue, Apr 10,
                                                          2018 at 7:01
                                                          AM, Vlad
                                                          Kopylov <span dir="ltr">&lt;<a href="mailto:vladkopy@gmail.com" target="_blank">vladkopy@gmail.com</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>Wish I
                                                          knew or was
                                                          able to get
                                                          detailed
                                                          description of
                                                          those options
                                                          myself.<br>
                                                          </div>
                                                          <div>here is
                                                          direct-io-mode 
                                                          <a href="https://serverfault.com/questions/517775/glusterfs-direct-i-o-mode" target="_blank">https://serverfault.com/questi<wbr>ons/517775/glusterfs-direct-i-<wbr>o-mode</a><br>
                                                          </div>
                                                          Same as you I
                                                          ran tests on a
                                                          large volume
                                                          of files,
                                                          finding that
                                                          main delays
                                                          are in
                                                          attribute
                                                          calls, ending
                                                          up with those
                                                          mount options
                                                          to add
                                                          performance.<br>
                                                          </div>
                                                          <div>I
                                                          discovered
                                                          those options
                                                          through
                                                          basically
                                                          googling this
                                                          user list with
                                                          people sharing
                                                          their tests.<br>
                                                          </div>
                                                          Not sure I
                                                          would share
                                                          your optimism,
                                                          and rather
                                                          then going up
                                                          I downgraded
                                                          to 3.12 and
                                                          have no dir
                                                          view issue
                                                          now. Though I
                                                          had to
                                                          recreate the
                                                          cluster and
                                                          had to re-add
                                                          bricks with
                                                          existing data.<br>
                                                          </div>
                                                          <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420HOEnZb">
                                                          <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420h5">
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Tue, Apr 10,
                                                          2018 at 1:47
                                                          AM, Artem
                                                          Russakovskii <span dir="ltr">&lt;<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div dir="ltr">Hi
                                                          Vlad,
                                                          <div><br>
                                                          </div>
                                                          <div>I&#39;m using
                                                          only
                                                          localhost:
                                                          mounts.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Can you
                                                          please explain
                                                          what effect
                                                          each option
                                                          has on
                                                          performance
                                                          issues shown
                                                          in my posts?
                                                          &quot;negative-timeout=10,attribute<wbr>-timeout=30,fopen-keep-cache,d<wbr>irect-io-mode=enable,fetch-att<wbr>empts=5&quot;
                                                          From what I
                                                          remember,
                                                          direct-io-mode=enable
                                                          didn&#39;t make a
                                                          difference in
                                                          my tests, but
                                                          I suppose I
                                                          can try again.
                                                          The
                                                          explanations
                                                          about
                                                          direct-io-mode
                                                          are quite
                                                          confusing on
                                                          the web in
                                                          various
                                                          guides, saying
                                                          enabling it
                                                          could make
                                                          performance
                                                          worse in some
                                                          situations and
                                                          better in
                                                          others due to
                                                          OS file cache.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>There are
                                                          also these
                                                          gluster volume
                                                          settings,
                                                          adding to the
                                                          confusion:</div>
                                                          <div>
                                                          <div>Option:
                                                          performance.strict-o-direct</div>
                                                          <div>Default
                                                          Value: off</div>
                                                          <div>Description:
                                                          This option
                                                          when set to
                                                          off, ignores
                                                          the O_DIRECT
                                                          flag.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Option:
                                                          performance.nfs.strict-o-direc<wbr>t</div>
                                                          <div>Default
                                                          Value: off</div>
                                                          <div>Description:
                                                          This option
                                                          when set to
                                                          off, ignores
                                                          the O_DIRECT
                                                          flag.</div>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Re: 4.0.
                                                          I moved to 4.0
                                                          after finding
                                                          out that it
                                                          fixes the
                                                          disappearing
                                                          dirs bug
                                                          related to
                                                          cluster.readdir-optimize
                                                          if you
                                                          remember (<a href="http://lists.gluster.org/pipermail/gluster-users/2018-April/033830.html" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2018-April<wbr>/033830.html</a>).
                                                          I was already
                                                          on 3.13 by
                                                          then, and 4.0
                                                          resolved the
                                                          issue. It&#39;s
                                                          been stable
                                                          for me so far,
                                                          thankfully.</div>
                                                          <div class="gmail_extra"><span><br clear="all">
                                                          <div>
                                                          <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114gmail_signature" data-smartmail="gmail_signature">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr"><br>
                                                          Sincerely,<br>
                                                          Artem<br>
                                                          <br>
                                                          --<br>
                                                          Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
                                                          Mirror</a><span style="font-size:12.8000001907349px">, Illogical Robot LLC</span></div>
                                                          <div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
                                                          | <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          </span>
                                                          <div>
                                                          <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111h5">
                                                          <div class="gmail_quote">On
                                                          Mon, Apr 9,
                                                          2018 at 10:38
                                                          PM, Vlad
                                                          Kopylov <span dir="ltr">&lt;<a href="mailto:vladkopy@gmail.com" target="_blank">vladkopy@gmail.com</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>you
                                                          definitely
                                                          need mount
                                                          options to
                                                          /etc/fstab<br>
                                                          </div>
                                                          use ones from
                                                          here <a href="http://lists.gluster.org/pipermail/gluster-users/2018-April/033811.html" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2018-April/<wbr>033811.html</a><br>
                                                          <br>
                                                          </div>
                                                          <div>I went on
                                                          with using
                                                          local mounts
                                                          to achieve
                                                          performance as
                                                          well<br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          Also, 3.12 or
                                                          3.10 branches
                                                          would be
                                                          preferable for
                                                          production <br>
                                                          </div>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">
                                                          <div>
                                                          <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114h5">On
                                                          Fri, Apr 6,
                                                          2018 at 4:12
                                                          AM, Artem
                                                          Russakovskii <span dir="ltr">&lt;<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>&gt;</span>
                                                          wrote:<br>
                                                          </div>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div>
                                                          <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114h5">
                                                          <div dir="ltr">Hi
                                                          again,
                                                          <div><br>
                                                          </div>
                                                          <div>I&#39;d like
                                                          to expand on
                                                          the
                                                          performance
                                                          issues and
                                                          plead for
                                                          help. Here&#39;s
                                                          one case which
                                                          shows these
                                                          odd hiccups: <a href="https://i.imgur.com/CXBPjTK.gifv" target="_blank">https://i.imgur.com/C<wbr>XBPjTK.gifv</a>.
                                                          <div><br>
                                                          </div>
                                                          <div>In this
                                                          GIF where I
                                                          switch <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">back
                                                          and forth</span>
                                                          between copy
                                                          operations on
                                                          2 servers, I&#39;m
                                                          copying a 10GB
                                                          dir full of
                                                          .apk and image
                                                          files.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>On server
                                                          &quot;hive&quot; I&#39;m
                                                          copying
                                                          straight from
                                                          the main disk
                                                          to an attached
                                                          volume block
                                                          (xfs). As you
                                                          can see, the
                                                          transfers are
                                                          relatively
                                                          speedy and
                                                          don&#39;t hiccup.</div>
                                                          <div>On server
                                                          &quot;citadel&quot; I&#39;m
                                                          copying the
                                                          same set of
                                                          data to a
                                                          4-replicate
                                                          gluster which
                                                          uses block
                                                          storage as a
                                                          brick. As you
                                                          can see,
                                                          performance is
                                                          much worse,
                                                          and there are
                                                          frequent
                                                          pauses for
                                                          many seconds
                                                          where nothing
                                                          seems to be
                                                          happening -
                                                          just freezes.<br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>All 4
                                                          servers have
                                                          the same
                                                          specs, and all
                                                          of them have
                                                          performance
                                                          issues with
                                                          gluster and no
                                                          such issues
                                                          when raw xfs
                                                          block storage
                                                          is used.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>hive has
                                                          long finished
                                                          copying the
                                                          data, while
                                                          citadel is
                                                          barely
                                                          chugging along
                                                          and is
                                                          expected to
                                                          take probably
                                                          half an hour
                                                          to an hour. I
                                                          have over 1TB
                                                          of data to
                                                          migrate, at
                                                          which point if
                                                          we went live,
                                                          I&#39;m not even
                                                          sure gluster
                                                          would be able
                                                          to keep up
                                                          instead of
                                                          bringing the
                                                          machines and
                                                          services down.</div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Here&#39;s
                                                          the cluster
                                                          config, though
                                                          it didn&#39;t seem
                                                          to make any
                                                          difference
                                                          performance-wise
                                                          before I
                                                          applied the
                                                          customizations
                                                          vs after.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>
                                                          <div>Volume
                                                          Name:
                                                          apkmirror_data1</div>
                                                          <div>Type:
                                                          Replicate</div>
                                                          <div>Volume
                                                          ID:
                                                          11ecee7e-d4f8-497a-9994-ceb144<wbr>d6841e</div>
                                                          <div>Status:
                                                          Started</div>
                                                          <div>Snapshot
                                                          Count: 0</div>
                                                          <div>Number of
                                                          Bricks: 1 x 4
                                                          = 4</div>
                                                          <div>Transport-type:
                                                          tcp</div>
                                                          <div>Bricks:</div>
                                                          <div>Brick1:
                                                          nexus2:/mnt/nexus2_block1/apkm<wbr>irror_data1</div>
                                                          <div>Brick2:
                                                          forge:/mnt/forge_block1/apkmir<wbr>ror_data1</div>
                                                          <div>Brick3:
                                                          hive:/mnt/hive_block1/apkmirro<wbr>r_data1</div>
                                                          <div>Brick4:
                                                          citadel:/mnt/citadel_block1/ap<wbr>kmirror_data1</div>
                                                          <div>Options
                                                          Reconfigured:</div>
                                                          <div>cluster.quorum-count:
                                                          1</div>
                                                          <div>cluster.quorum-type:
                                                          fixed</div>
                                                          <div>network.ping-timeout:
                                                          5</div>
                                                          <div>network.remote-dio:
                                                          enable</div>
                                                          <div>performance.rda-cache-limit:
                                                          256MB</div>
                                                          <div>performance.readdir-ahead:
                                                          on</div>
                                                          <div>performance.parallel-readdir:
                                                          on</div>
                                                          <div>network.inode-lru-limit:
                                                          500000</div>
                                                          <div>performance.md-cache-timeout:
                                                          600</div>
                                                          <div>performance.cache-invalidation<wbr>:
                                                          on</div>
                                                          <div>performance.stat-prefetch:
                                                          on</div>
                                                          <div>features.cache-invalidation-ti<wbr>meout:
                                                          600</div>
                                                          <div>features.cache-invalidation:
                                                          on</div>
                                                          <div>cluster.readdir-optimize:
                                                          on</div>
                                                          <div>performance.io-thread-count:
                                                          32</div>
                                                          <div>server.event-threads:
                                                          4</div>
                                                          <div>client.event-threads:
                                                          4</div>
                                                          <div>performance.read-ahead:
                                                          off</div>
                                                          <div>cluster.lookup-optimize:
                                                          on</div>
                                                          <div>performance.cache-size:
                                                          1GB</div>
                                                          <div>cluster.self-heal-daemon:
                                                          enable</div>
                                                          <div>transport.address-family:
                                                          inet</div>
                                                          <div>nfs.disable:
                                                          on</div>
                                                          <div>performance.client-io-threads:
                                                          on</div>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>The
                                                          mounts are
                                                          done as
                                                          follows in
                                                          /etc/fstab:</div>
                                                          <div>/dev/disk/by-id/scsi-0Linode_V<wbr>olume_citadel_block1
/mnt/citadel_block1 xfs defaults 0 2<br>
                                                          </div>
                                                          <div>localhost:/apkmirror_data1
/mnt/apkmirror_data1 glusterfs defaults,_netdev 0 0<br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>I&#39;m
                                                          really not
                                                          sure if
                                                          direct-io-mode
                                                          mount tweaks
                                                          would do
                                                          anything here,
                                                          what the value
                                                          should be set
                                                          to, and what
                                                          it is by
                                                          default.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>The OS is
                                                          OpenSUSE 42.3,
                                                          64-bit. 80GB
                                                          of RAM, 20
                                                          CPUs, hosted
                                                          by Linode.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>I&#39;d
                                                          really
                                                          appreciate any
                                                          help in the
                                                          matter. </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Thank
                                                          you.</div>
                                                          </div>
                                                          </div>
                                                          <div class="gmail_extra"><span><br clear="all">
                                                          <div>
                                                          <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114m_-7640979976631609660m_7374879952101670984gmail_signature" data-smartmail="gmail_signature">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr"><br>
                                                          Sincerely,<br>
                                                          Artem<br>
                                                          <br>
                                                          --<br>
                                                          Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
                                                          Mirror</a><span style="font-size:12.8000001907349px">, Illogical Robot LLC</span></div>
                                                          <div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
                                                          | <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          </span>
                                                          <div>
                                                          <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114m_-7640979976631609660h5">
                                                          <div class="gmail_quote">On
                                                          Thu, Apr 5,
                                                          2018 at 11:13
                                                          PM, Artem
                                                          Russakovskii <span dir="ltr">&lt;<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div dir="ltr">Hi,
                                                          <div><br>
                                                          </div>
                                                          <div>I&#39;m
                                                          trying to
                                                          squeeze
                                                          performance
                                                          out of gluster
                                                          on 4 80GB RAM
                                                          20-CPU
                                                          machines where
                                                          Gluster runs
                                                          on attached
                                                          block storage
                                                          (Linode) in (4
                                                          replicate
                                                          bricks), and
                                                          so far
                                                          everything I
                                                          tried results
                                                          in sub-optimal
                                                          performance.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>There are
                                                          many files -
                                                          mostly images,
                                                          several
                                                          million - and
                                                          many
                                                          operations
                                                          take minutes,
                                                          copying
                                                          multiple files
                                                          (even if
                                                          they&#39;re small)
                                                          suddenly
                                                          freezes up for
                                                          seconds at a
                                                          time, then
                                                          continues,
                                                          iostat
                                                          frequently
                                                          shows large
                                                          r_await and
                                                          w_awaits with
                                                          100%
                                                          utilization
                                                          for the
                                                          attached block
                                                          device, etc.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>But
                                                          anyway, there
                                                          are many
                                                          guides out
                                                          there for
                                                          small-file
                                                          performance
                                                          improvements,
                                                          but more
                                                          explanation is
                                                          needed, and I
                                                          think more
                                                          tweaks should
                                                          be possible.<br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>My
                                                          question today
                                                          is
                                                          about performance.cache-size.
                                                          Is this a size
                                                          of cache in
                                                          RAM? If so,
                                                          how do I view
                                                          the current
                                                          cache size to
                                                          see if it gets
                                                          full and I
                                                          should
                                                          increase its
                                                          size? Is it
                                                          advisable to
                                                          bump it up if
                                                          I have many
                                                          tens of gigs
                                                          of RAM free? </div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>More
                                                          generally, in
                                                          the last 2
                                                          months since I
                                                          first started
                                                          working with
                                                          gluster and
                                                          set a
                                                          production
                                                          system live,
                                                          I&#39;ve been
                                                          feeling
                                                          frustrated
                                                          because
                                                          Gluster has a
                                                          lot of
                                                          poorly-documented
                                                          and confusing
                                                          options. I
                                                          really wish
                                                          documentation
                                                          could be
                                                          improved with
                                                          examples and
                                                          better
                                                          explanations.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Specifically,
                                                          it&#39;d be
                                                          absolutely
                                                          amazing if the
                                                          docs offered a
                                                          strategy for
                                                          setting each
                                                          value and ways
                                                          of determining
                                                          more optimal
                                                          values. For
                                                          example,
                                                          for performance.cache-size,
                                                          if it said
                                                          something like
                                                          &quot;run command
                                                          abc to see
                                                          your current
                                                          cache size,
                                                          and if it&#39;s
                                                          hurting, up
                                                          it, but be
                                                          aware that
                                                          it&#39;s limited
                                                          by RAM,&quot; it&#39;d
                                                          be already a
                                                          huge
                                                          improvement to
                                                          the docs. And
                                                          so on with
                                                          other options.</div>
                                                          <div>
                                                          <div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
                                                          </div>
                                                          <div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
                                                          </div>
                                                          <div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
                                                          </div>
                                                          <div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">The
                                                          gluster team
                                                          is quite
                                                          helpful on
                                                          this mailing
                                                          list, but in a
                                                          reactive
                                                          rather than
                                                          proactive way.
                                                          Perhaps it&#39;s
                                                          tunnel vision
                                                          once you&#39;ve
                                                          worked on a
                                                          project for so
                                                          long where
                                                          less technical
                                                          explanations
                                                          and even
                                                          proper
                                                          documentation
                                                          of options
                                                          takes a back
                                                          seat, but I
                                                          encourage you
                                                          to be more
                                                          proactive
                                                          about helping
                                                          us understand
                                                          and optimize
                                                          Gluster.</div>
                                                          <div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
                                                          </div>
                                                          <div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">Thank
                                                          you.</div>
                                                          <div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
                                                          </div>
                                                          <div>
                                                          <div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114m_-7640979976631609660m_7374879952101670984m_1420630595765814996gmail_signature">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">Sincerely,<br>
                                                          Artem<br>
                                                          <br>
                                                          --<br>
                                                          Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8px" target="_blank">APK
                                                          Mirror</a><span style="font-size:12.8px">, Illogical Robot LLC</span></div>
                                                          <div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
                                                          | <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          <span>______________________________<wbr>_________________<br>
                                                          Gluster-users
                                                          mailing list<br>
                                                          <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
                                                          <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
                                                          </span></blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                        </div>
                                                      </div>
                                                    </div>
                                                  </blockquote>
                                                </div>
                                                <br>
                                              </div>
                                              <br>
                                              <fieldset class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854mimeAttachmentHeader"></fieldset>
                                              <br>
                                              <pre>______________________________<wbr>_________________
Gluster-users mailing list
<a class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854moz-txt-link-freetext" href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a></pre>
                                            </blockquote>
                                            <br>
                                          </div>
                                        </div>
                                      </div>
                                    </blockquote>
                                  </div>
                                  <br>
                                </div>
                              </blockquote>
                              <br>
                            </div>
                          </div>
                        </div>
                      </blockquote>
                    </div>
                    <br>
                  </div>
                </div>
              </div>
            </blockquote>
          </div>
          <br>
        </div>
      </div>
    </blockquote>
    <br>
  </div>

</blockquote></div><br></div>