<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p><br>
    </p>
    <br>
    <div class="moz-cite-prefix">On 04/18/2018 10:35 AM, Artem
      Russakovskii wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAD+dzQfp0AhjTzqYsdpGvv_-pH=gWn2z-2wBYQNb2BiCrGzfCw@mail.gmail.com">
      <div dir="ltr">Hi Ravi,
        <div><br>
        </div>
        <div>Could you please expand on how these would help?</div>
        <div><br>
        </div>
        <div>By forcing full here, we move the logic from the CPU to
          network, thus decreasing CPU utilization, is that right?</div>
      </div>
    </blockquote>
    Yes, 'diff' employs the rchecksum FOP which does a sha256  checksum
    which can consume CPU. So yes it is sort of shifting the load from
    CPU to the network. But if your average file size is small, it would
    make sense to copy the entire file instead of computing checksums.<br>
    <br>
    <blockquote type="cite"
cite="mid:CAD+dzQfp0AhjTzqYsdpGvv_-pH=gWn2z-2wBYQNb2BiCrGzfCw@mail.gmail.com">
      <div dir="ltr">
        <div> This is assuming the CPU and disk utilization are caused
          by the differ and not by lstat and other calls or something.</div>
        <div>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px
            0.8ex;border-left:1px solid
            rgb(204,204,204);padding-left:1ex">Option:
            cluster.data-self-heal-algorithm<br>
            Default Value: (null)<br>
            Description: Select between "full", "diff". The "full"
            algorithm copies the entire file from source to sink. The
            "diff" algorithm copies to sink only those blocks whose
            checksums don't match with those of source. If no option is
            configured the option is chosen dynamically as follows: If
            the file does not exist on one of the sinks or empty file
            exists or if the source file size is about the same as page
            size the entire file will be read and written i.e "full"
            algo, otherwise "diff" algo is chosen.</blockquote>
        </div>
        <div><br>
        </div>
        <div>I really have no idea what this means and how/why it would
          help. Any more info on this option?</div>
      </div>
    </blockquote>
    <br>
<a class="moz-txt-link-freetext" href="https://github.com/gluster/glusterfs-specs/blob/master/done/GlusterFS%203.8/granular-entry-self-healing.md">https://github.com/gluster/glusterfs-specs/blob/master/done/GlusterFS%203.8/granular-entry-self-healing.md</a>
    should help.<br>
    Regards,<br>
    Ravi<br>
    <br>
    <blockquote type="cite"
cite="mid:CAD+dzQfp0AhjTzqYsdpGvv_-pH=gWn2z-2wBYQNb2BiCrGzfCw@mail.gmail.com">
      <div dir="ltr">
        <div>
          <blockquote class="gmail_quote" style="margin:0px 0px 0px
            0.8ex;border-left:1px solid
            rgb(204,204,204);padding-left:1ex">Option:
            cluster.granular-entry-heal<br>
            Default Value: no<br>
            Description: If this option is enabled, self-heal will
            resort to granular way of recording changelogs and doing
            entry self-heal.</blockquote>
        </div>
        <div><br>
        </div>
        <div>Thank you.</div>
      </div>
      <div class="gmail_extra"><br clear="all">
        <div>
          <div class="gmail_signature" data-smartmail="gmail_signature">
            <div dir="ltr">
              <div>
                <div dir="ltr">
                  <div>
                    <div dir="ltr">
                      <div>
                        <div dir="ltr">
                          <div>
                            <div dir="ltr"><br>
                              Sincerely,<br>
                              Artem<br>
                              <br>
                              --<br>
                              Founder, <a
                                href="http://www.androidpolice.com"
                                target="_blank" moz-do-not-send="true">Android
                                Police</a>, <a
                                href="http://www.apkmirror.com/"
                                style="font-size:12.8000001907349px"
                                target="_blank" moz-do-not-send="true">APK
                                Mirror</a><span
                                style="font-size:12.8000001907349px">,
                                Illogical Robot LLC</span></div>
                            <div dir="ltr"><a href="http://beerpla.net/"
                                target="_blank" moz-do-not-send="true">beerpla.net</a>
                              | <a
                                href="https://plus.google.com/+ArtemRussakovskii"
                                target="_blank" moz-do-not-send="true">+ArtemRussakovskii</a>
                              | <a href="http://twitter.com/ArtemR"
                                target="_blank" moz-do-not-send="true">@ArtemR</a><br>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </div>
        </div>
        <br>
        <div class="gmail_quote">On Tue, Apr 17, 2018 at 9:58 PM,
          Ravishankar N <span dir="ltr">&lt;<a
              href="mailto:ravishankar@redhat.com" target="_blank"
              moz-do-not-send="true">ravishankar@redhat.com</a>&gt;</span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div text="#000000" bgcolor="#FFFFFF">
              <div>
                <div class="h5"> <br>
                  <div class="m_979837676701406854moz-cite-prefix">On
                    04/18/2018 10:14 AM, Artem Russakovskii wrote:<br>
                  </div>
                  <blockquote type="cite">
                    <div dir="ltr">Following up here on a related and
                      very serious for us issue.
                      <div><br>
                      </div>
                      <div>I took down one of the 4 replicate gluster
                        servers for maintenance today. There are 2
                        gluster volumes totaling about 600GB. Not that
                        much data. After the server comes back online,
                        it starts auto healing and pretty much all
                        operations on gluster freeze for many minutes.</div>
                      <div><br>
                      </div>
                      <div>For example, I was trying to run an ls -alrt
                        in a folder with 7300 files, and it took a good
                        15-20 minutes before returning.</div>
                      <div><br>
                      </div>
                      <div>During this time, I can see iostat show 100%
                        utilization on the brick, heal status takes many
                        minutes to return, glusterfsd uses up tons of
                        CPU (I saw it spike to 600%). gluster already
                        has massive performance issues for me, but
                        healing after a 4-hour downtime is on another
                        level of bad perf.</div>
                      <div><br>
                      </div>
                      <div>For example, this command took many minutes
                        to run:</div>
                      <div><br>
                      </div>
                      <div>
                        <div>gluster volume heal androidpolice_data3
                          info summary</div>
                        <div>Brick nexus2:/mnt/nexus2_block4/<wbr>androidpolice_data3</div>
                        <div>Status: Connected</div>
                        <div>Total Number of entries: 91</div>
                        <div>Number of entries in heal pending: 90</div>
                        <div>Number of entries in split-brain: 0</div>
                        <div>Number of entries possibly healing: 1</div>
                        <div><br>
                        </div>
                        <div>Brick forge:/mnt/forge_block4/<wbr>androidpolice_data3</div>
                        <div>Status: Connected</div>
                        <div>Total Number of entries: 87</div>
                        <div>Number of entries in heal pending: 86</div>
                        <div>Number of entries in split-brain: 0</div>
                        <div>Number of entries possibly healing: 1</div>
                        <div><br>
                        </div>
                        <div>Brick hive:/mnt/hive_block4/<wbr>androidpolice_data3</div>
                        <div>Status: Connected</div>
                        <div>Total Number of entries: 87</div>
                        <div>Number of entries in heal pending: 86</div>
                        <div>Number of entries in split-brain: 0</div>
                        <div>Number of entries possibly healing: 1</div>
                        <div><br>
                        </div>
                        <div>Brick citadel:/mnt/citadel_block4/<wbr>androidpolice_data3</div>
                        <div>Status: Connected</div>
                        <div>Total Number of entries: 0</div>
                        <div>Number of entries in heal pending: 0</div>
                        <div>Number of entries in split-brain: 0</div>
                        <div>Number of entries possibly healing: 0</div>
                      </div>
                      <div><br>
                      </div>
                      <div><br>
                      </div>
                      <div>Statistics showed a diminishing number of
                        failed heals:</div>
                      <div>...</div>
                      <div>
                        <div>Ending time of crawl: Tue Apr 17 21:13:08
                          2018</div>
                        <div><br>
                        </div>
                        <div>Type of crawl: INDEX</div>
                        <div>No. of entries healed: 2</div>
                        <div>No. of entries in split-brain: 0</div>
                        <div>No. of heal failed entries: 102</div>
                        <div><br>
                        </div>
                        <div>Starting time of crawl: Tue Apr 17 21:13:09
                          2018</div>
                        <div><br>
                        </div>
                        <div>Ending time of crawl: Tue Apr 17 21:14:30
                          2018</div>
                        <div><br>
                        </div>
                        <div>Type of crawl: INDEX</div>
                        <div>No. of entries healed: 4</div>
                        <div>No. of entries in split-brain: 0</div>
                        <div>No. of heal failed entries: 91</div>
                        <div><br>
                        </div>
                        <div>Starting time of crawl: Tue Apr 17 21:14:31
                          2018</div>
                        <div><br>
                        </div>
                        <div>Ending time of crawl: Tue Apr 17 21:15:34
                          2018</div>
                        <div><br>
                        </div>
                        <div>Type of crawl: INDEX</div>
                        <div>No. of entries healed: 0</div>
                        <div>No. of entries in split-brain: 0</div>
                        <div>No. of heal failed entries: 88</div>
                      </div>
                      <div>...</div>
                      <div><br>
                      </div>
                      <div>Eventually, everything heals and goes back to
                        at least where the roof isn't on fire anymore.</div>
                      <div><br>
                      </div>
                      <div>The server stats and volume options were
                        given in one of the previous replies to this
                        thread.</div>
                      <div><br>
                      </div>
                      <div>Any ideas or things I could run and show the
                        output of to help diagnose? I'm also very open
                        to working with someone on the team on a live
                        debugging session if there's interest.</div>
                    </div>
                  </blockquote>
                  <br>
                </div>
              </div>
              It is likely that self-heal is causing the CPU spike due
              to the flood of lookups/ locks and checksum fops that the
              self-heal-daemon sends to the bricks.<br>
              There's a script to control shd's cpu usage using cgroups.
              That should help in regulating self-heal traffic: <a
                class="m_979837676701406854moz-txt-link-freetext"
                href="https://review.gluster.org/#/c/18404/"
                target="_blank" moz-do-not-send="true">https://review.gluster.org/#/<wbr>c/18404/</a>
              (see extras/control-cpu-load.sh)<br>
              Other self-heal related volume options that you could
              change are setting 'cluster.data-self-heal-<wbr>algorithm'
              to 'full' and 'granular-entry-heal' to 'enable'.  `gluster
              volume set help` should give you more information about
              these options.<br>
              Thanks,<br>
              Ravi
              <div>
                <div class="h5"><br>
                  <br>
                  <blockquote type="cite">
                    <div dir="ltr">
                      <div><br>
                      </div>
                      <div>Thank you.</div>
                    </div>
                    <div class="gmail_extra"><br clear="all">
                      <div>
                        <div class="m_979837676701406854gmail_signature"
                          data-smartmail="gmail_signature">
                          <div dir="ltr">
                            <div>
                              <div dir="ltr">
                                <div>
                                  <div dir="ltr">
                                    <div>
                                      <div dir="ltr">
                                        <div>
                                          <div dir="ltr"><br>
                                            Sincerely,<br>
                                            Artem<br>
                                            <br>
                                            --<br>
                                            Founder, <a
                                              href="http://www.androidpolice.com"
                                              target="_blank"
                                              moz-do-not-send="true">Android
                                              Police</a>, <a
                                              href="http://www.apkmirror.com/"
style="font-size:12.8000001907349px" target="_blank"
                                              moz-do-not-send="true">APK
                                              Mirror</a><span
                                              style="font-size:12.8000001907349px">,
                                              Illogical Robot LLC</span></div>
                                          <div dir="ltr"><a
                                              href="http://beerpla.net/"
                                              target="_blank"
                                              moz-do-not-send="true">beerpla.net</a>
                                            | <a
                                              href="https://plus.google.com/+ArtemRussakovskii"
                                              target="_blank"
                                              moz-do-not-send="true">+ArtemRussakovskii</a>
                                            | <a
                                              href="http://twitter.com/ArtemR"
                                              target="_blank"
                                              moz-do-not-send="true">@ArtemR</a><br>
                                          </div>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                      <br>
                      <div class="gmail_quote">On Tue, Apr 10, 2018 at
                        9:56 AM, Artem Russakovskii <span dir="ltr">&lt;<a
                            href="mailto:archon810@gmail.com"
                            target="_blank" moz-do-not-send="true">archon810@gmail.com</a>&gt;</span>
                        wrote:<br>
                        <blockquote class="gmail_quote" style="margin:0
                          0 0 .8ex;border-left:1px #ccc
                          solid;padding-left:1ex">
                          <div dir="ltr">Hi Vlad,
                            <div><br>
                            </div>
                            <div>I actually saw that post already and
                              even asked a question 4 days ago (<a
href="https://serverfault.com/questions/517775/glusterfs-direct-i-o-mode#comment1172497_540917"
                                target="_blank" moz-do-not-send="true">https://serverfault.com/quest<wbr>ions/517775/glusterfs-direct-<wbr>i-o-mode#comment1172497_540917</a><wbr>).
                              The accepted answer also seems to go
                              against your suggestion to enable
                              direct-io-mode as it says it should be
                              disabled for better performance when used
                              just for file accesses.</div>
                            <div><br>
                            </div>
                            <div>It'd be great if someone from the
                              Gluster team chimed in about this thread.</div>
                          </div>
                          <div class="gmail_extra"><span><br clear="all">
                              <div>
                                <div
                                  class="m_979837676701406854m_3369104992988144420gmail_signature"
                                  data-smartmail="gmail_signature">
                                  <div dir="ltr">
                                    <div>
                                      <div dir="ltr">
                                        <div>
                                          <div dir="ltr">
                                            <div>
                                              <div dir="ltr">
                                                <div>
                                                  <div dir="ltr"><br>
                                                    Sincerely,<br>
                                                    Artem<br>
                                                    <br>
                                                    --<br>
                                                    Founder, <a
                                                      href="http://www.androidpolice.com"
                                                      target="_blank"
                                                      moz-do-not-send="true">Android
                                                      Police</a>, <a
                                                      href="http://www.apkmirror.com/"
style="font-size:12.8000001907349px" target="_blank"
                                                      moz-do-not-send="true">APK
                                                      Mirror</a><span
                                                      style="font-size:12.8000001907349px">,
                                                      Illogical Robot
                                                      LLC</span></div>
                                                  <div dir="ltr"><a
                                                      href="http://beerpla.net/"
                                                      target="_blank"
                                                      moz-do-not-send="true">beerpla.net</a>
                                                    | <a
                                                      href="https://plus.google.com/+ArtemRussakovskii"
                                                      target="_blank"
                                                      moz-do-not-send="true">+ArtemRussakovskii</a>
                                                    | <a
                                                      href="http://twitter.com/ArtemR"
                                                      target="_blank"
                                                      moz-do-not-send="true">@ArtemR</a><br>
                                                  </div>
                                                </div>
                                              </div>
                                            </div>
                                          </div>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                              <br>
                            </span>
                            <div>
                              <div class="m_979837676701406854h5">
                                <div class="gmail_quote">On Tue, Apr 10,
                                  2018 at 7:01 AM, Vlad Kopylov <span
                                    dir="ltr">&lt;<a
                                      href="mailto:vladkopy@gmail.com"
                                      target="_blank"
                                      moz-do-not-send="true">vladkopy@gmail.com</a>&gt;</span>
                                  wrote:<br>
                                  <blockquote class="gmail_quote"
                                    style="margin:0 0 0
                                    .8ex;border-left:1px #ccc
                                    solid;padding-left:1ex">
                                    <div dir="ltr">
                                      <div>
                                        <div>Wish I knew or was able to
                                          get detailed description of
                                          those options myself.<br>
                                        </div>
                                        <div>here is direct-io-mode  <a
href="https://serverfault.com/questions/517775/glusterfs-direct-i-o-mode"
                                            target="_blank"
                                            moz-do-not-send="true">https://serverfault.com/questi<wbr>ons/517775/glusterfs-direct-i-<wbr>o-mode</a><br>
                                        </div>
                                        Same as you I ran tests on a
                                        large volume of files, finding
                                        that main delays are in
                                        attribute calls, ending up with
                                        those mount options to add
                                        performance.<br>
                                      </div>
                                      <div>I discovered those options
                                        through basically googling this
                                        user list with people sharing
                                        their tests.<br>
                                      </div>
                                      Not sure I would share your
                                      optimism, and rather then going up
                                      I downgraded to 3.12 and have no
                                      dir view issue now. Though I had
                                      to recreate the cluster and had to
                                      re-add bricks with existing data.<br>
                                    </div>
                                    <div
                                      class="m_979837676701406854m_3369104992988144420HOEnZb">
                                      <div
                                        class="m_979837676701406854m_3369104992988144420h5">
                                        <div class="gmail_extra"><br>
                                          <div class="gmail_quote">On
                                            Tue, Apr 10, 2018 at 1:47
                                            AM, Artem Russakovskii <span
                                              dir="ltr">&lt;<a
                                                href="mailto:archon810@gmail.com"
                                                target="_blank"
                                                moz-do-not-send="true">archon810@gmail.com</a>&gt;</span>
                                            wrote:<br>
                                            <blockquote
                                              class="gmail_quote"
                                              style="margin:0 0 0
                                              .8ex;border-left:1px #ccc
                                              solid;padding-left:1ex">
                                              <div dir="ltr">Hi Vlad,
                                                <div><br>
                                                </div>
                                                <div>I'm using only
                                                  localhost: mounts.</div>
                                                <div><br>
                                                </div>
                                                <div>Can you please
                                                  explain what effect
                                                  each option has on
                                                  performance issues
                                                  shown in my posts?
                                                  "negative-timeout=10,attribute<wbr>-timeout=30,fopen-keep-cache,d<wbr>irect-io-mode=enable,fetch-att<wbr>empts=5"
                                                  From what I remember,
                                                  direct-io-mode=enable
                                                  didn't make a
                                                  difference in my
                                                  tests, but I suppose I
                                                  can try again. The
                                                  explanations about
                                                  direct-io-mode are
                                                  quite confusing on the
                                                  web in various guides,
                                                  saying enabling it
                                                  could make performance
                                                  worse in some
                                                  situations and better
                                                  in others due to OS
                                                  file cache.</div>
                                                <div><br>
                                                </div>
                                                <div>There are also
                                                  these gluster volume
                                                  settings, adding to
                                                  the confusion:</div>
                                                <div>
                                                  <div>Option:
                                                    performance.strict-o-direct</div>
                                                  <div>Default Value:
                                                    off</div>
                                                  <div>Description: This
                                                    option when set to
                                                    off, ignores the
                                                    O_DIRECT flag.</div>
                                                  <div><br>
                                                  </div>
                                                  <div>Option:
                                                    performance.nfs.strict-o-direc<wbr>t</div>
                                                  <div>Default Value:
                                                    off</div>
                                                  <div>Description: This
                                                    option when set to
                                                    off, ignores the
                                                    O_DIRECT flag.</div>
                                                </div>
                                                <div><br>
                                                </div>
                                                <div>Re: 4.0. I moved to
                                                  4.0 after finding out
                                                  that it fixes the
                                                  disappearing dirs bug
                                                  related to
                                                  cluster.readdir-optimize
                                                  if you remember (<a
href="http://lists.gluster.org/pipermail/gluster-users/2018-April/033830.html"
                                                    target="_blank"
                                                    moz-do-not-send="true">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2018-April<wbr>/033830.html</a>).
                                                  I was already on 3.13
                                                  by then, and 4.0
                                                  resolved the issue.
                                                  It's been stable for
                                                  me so far, thankfully.</div>
                                                <div class="gmail_extra"><span><br
                                                      clear="all">
                                                    <div>
                                                      <div
class="m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114gmail_signature"
data-smartmail="gmail_signature">
                                                        <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr"><br>
                                                          Sincerely,<br>
                                                          Artem<br>
                                                          <br>
                                                          --<br>
                                                          Founder, <a
                                                          href="http://www.androidpolice.com"
target="_blank" moz-do-not-send="true">Android Police</a>, <a
                                                          href="http://www.apkmirror.com/"
style="font-size:12.8000001907349px" target="_blank"
                                                          moz-do-not-send="true">APK
                                                          Mirror</a><span
style="font-size:12.8000001907349px">, Illogical Robot LLC</span></div>
                                                          <div dir="ltr"><a
href="http://beerpla.net/" target="_blank" moz-do-not-send="true">beerpla.net</a>
                                                          | <a
                                                          href="https://plus.google.com/+ArtemRussakovskii"
target="_blank" moz-do-not-send="true">+ArtemRussakovskii</a> | <a
                                                          href="http://twitter.com/ArtemR"
target="_blank" moz-do-not-send="true">@ArtemR</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                        </div>
                                                      </div>
                                                    </div>
                                                    <br>
                                                  </span>
                                                  <div>
                                                    <div
class="m_979837676701406854m_3369104992988144420m_-2442537929453809111h5">
                                                      <div
                                                        class="gmail_quote">On
                                                        Mon, Apr 9, 2018
                                                        at 10:38 PM,
                                                        Vlad Kopylov <span
                                                          dir="ltr">&lt;<a
href="mailto:vladkopy@gmail.com" target="_blank" moz-do-not-send="true">vladkopy@gmail.com</a>&gt;</span>
                                                        wrote:<br>
                                                        <blockquote
                                                          class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>you
                                                          definitely
                                                          need mount
                                                          options to
                                                          /etc/fstab<br>
                                                          </div>
                                                          use ones from
                                                          here <a
href="http://lists.gluster.org/pipermail/gluster-users/2018-April/033811.html"
target="_blank" moz-do-not-send="true">http://lists.gluster.org/piper<wbr>mail/gluster-users/2018-April/<wbr>033811.html</a><br>
                                                          <br>
                                                          </div>
                                                          <div>I went on
                                                          with using
                                                          local mounts
                                                          to achieve
                                                          performance as
                                                          well<br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          Also, 3.12 or
                                                          3.10 branches
                                                          would be
                                                          preferable for
                                                          production <br>
                                                          </div>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <div
                                                          class="gmail_quote">
                                                          <div>
                                                          <div
class="m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114h5">On
                                                          Fri, Apr 6,
                                                          2018 at 4:12
                                                          AM, Artem
                                                          Russakovskii <span
                                                          dir="ltr">&lt;<a
href="mailto:archon810@gmail.com" target="_blank" moz-do-not-send="true">archon810@gmail.com</a>&gt;</span>
                                                          wrote:<br>
                                                          </div>
                                                          </div>
                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div>
                                                          <div
class="m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114h5">
                                                          <div dir="ltr">Hi
                                                          again,
                                                          <div><br>
                                                          </div>
                                                          <div>I'd like
                                                          to expand on
                                                          the
                                                          performance
                                                          issues and
                                                          plead for
                                                          help. Here's
                                                          one case which
                                                          shows these
                                                          odd hiccups: <a
href="https://i.imgur.com/CXBPjTK.gifv" target="_blank"
                                                          moz-do-not-send="true">https://i.imgur.com/C<wbr>XBPjTK.gifv</a>.
                                                          <div><br>
                                                          </div>
                                                          <div>In this
                                                          GIF where I
                                                          switch <span
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">back
                                                          and forth</span>
                                                          between copy
                                                          operations on
                                                          2 servers, I'm
                                                          copying a 10GB
                                                          dir full of
                                                          .apk and image
                                                          files.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>On server
                                                          "hive" I'm
                                                          copying
                                                          straight from
                                                          the main disk
                                                          to an attached
                                                          volume block
                                                          (xfs). As you
                                                          can see, the
                                                          transfers are
                                                          relatively
                                                          speedy and
                                                          don't hiccup.</div>
                                                          <div>On server
                                                          "citadel" I'm
                                                          copying the
                                                          same set of
                                                          data to a
                                                          4-replicate
                                                          gluster which
                                                          uses block
                                                          storage as a
                                                          brick. As you
                                                          can see,
                                                          performance is
                                                          much worse,
                                                          and there are
                                                          frequent
                                                          pauses for
                                                          many seconds
                                                          where nothing
                                                          seems to be
                                                          happening -
                                                          just freezes.<br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>All 4
                                                          servers have
                                                          the same
                                                          specs, and all
                                                          of them have
                                                          performance
                                                          issues with
                                                          gluster and no
                                                          such issues
                                                          when raw xfs
                                                          block storage
                                                          is used.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>hive has
                                                          long finished
                                                          copying the
                                                          data, while
                                                          citadel is
                                                          barely
                                                          chugging along
                                                          and is
                                                          expected to
                                                          take probably
                                                          half an hour
                                                          to an hour. I
                                                          have over 1TB
                                                          of data to
                                                          migrate, at
                                                          which point if
                                                          we went live,
                                                          I'm not even
                                                          sure gluster
                                                          would be able
                                                          to keep up
                                                          instead of
                                                          bringing the
                                                          machines and
                                                          services down.</div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Here's
                                                          the cluster
                                                          config, though
                                                          it didn't seem
                                                          to make any
                                                          difference
                                                          performance-wise
                                                          before I
                                                          applied the
                                                          customizations
                                                          vs after.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>
                                                          <div>Volume
                                                          Name:
                                                          apkmirror_data1</div>
                                                          <div>Type:
                                                          Replicate</div>
                                                          <div>Volume
                                                          ID:
                                                          11ecee7e-d4f8-497a-9994-ceb144<wbr>d6841e</div>
                                                          <div>Status:
                                                          Started</div>
                                                          <div>Snapshot
                                                          Count: 0</div>
                                                          <div>Number of
                                                          Bricks: 1 x 4
                                                          = 4</div>
                                                          <div>Transport-type:
                                                          tcp</div>
                                                          <div>Bricks:</div>
                                                          <div>Brick1:
                                                          nexus2:/mnt/nexus2_block1/apkm<wbr>irror_data1</div>
                                                          <div>Brick2:
                                                          forge:/mnt/forge_block1/apkmir<wbr>ror_data1</div>
                                                          <div>Brick3:
                                                          hive:/mnt/hive_block1/apkmirro<wbr>r_data1</div>
                                                          <div>Brick4:
                                                          citadel:/mnt/citadel_block1/ap<wbr>kmirror_data1</div>
                                                          <div>Options
                                                          Reconfigured:</div>
                                                          <div>cluster.quorum-count:
                                                          1</div>
                                                          <div>cluster.quorum-type:
                                                          fixed</div>
                                                          <div>network.ping-timeout:
                                                          5</div>
                                                          <div>network.remote-dio:
                                                          enable</div>
                                                          <div>performance.rda-cache-limit:
                                                          256MB</div>
                                                          <div>performance.readdir-ahead:
                                                          on</div>
                                                          <div>performance.parallel-readdir:
                                                          on</div>
                                                          <div>network.inode-lru-limit:
                                                          500000</div>
                                                          <div>performance.md-cache-timeout:
                                                          600</div>
                                                          <div>performance.cache-invalidation<wbr>:
                                                          on</div>
                                                          <div>performance.stat-prefetch:
                                                          on</div>
                                                          <div>features.cache-invalidation-ti<wbr>meout:
                                                          600</div>
                                                          <div>features.cache-invalidation:
                                                          on</div>
                                                          <div>cluster.readdir-optimize:
                                                          on</div>
                                                          <div>performance.io-thread-count:
                                                          32</div>
                                                          <div>server.event-threads:
                                                          4</div>
                                                          <div>client.event-threads:
                                                          4</div>
                                                          <div>performance.read-ahead:
                                                          off</div>
                                                          <div>cluster.lookup-optimize:
                                                          on</div>
                                                          <div>performance.cache-size:
                                                          1GB</div>
                                                          <div>cluster.self-heal-daemon:
                                                          enable</div>
                                                          <div>transport.address-family:
                                                          inet</div>
                                                          <div>nfs.disable:
                                                          on</div>
                                                          <div>performance.client-io-threads:
                                                          on</div>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>The
                                                          mounts are
                                                          done as
                                                          follows in
                                                          /etc/fstab:</div>
                                                          <div>/dev/disk/by-id/scsi-0Linode_V<wbr>olume_citadel_block1
/mnt/citadel_block1 xfs defaults 0 2<br>
                                                          </div>
                                                          <div>localhost:/apkmirror_data1
/mnt/apkmirror_data1 glusterfs defaults,_netdev 0 0<br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>I'm
                                                          really not
                                                          sure if
                                                          direct-io-mode
                                                          mount tweaks
                                                          would do
                                                          anything here,
                                                          what the value
                                                          should be set
                                                          to, and what
                                                          it is by
                                                          default.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>The OS is
                                                          OpenSUSE 42.3,
                                                          64-bit. 80GB
                                                          of RAM, 20
                                                          CPUs, hosted
                                                          by Linode.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>I'd
                                                          really
                                                          appreciate any
                                                          help in the
                                                          matter. </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Thank
                                                          you.</div>
                                                          </div>
                                                          </div>
                                                          <div
                                                          class="gmail_extra"><span><br
                                                          clear="all">
                                                          <div>
                                                          <div
class="m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114m_-7640979976631609660m_7374879952101670984gmail_signature"
data-smartmail="gmail_signature">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr"><br>
                                                          Sincerely,<br>
                                                          Artem<br>
                                                          <br>
                                                          --<br>
                                                          Founder, <a
                                                          href="http://www.androidpolice.com"
target="_blank" moz-do-not-send="true">Android Police</a>, <a
                                                          href="http://www.apkmirror.com/"
style="font-size:12.8000001907349px" target="_blank"
                                                          moz-do-not-send="true">APK
                                                          Mirror</a><span
style="font-size:12.8000001907349px">, Illogical Robot LLC</span></div>
                                                          <div dir="ltr"><a
href="http://beerpla.net/" target="_blank" moz-do-not-send="true">beerpla.net</a>
                                                          | <a
                                                          href="https://plus.google.com/+ArtemRussakovskii"
target="_blank" moz-do-not-send="true">+ArtemRussakovskii</a> | <a
                                                          href="http://twitter.com/ArtemR"
target="_blank" moz-do-not-send="true">@ArtemR</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          </span>
                                                          <div>
                                                          <div
class="m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114m_-7640979976631609660h5">
                                                          <div
                                                          class="gmail_quote">On
                                                          Thu, Apr 5,
                                                          2018 at 11:13
                                                          PM, Artem
                                                          Russakovskii <span
                                                          dir="ltr">&lt;<a
href="mailto:archon810@gmail.com" target="_blank" moz-do-not-send="true">archon810@gmail.com</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div dir="ltr">Hi,
                                                          <div><br>
                                                          </div>
                                                          <div>I'm
                                                          trying to
                                                          squeeze
                                                          performance
                                                          out of gluster
                                                          on 4 80GB RAM
                                                          20-CPU
                                                          machines where
                                                          Gluster runs
                                                          on attached
                                                          block storage
                                                          (Linode) in (4
                                                          replicate
                                                          bricks), and
                                                          so far
                                                          everything I
                                                          tried results
                                                          in sub-optimal
                                                          performance.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>There are
                                                          many files -
                                                          mostly images,
                                                          several
                                                          million - and
                                                          many
                                                          operations
                                                          take minutes,
                                                          copying
                                                          multiple files
                                                          (even if
                                                          they're small)
                                                          suddenly
                                                          freezes up for
                                                          seconds at a
                                                          time, then
                                                          continues,
                                                          iostat
                                                          frequently
                                                          shows large
                                                          r_await and
                                                          w_awaits with
                                                          100%
                                                          utilization
                                                          for the
                                                          attached block
                                                          device, etc.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>But
                                                          anyway, there
                                                          are many
                                                          guides out
                                                          there for
                                                          small-file
                                                          performance
                                                          improvements,
                                                          but more
                                                          explanation is
                                                          needed, and I
                                                          think more
                                                          tweaks should
                                                          be possible.<br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>My
                                                          question today
                                                          is
                                                          about performance.cache-size.
                                                          Is this a size
                                                          of cache in
                                                          RAM? If so,
                                                          how do I view
                                                          the current
                                                          cache size to
                                                          see if it gets
                                                          full and I
                                                          should
                                                          increase its
                                                          size? Is it
                                                          advisable to
                                                          bump it up if
                                                          I have many
                                                          tens of gigs
                                                          of RAM free? </div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>More
                                                          generally, in
                                                          the last 2
                                                          months since I
                                                          first started
                                                          working with
                                                          gluster and
                                                          set a
                                                          production
                                                          system live,
                                                          I've been
                                                          feeling
                                                          frustrated
                                                          because
                                                          Gluster has a
                                                          lot of
                                                          poorly-documented
                                                          and confusing
                                                          options. I
                                                          really wish
                                                          documentation
                                                          could be
                                                          improved with
                                                          examples and
                                                          better
                                                          explanations.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Specifically,
                                                          it'd be
                                                          absolutely
                                                          amazing if the
                                                          docs offered a
                                                          strategy for
                                                          setting each
                                                          value and ways
                                                          of determining
                                                          more optimal
                                                          values. For
                                                          example,
                                                          for performance.cache-size,
                                                          if it said
                                                          something like
                                                          "run command
                                                          abc to see
                                                          your current
                                                          cache size,
                                                          and if it's
                                                          hurting, up
                                                          it, but be
                                                          aware that
                                                          it's limited
                                                          by RAM," it'd
                                                          be already a
                                                          huge
                                                          improvement to
                                                          the docs. And
                                                          so on with
                                                          other options.</div>
                                                          <div>
                                                          <div
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
                                                          </div>
                                                          <div
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
                                                          </div>
                                                          <div
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
                                                          </div>
                                                          <div
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">The
                                                          gluster team
                                                          is quite
                                                          helpful on
                                                          this mailing
                                                          list, but in a
                                                          reactive
                                                          rather than
                                                          proactive way.
                                                          Perhaps it's
                                                          tunnel vision
                                                          once you've
                                                          worked on a
                                                          project for so
                                                          long where
                                                          less technical
                                                          explanations
                                                          and even
                                                          proper
                                                          documentation
                                                          of options
                                                          takes a back
                                                          seat, but I
                                                          encourage you
                                                          to be more
                                                          proactive
                                                          about helping
                                                          us understand
                                                          and optimize
                                                          Gluster.</div>
                                                          <div
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
                                                          </div>
                                                          <div
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">Thank
                                                          you.</div>
                                                          <div
style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
                                                          </div>
                                                          <div>
                                                          <div
class="m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114m_-7640979976631609660m_7374879952101670984m_1420630595765814996gmail_signature">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">Sincerely,<br>
                                                          Artem<br>
                                                          <br>
                                                          --<br>
                                                          Founder, <a
                                                          href="http://www.androidpolice.com"
target="_blank" moz-do-not-send="true">Android Police</a>, <a
                                                          href="http://www.apkmirror.com/"
style="font-size:12.8px" target="_blank" moz-do-not-send="true">APK
                                                          Mirror</a><span
style="font-size:12.8px">, Illogical Robot LLC</span></div>
                                                          <div dir="ltr"><a
href="http://beerpla.net/" target="_blank" moz-do-not-send="true">beerpla.net</a>
                                                          | <a
                                                          href="https://plus.google.com/+ArtemRussakovskii"
target="_blank" moz-do-not-send="true">+ArtemRussakovskii</a> | <a
                                                          href="http://twitter.com/ArtemR"
target="_blank" moz-do-not-send="true">@ArtemR</a><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          <span>______________________________<wbr>_________________<br>
                                                          Gluster-users
                                                          mailing list<br>
                                                          <a
                                                          href="mailto:Gluster-users@gluster.org"
target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a><br>
                                                          <a
                                                          href="http://lists.gluster.org/mailman/listinfo/gluster-users"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
                                                          </span></blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                        </blockquote>
                                                      </div>
                                                      <br>
                                                    </div>
                                                  </div>
                                                </div>
                                              </div>
                                            </blockquote>
                                          </div>
                                          <br>
                                        </div>
                                      </div>
                                    </div>
                                  </blockquote>
                                </div>
                                <br>
                              </div>
                            </div>
                          </div>
                        </blockquote>
                      </div>
                      <br>
                    </div>
                    <br>
                    <fieldset
                      class="m_979837676701406854mimeAttachmentHeader"></fieldset>
                    <br>
                    <pre>______________________________<wbr>_________________
Gluster-users mailing list
<a class="m_979837676701406854moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a>
<a class="m_979837676701406854moz-txt-link-freetext" href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank" moz-do-not-send="true">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a></pre>
                  </blockquote>
                  <br>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
  </body>
</html>