<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 29, 2018 at 8:24 PM, Pat Haley <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div text="#000000" bgcolor="#FFFFFF">
    <br>
    Hi Raghavendra,<br>
    <br>
    Our technician was able to try the manual setting today.  He found
    that our upper limit for <span>performance.md-cache-<wbr>timeout was 60
      not 600, so he used that value, along with the </span><span>network.inode-lru-limit=50000.<br>
      <br>
      The result was another small (~1%) increase in speed.  Does this
      suggest some addition tests/changes we could try?<br></span></div></blockquote><div><br></div><div>Can you set gluster option diagnostics.client-log-level to TRACE  and run sequential read tests again (with md-cache-timeout value of 60)? <br></div><div><br></div><div>#gluster volume set &lt;volname&gt; diagnostics.client-log-level TRACE</div><div><br></div><div>Also are you sure that open-behind was turned off? Can you give the output of,</div><div><br></div><div># gluster volume info &lt;volname&gt;</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><span>
      <br>
      Thanks<span class="HOEnZb"><font color="#888888"><br>
      <br>
      Pat<br>
      <br>
      <br>
    </font></span></span><div><div class="h5"><br>
    <br>
    <div class="m_-2192927334915852989moz-cite-prefix">On 06/25/2018 09:39 PM, Raghavendra
      Gowdappa wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr"><br>
        <div class="gmail_extra"><br>
          <div class="gmail_quote">On Tue, Jun 26, 2018 at 3:21 AM, Pat
            Haley <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
              <div bgcolor="#FFFFFF"> <br>
                Hi Raghavendra,<br>
                <br>
                Setting the performance.write-behind off had a small
                improvement on the write speed (~3%),<br>
                <br>
                We were unable to turn on &quot;group metadata-cache&quot;.  When
                we try get errors like<br>
                <br>
                # gluster volume set data-volume group metadata-cache<br>
                &#39;/var/lib/glusterd/groups/meta<wbr>data-cache&#39; file
                format not valid.<br>
                <br>
                Was metadata-cache available for gluster 3.7.11? We ask
                because the release notes for 3.11 mentions “Feature for
                metadata-caching/small file performance is production
                ready.” (<a class="m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-freetext" href="https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/" target="_blank">https://gluster.readthedocs.i<wbr>o/en/latest/release-notes/3.11<wbr>.0/</a>).<br>
                <br>
                Do any of these results suggest anything?  If not, what
                further tests would be useful?<br>
              </div>
            </blockquote>
            <div><br>
            </div>
            <div>Group metadata-cache is just a bunch of options one
              sets on a volume. So, You can set them manually using
              gluster cli. Following are the options and their values:</div>
            <div><br>
            </div>
            <div>performance.md-cache-timeout=<wbr>600<br>
              network.inode-lru-limit=50000<br>
              <br>
            </div>
            <div><br>
            </div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
              <div bgcolor="#FFFFFF"> <br>
                Thanks<span class="m_-2192927334915852989gmail-HOEnZb"><font color="#888888"><br>
                    <br>
                    Pat</font></span>
                <div>
                  <div class="m_-2192927334915852989gmail-h5"><br>
                    <br>
                     <br>
                    <br>
                    <div class="m_-2192927334915852989gmail-m_-5807630702261925266moz-cite-prefix">On
                      06/22/2018 07:51 AM, Raghavendra Gowdappa wrote:<br>
                    </div>
                    <blockquote type="cite">
                      <div dir="ltr"><br>
                        <div class="gmail_extra"><br>
                          <div class="gmail_quote">On Thu, Jun 21, 2018
                            at 8:41 PM, Pat Haley <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                            wrote:<br>
                            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                              <div bgcolor="#FFFFFF"> <br>
                                Hi Raghavendra,<br>
                                <br>
                                Thanks for the suggestions.  Our
                                technician will be in on Monday.  We&#39;ll
                                test then and let you know the results.<br>
                                <br>
                                One question I have, is the &quot;group
                                metadata-cache&quot; option supposed to
                                directly impact the performance or is it
                                to help collect data?  If the latter,
                                where will the data be located?<br>
                              </div>
                            </blockquote>
                            <div><br>
                            </div>
                            <div>It impacts performance.</div>
                            <div> <br>
                            </div>
                            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                              <div bgcolor="#FFFFFF"> <br>
                                Thanks again.<span class="m_-2192927334915852989gmail-m_-5807630702261925266HOEnZb"><font color="#888888"><br>
                                    <br>
                                    Pat</font></span>
                                <div>
                                  <div class="m_-2192927334915852989gmail-m_-5807630702261925266h5"><br>
                                    <br>
                                    <br>
                                    <div class="m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-cite-prefix">On
                                      06/21/2018 01:01 AM, Raghavendra
                                      Gowdappa wrote:<br>
                                    </div>
                                    <blockquote type="cite">
                                      <div dir="ltr"><br>
                                        <div class="gmail_extra"><br>
                                          <div class="gmail_quote">On
                                            Thu, Jun 21, 2018 at 10:24
                                            AM, Raghavendra Gowdappa <span dir="ltr">&lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;</span>
                                            wrote:<br>
                                            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                              <div dir="ltr">
                                                <div>
                                                  <div>
                                                    <div>
                                                      <div>
                                                        <div>
                                                          <div>
                                                          <div>For the
                                                          case of writes
                                                          to glusterfs
                                                          mount,<br>
                                                          <br>
                                                          </div>
                                                          I saw in
                                                          earlier
                                                          conversations
                                                          that there are
                                                          too many
                                                          lookups, but
                                                          small number
                                                          of writes.
                                                          Since writes
                                                          cached in
                                                          write-behind
                                                          would
                                                          invalidate
                                                          metadata
                                                          cache, lookups
                                                          won&#39;t be
                                                          absorbed by
                                                          md-cache. I am
                                                          wondering what
                                                          would results
                                                          look like if
                                                          we turn off
                                                          performance.write-behind.<br>
                                                          <br>
                                                          </div>
                                                          @Pat,<br>
                                                          <br>
                                                        </div>
                                                        Can you set,<br>
                                                        <br>
                                                      </div>
                                                      # gluster volume
                                                      set
                                                      &lt;volname&gt;
                                                      performance.write-behind
                                                      off<br>
                                                    </div>
                                                  </div>
                                                </div>
                                              </div>
                                            </blockquote>
                                            <div><br>
                                            </div>
                                            <div>Please turn on &quot;group
                                              metadata-cache&quot; for write
                                              tests too.</div>
                                            <div> <br>
                                            </div>
                                            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                              <div dir="ltr">
                                                <div>
                                                  <div>
                                                    <div><br>
                                                    </div>
                                                    and redo the tests
                                                    writing to glusterfs
                                                    mount? Let us know
                                                    about the results
                                                    you see.<br>
                                                    <br>
                                                  </div>
                                                  regards,<br>
                                                </div>
                                                Raghavendra<br>
                                              </div>
                                              <div class="m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663HOEnZb">
                                                <div class="m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663h5">
                                                  <div class="gmail_extra"><br>
                                                    <div class="gmail_quote">On
                                                      Thu, Jun 21, 2018
                                                      at 8:33 AM,
                                                      Raghavendra
                                                      Gowdappa <span dir="ltr">&lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;</span>
                                                      wrote:<br>
                                                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                        <div dir="ltr"><br>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote"><span>On
                                                          Thu, Jun 21,
                                                          2018 at 8:32
                                                          AM,
                                                          Raghavendra
                                                          Gowdappa <span dir="ltr">&lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>For the
                                                          case of
                                                          reading from
                                                          Glusterfs
                                                          mount,
                                                          read-ahead
                                                          should help.
                                                          However, we&#39;ve
                                                          known issues
                                                          with
                                                          read-ahead[1][2].
                                                          To work around
                                                          these, can you
                                                          try with,<br>
                                                          <br>
                                                          </div>
                                                          1. Turn off
                                                          performance.open-behind<br>
                                                          </div>
                                                          #gluster
                                                          volume set
                                                          &lt;volname&gt;
performance.open-behind off<br>
                                                          <br>
                                                          <div>2. enable
                                                          group meta
                                                          metadata-cache<br>
                                                          # gluster
                                                          volume set
                                                          &lt;volname&gt;
                                                          group
                                                          metadata-cache</div>
                                                          </div>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          </span>
                                                          <div>[1]  <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1084508" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1084508</a></div>
                                                          <div>[2] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1214489" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1214489</a></div>
                                                          <span>
                                                          <div><br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>
                                                          <div class="m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663m_-7085830777103976589m_-1104802400275053683gmail-m_5499637752699686457h5"><br>
                                                          <br>
                                                          <div>
                                                          <div>
                                                          <div>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Thu, Jun 21,
                                                          2018 at 5:00
                                                          AM, Pat Haley
                                                          <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
                                                          Hi,<br>
                                                          <br>
                                                          We were
                                                          recently
                                                          revisiting our
                                                          problems with
                                                          the slowness
                                                          of gluster
                                                          writes (<a href="http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html" rel="noreferrer" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2017-April<wbr>/030529.html</a>).
                                                          Specifically
                                                          we were
                                                          testing the
                                                          suggestions in
                                                          a recent post
                                                          (<a href="http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html" rel="noreferrer" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2018-March<wbr>/033699.html</a>).
                                                          The first two
                                                          suggestions
                                                          (specifying a
negative-timeout in the mount settings or adding
rpc-auth-allow-insecure to glusterd.vol) did not improve our
                                                          performance,
                                                          while setting
&quot;disperse.eager-lock off&quot; provided a tiny (5%) speed-up.<br>
                                                          <br>
                                                          Some of the
                                                          various tests
                                                          we have tried
                                                          earlier can be
                                                          seen in the
                                                          links below. 
                                                          Do any of the
                                                          above
                                                          observations
                                                          suggest what
                                                          we could try
                                                          next to either
                                                          improve the
                                                          speed or debug
                                                          the issue? 
                                                          Thanks<br>
                                                          <br>
                                                          <a href="http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-June/0<wbr>31565.html</a><br>
                                                          <a href="http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-May/03<wbr>0937.html</a><br>
                                                          <br>
                                                          Pat<br>
                                                          <br>
                                                          -- <br>
                                                          <br>
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-<br>
                                                          Pat Haley     
                                                                       
                                                                Email: 
                                                          <a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a><br>
                                                          Center for
                                                          Ocean
                                                          Engineering   
                                                             Phone: 
                                                          (617) 253-6824<br>
                                                          Dept. of
                                                          Mechanical
                                                          Engineering   
                                                          Fax:    (617)
                                                          253-8125<br>
                                                          MIT, Room
                                                          5-213         
                                                                    <a href="http://web.mit.edu/phaley/www/" rel="noreferrer" target="_blank">http://web.mit.edu/phaley/www/</a><br>
                                                          77
                                                          Massachusetts
                                                          Avenue<br>
                                                          Cambridge, MA 
                                                          02139-4301<br>
                                                          <br>
______________________________<wbr>_________________<br>
                                                          Gluster-users
                                                          mailing list<br>
                                                          <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
                                                          <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a></blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </span></div>
                                                          <br>
                                                          </div>
                                                        </div>
                                                      </blockquote>
                                                    </div>
                                                    <br>
                                                  </div>
                                                </div>
                                              </div>
                                            </blockquote>
                                          </div>
                                          <br>
                                        </div>
                                      </div>
                                    </blockquote>
                                    <br>
                                    <pre class="m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                                  </div>
                                </div>
                              </div>
                              <br>
                              ______________________________<wbr>_________________<br>
                              Gluster-users mailing list<br>
                              <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
                              <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
                            </blockquote>
                          </div>
                          <br>
                        </div>
                      </div>
                    </blockquote>
                    <br>
                    <pre class="m_-2192927334915852989gmail-m_-5807630702261925266moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                  </div>
                </div>
              </div>
            </blockquote>
          </div>
          <br>
        </div>
      </div>
    </blockquote>
    <br>
    <pre class="m_-2192927334915852989moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_-2192927334915852989moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_-2192927334915852989moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
  </div></div></div>

</blockquote></div><br></div></div>