<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 26, 2018 at 3:21 AM, Pat Haley <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF">
    <br>
    Hi Raghavendra,<br>
    <br>
    Setting the performance.write-behind off had a small improvement on
    the write speed (~3%),<br>
    <br>
    We were unable to turn on &quot;group metadata-cache&quot;.  When we try get
    errors like<br>
    <br>
    # gluster volume set data-volume group metadata-cache<br>
    &#39;/var/lib/glusterd/groups/<wbr>metadata-cache&#39; file format not valid.<br>
    <br>
    Was metadata-cache available for gluster 3.7.11? We ask because the
    release notes for 3.11 mentions “Feature for metadata-caching/small
    file performance is production ready.”
    (<a class="gmail-m_-5807630702261925266moz-txt-link-freetext" href="https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/" target="_blank">https://gluster.readthedocs.<wbr>io/en/latest/release-notes/3.<wbr>11.0/</a>).<br>
    <br>
    Do any of these results suggest anything?  If not, what further
    tests would be useful?<br></div></blockquote><div><br></div><div>Group metadata-cache is just a bunch of options one sets on a volume. So, You can set them manually using gluster cli. Following are the options and their values:</div><div><br></div><div>performance.md-cache-timeout=600<br>network.inode-lru-limit=50000<br><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
    <br>
    Thanks<span class="gmail-HOEnZb"><font color="#888888"><br>
    <br>
    Pat</font></span><div><div class="gmail-h5"><br>
    <br>
     <br>
    <br>
    <div class="gmail-m_-5807630702261925266moz-cite-prefix">On 06/22/2018 07:51 AM, Raghavendra
      Gowdappa wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr"><br>
        <div class="gmail_extra"><br>
          <div class="gmail_quote">On Thu, Jun 21, 2018 at 8:41 PM, Pat
            Haley <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
              <div bgcolor="#FFFFFF"> <br>
                Hi Raghavendra,<br>
                <br>
                Thanks for the suggestions.  Our technician will be in
                on Monday.  We&#39;ll test then and let you know the
                results.<br>
                <br>
                One question I have, is the &quot;group metadata-cache&quot;
                option supposed to directly impact the performance or is
                it to help collect data?  If the latter, where will the
                data be located?<br>
              </div>
            </blockquote>
            <div><br>
            </div>
            <div>It impacts performance.</div>
            <div> <br>
            </div>
            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
              <div bgcolor="#FFFFFF"> <br>
                Thanks again.<span class="gmail-m_-5807630702261925266HOEnZb"><font color="#888888"><br>
                    <br>
                    Pat</font></span>
                <div>
                  <div class="gmail-m_-5807630702261925266h5"><br>
                    <br>
                    <br>
                    <div class="gmail-m_-5807630702261925266m_-4085817742671750663moz-cite-prefix">On
                      06/21/2018 01:01 AM, Raghavendra Gowdappa wrote:<br>
                    </div>
                    <blockquote type="cite">
                      <div dir="ltr"><br>
                        <div class="gmail_extra"><br>
                          <div class="gmail_quote">On Thu, Jun 21, 2018
                            at 10:24 AM, Raghavendra Gowdappa <span dir="ltr">&lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;</span>
                            wrote:<br>
                            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                              <div dir="ltr">
                                <div>
                                  <div>
                                    <div>
                                      <div>
                                        <div>
                                          <div>
                                            <div>For the case of writes
                                              to glusterfs mount,<br>
                                              <br>
                                            </div>
                                            I saw in earlier
                                            conversations that there are
                                            too many lookups, but small
                                            number of writes. Since
                                            writes cached in
                                            write-behind would
                                            invalidate metadata cache,
                                            lookups won&#39;t be absorbed by
                                            md-cache. I am wondering
                                            what would results look like
                                            if we turn off
                                            performance.write-behind.<br>
                                            <br>
                                          </div>
                                          @Pat,<br>
                                          <br>
                                        </div>
                                        Can you set,<br>
                                        <br>
                                      </div>
                                      # gluster volume set
                                      &lt;volname&gt;
                                      performance.write-behind off<br>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </blockquote>
                            <div><br>
                            </div>
                            <div>Please turn on &quot;group metadata-cache&quot;
                              for write tests too.</div>
                            <div> <br>
                            </div>
                            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                              <div dir="ltr">
                                <div>
                                  <div>
                                    <div><br>
                                    </div>
                                    and redo the tests writing to
                                    glusterfs mount? Let us know about
                                    the results you see.<br>
                                    <br>
                                  </div>
                                  regards,<br>
                                </div>
                                Raghavendra<br>
                              </div>
                              <div class="gmail-m_-5807630702261925266m_-4085817742671750663HOEnZb">
                                <div class="gmail-m_-5807630702261925266m_-4085817742671750663h5">
                                  <div class="gmail_extra"><br>
                                    <div class="gmail_quote">On Thu, Jun
                                      21, 2018 at 8:33 AM, Raghavendra
                                      Gowdappa <span dir="ltr">&lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;</span>
                                      wrote:<br>
                                      <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                        <div dir="ltr"><br>
                                          <div class="gmail_extra"><br>
                                            <div class="gmail_quote"><span>On
                                                Thu, Jun 21, 2018 at
                                                8:32 AM, Raghavendra
                                                Gowdappa <span dir="ltr">&lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;</span>
                                                wrote:<br>
                                                <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                  <div dir="ltr">
                                                    <div>
                                                      <div>For the case
                                                        of reading from
                                                        Glusterfs mount,
                                                        read-ahead
                                                        should help.
                                                        However, we&#39;ve
                                                        known issues
                                                        with
                                                        read-ahead[1][2].
                                                        To work around
                                                        these, can you
                                                        try with,<br>
                                                        <br>
                                                      </div>
                                                      1. Turn off
                                                      performance.open-behind<br>
                                                    </div>
                                                    #gluster volume set
                                                    &lt;volname&gt;
                                                    performance.open-behind
                                                    off<br>
                                                    <br>
                                                    <div>2. enable group
                                                      meta
                                                      metadata-cache<br>
                                                      # gluster volume
                                                      set
                                                      &lt;volname&gt;
                                                      group
                                                      metadata-cache</div>
                                                  </div>
                                                </blockquote>
                                                <div><br>
                                                </div>
                                              </span>
                                              <div>[1]  <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1084508" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1084508</a></div>
                                              <div>[2] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1214489" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1214489</a></div>
                                              <span>
                                                <div><br>
                                                </div>
                                                <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                  <div dir="ltr">
                                                    <div>
                                                      <div>
                                                        <div class="gmail-m_-5807630702261925266m_-4085817742671750663m_-7085830777103976589m_-1104802400275053683gmail-m_5499637752699686457h5"><br>
                                                          <br>
                                                          <div>
                                                          <div>
                                                          <div>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Thu, Jun 21,
                                                          2018 at 5:00
                                                          AM, Pat Haley
                                                          <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
                                                          Hi,<br>
                                                          <br>
                                                          We were
                                                          recently
                                                          revisiting our
                                                          problems with
                                                          the slowness
                                                          of gluster
                                                          writes (<a href="http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html" rel="noreferrer" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2017-April<wbr>/030529.html</a>).
                                                          Specifically
                                                          we were
                                                          testing the
                                                          suggestions in
                                                          a recent post
                                                          (<a href="http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html" rel="noreferrer" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2018-March<wbr>/033699.html</a>).
                                                          The first two
                                                          suggestions
                                                          (specifying a
negative-timeout in the mount settings or adding
rpc-auth-allow-insecure to glusterd.vol) did not improve our
                                                          performance,
                                                          while setting
&quot;disperse.eager-lock off&quot; provided a tiny (5%) speed-up.<br>
                                                          <br>
                                                          Some of the
                                                          various tests
                                                          we have tried
                                                          earlier can be
                                                          seen in the
                                                          links below. 
                                                          Do any of the
                                                          above
                                                          observations
                                                          suggest what
                                                          we could try
                                                          next to either
                                                          improve the
                                                          speed or debug
                                                          the issue? 
                                                          Thanks<br>
                                                          <br>
                                                          <a href="http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-June/0<wbr>31565.html</a><br>
                                                          <a href="http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-May/03<wbr>0937.html</a><br>
                                                          <br>
                                                          Pat<br>
                                                          <br>
                                                          -- <br>
                                                          <br>
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-<br>
                                                          Pat Haley     
                                                                       
                                                                Email: 
                                                          <a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a><br>
                                                          Center for
                                                          Ocean
                                                          Engineering   
                                                             Phone: 
                                                          (617) 253-6824<br>
                                                          Dept. of
                                                          Mechanical
                                                          Engineering   
                                                          Fax:    (617)
                                                          253-8125<br>
                                                          MIT, Room
                                                          5-213         
                                                                    <a href="http://web.mit.edu/phaley/www/" rel="noreferrer" target="_blank">http://web.mit.edu/phaley/www/</a><br>
                                                          77
                                                          Massachusetts
                                                          Avenue<br>
                                                          Cambridge, MA 
                                                          02139-4301<br>
                                                          <br>
______________________________<wbr>_________________<br>
                                                          Gluster-users
                                                          mailing list<br>
                                                          <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
                                                          <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a></blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                        </div>
                                                      </div>
                                                    </div>
                                                  </div>
                                                </blockquote>
                                              </span></div>
                                            <br>
                                          </div>
                                        </div>
                                      </blockquote>
                                    </div>
                                    <br>
                                  </div>
                                </div>
                              </div>
                            </blockquote>
                          </div>
                          <br>
                        </div>
                      </div>
                    </blockquote>
                    <br>
                    <pre class="gmail-m_-5807630702261925266m_-4085817742671750663moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="gmail-m_-5807630702261925266m_-4085817742671750663moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="gmail-m_-5807630702261925266m_-4085817742671750663moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                  </div>
                </div>
              </div>
              <br>
              ______________________________<wbr>_________________<br>
              Gluster-users mailing list<br>
              <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
              <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
            </blockquote>
          </div>
          <br>
        </div>
      </div>
    </blockquote>
    <br>
    <pre class="gmail-m_-5807630702261925266moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="gmail-m_-5807630702261925266moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="gmail-m_-5807630702261925266moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
  </div></div></div>

</blockquote></div><br></div></div>