<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
Hi Raghavendra,<br>
<br>
Our technician was able to try the manual setting today. He found
that our upper limit for <span>performance.md-cache-timeout was 60
not 600, so he used that value, along with the </span><span>network.inode-lru-limit=50000.<br>
<br>
The result was another small (~1%) increase in speed. Does this
suggest some addition tests/changes we could try?<br>
<br>
Thanks<br>
<br>
Pat<br>
<br>
<br>
</span><br>
<br>
<div class="moz-cite-prefix">On 06/25/2018 09:39 PM, Raghavendra
Gowdappa wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAFkORY9w=OXas==AtvE=muaq2Dn+GAc8KkfxPyacF25zRPc_hQ@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Jun 26, 2018 at 3:21 AM, Pat
Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu"
target="_blank" moz-do-not-send="true">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> <br>
Hi Raghavendra,<br>
<br>
Setting the performance.write-behind off had a small
improvement on the write speed (~3%),<br>
<br>
We were unable to turn on "group metadata-cache". When
we try get errors like<br>
<br>
# gluster volume set data-volume group metadata-cache<br>
'/var/lib/glusterd/groups/<wbr>metadata-cache' file
format not valid.<br>
<br>
Was metadata-cache available for gluster 3.7.11? We ask
because the release notes for 3.11 mentions “Feature for
metadata-caching/small file performance is production
ready.” (<a
class="gmail-m_-5807630702261925266moz-txt-link-freetext"
href="https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/"
target="_blank" moz-do-not-send="true">https://gluster.readthedocs.<wbr>io/en/latest/release-notes/3.<wbr>11.0/</a>).<br>
<br>
Do any of these results suggest anything? If not, what
further tests would be useful?<br>
</div>
</blockquote>
<div><br>
</div>
<div>Group metadata-cache is just a bunch of options one
sets on a volume. So, You can set them manually using
gluster cli. Following are the options and their values:</div>
<div><br>
</div>
<div>performance.md-cache-timeout=600<br>
network.inode-lru-limit=50000<br>
<br>
</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> <br>
Thanks<span class="gmail-HOEnZb"><font color="#888888"><br>
<br>
Pat</font></span>
<div>
<div class="gmail-h5"><br>
<br>
<br>
<br>
<div
class="gmail-m_-5807630702261925266moz-cite-prefix">On
06/22/2018 07:51 AM, Raghavendra Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Jun 21, 2018
at 8:41 PM, Pat Haley <span dir="ltr"><<a
href="mailto:phaley@mit.edu"
target="_blank" moz-do-not-send="true">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> <br>
Hi Raghavendra,<br>
<br>
Thanks for the suggestions. Our
technician will be in on Monday. We'll
test then and let you know the results.<br>
<br>
One question I have, is the "group
metadata-cache" option supposed to
directly impact the performance or is it
to help collect data? If the latter,
where will the data be located?<br>
</div>
</blockquote>
<div><br>
</div>
<div>It impacts performance.</div>
<div> <br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> <br>
Thanks again.<span
class="gmail-m_-5807630702261925266HOEnZb"><font
color="#888888"><br>
<br>
Pat</font></span>
<div>
<div
class="gmail-m_-5807630702261925266h5"><br>
<br>
<br>
<div
class="gmail-m_-5807630702261925266m_-4085817742671750663moz-cite-prefix">On
06/21/2018 01:01 AM, Raghavendra
Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Jun 21, 2018 at 10:24
AM, Raghavendra Gowdappa <span
dir="ltr"><<a
href="mailto:rgowdapp@redhat.com"
target="_blank"
moz-do-not-send="true">rgowdapp@redhat.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>For the
case of writes
to glusterfs
mount,<br>
<br>
</div>
I saw in
earlier
conversations
that there are
too many
lookups, but
small number
of writes.
Since writes
cached in
write-behind
would
invalidate
metadata
cache, lookups
won't be
absorbed by
md-cache. I am
wondering what
would results
look like if
we turn off
performance.write-behind.<br>
<br>
</div>
@Pat,<br>
<br>
</div>
Can you set,<br>
<br>
</div>
# gluster volume
set
<volname>
performance.write-behind
off<br>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Please turn on "group
metadata-cache" for write
tests too.</div>
<div> <br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div><br>
</div>
and redo the tests
writing to glusterfs
mount? Let us know
about the results
you see.<br>
<br>
</div>
regards,<br>
</div>
Raghavendra<br>
</div>
<div
class="gmail-m_-5807630702261925266m_-4085817742671750663HOEnZb">
<div
class="gmail-m_-5807630702261925266m_-4085817742671750663h5">
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Thu, Jun 21, 2018
at 8:33 AM,
Raghavendra
Gowdappa <span
dir="ltr"><<a
href="mailto:rgowdapp@redhat.com" target="_blank" moz-do-not-send="true">rgowdapp@redhat.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><br>
<div
class="gmail_extra"><br>
<div
class="gmail_quote"><span>On
Thu, Jun 21,
2018 at 8:32
AM,
Raghavendra
Gowdappa <span
dir="ltr"><<a
href="mailto:rgowdapp@redhat.com" target="_blank" moz-do-not-send="true">rgowdapp@redhat.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>For the
case of
reading from
Glusterfs
mount,
read-ahead
should help.
However, we've
known issues
with
read-ahead[1][2].
To work around
these, can you
try with,<br>
<br>
</div>
1. Turn off
performance.open-behind<br>
</div>
#gluster
volume set
<volname>
performance.open-behind off<br>
<br>
<div>2. enable
group meta
metadata-cache<br>
# gluster
volume set
<volname>
group
metadata-cache</div>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>[1] <a
href="https://bugzilla.redhat.com/show_bug.cgi?id=1084508"
target="_blank"
moz-do-not-send="true">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1084508</a></div>
<div>[2] <a
href="https://bugzilla.redhat.com/show_bug.cgi?id=1214489"
target="_blank" moz-do-not-send="true">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1214489</a></div>
<span>
<div><br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div
class="gmail-m_-5807630702261925266m_-4085817742671750663m_-7085830777103976589m_-1104802400275053683gmail-m_5499637752699686457h5"><br>
<br>
<div>
<div>
<div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Thu, Jun 21,
2018 at 5:00
AM, Pat Haley
<span
dir="ltr"><<a
href="mailto:phaley@mit.edu" target="_blank" moz-do-not-send="true">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex"><br>
Hi,<br>
<br>
We were
recently
revisiting our
problems with
the slowness
of gluster
writes (<a
href="http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2017-April<wbr>/030529.html</a>).
Specifically
we were
testing the
suggestions in
a recent post
(<a
href="http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2018-March<wbr>/033699.html</a>).
The first two
suggestions
(specifying a
negative-timeout in the mount settings or adding
rpc-auth-allow-insecure to glusterd.vol) did not improve our
performance,
while setting
"disperse.eager-lock off" provided a tiny (5%) speed-up.<br>
<br>
Some of the
various tests
we have tried
earlier can be
seen in the
links below.
Do any of the
above
observations
suggest what
we could try
next to either
improve the
speed or debug
the issue?
Thanks<br>
<br>
<a
href="http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-June/0<wbr>31565.html</a><br>
<a
href="http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-May/03<wbr>0937.html</a><br>
<br>
Pat<br>
<br>
-- <br>
<br>
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-<br>
Pat Haley
Email:
<a
href="mailto:phaley@mit.edu"
target="_blank" moz-do-not-send="true">phaley@mit.edu</a><br>
Center for
Ocean
Engineering
Phone:
(617) 253-6824<br>
Dept. of
Mechanical
Engineering
Fax: (617)
253-8125<br>
MIT, Room
5-213
<a
href="http://web.mit.edu/phaley/www/" rel="noreferrer" target="_blank"
moz-do-not-send="true">http://web.mit.edu/phaley/www/</a><br>
77
Massachusetts
Avenue<br>
Cambridge, MA
02139-4301<br>
<br>
______________________________<wbr>_________________<br>
Gluster-users
mailing list<br>
<a
href="mailto:Gluster-users@gluster.org"
target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a><br>
<a
href="http://lists.gluster.org/mailman/listinfo/gluster-users"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a></blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</span></div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_-5807630702261925266m_-4085817742671750663moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="gmail-m_-5807630702261925266m_-4085817742671750663moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank" moz-do-not-send="true">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="gmail-m_-5807630702261925266m_-4085817742671750663moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank" moz-do-not-send="true">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org"
target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a><br>
<a
href="http://lists.gluster.org/mailman/listinfo/gluster-users"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="gmail-m_-5807630702261925266moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="gmail-m_-5807630702261925266moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank" moz-do-not-send="true">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="gmail-m_-5807630702261925266moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank" moz-do-not-send="true">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: <a class="moz-txt-link-abbreviated" href="mailto:phaley@mit.edu">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</body>
</html>