<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 29, 2018 at 8:24 PM, Pat Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<br>
Hi Raghavendra,<br>
<br>
Our technician was able to try the manual setting today. He found
that our upper limit for <span>performance.md-cache-<wbr>timeout was 60
not 600, so he used that value, along with the </span><span>network.inode-lru-limit=50000.<br>
<br>
The result was another small (~1%) increase in speed. Does this
suggest some addition tests/changes we could try?<br></span></div></blockquote><div><br></div><div>Can you set gluster option diagnostics.client-log-level to TRACE and run sequential read tests again (with md-cache-timeout value of 60)? <br></div><div><br></div><div>#gluster volume set <volname> diagnostics.client-log-level TRACE</div><div><br></div><div>Also are you sure that open-behind was turned off? Can you give the output of,</div><div><br></div><div># gluster volume info <volname></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><span>
<br>
Thanks<span class="HOEnZb"><font color="#888888"><br>
<br>
Pat<br>
<br>
<br>
</font></span></span><div><div class="h5"><br>
<br>
<div class="m_-2192927334915852989moz-cite-prefix">On 06/25/2018 09:39 PM, Raghavendra
Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Jun 26, 2018 at 3:21 AM, Pat
Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> <br>
Hi Raghavendra,<br>
<br>
Setting the performance.write-behind off had a small
improvement on the write speed (~3%),<br>
<br>
We were unable to turn on "group metadata-cache". When
we try get errors like<br>
<br>
# gluster volume set data-volume group metadata-cache<br>
'/var/lib/glusterd/groups/meta<wbr>data-cache' file
format not valid.<br>
<br>
Was metadata-cache available for gluster 3.7.11? We ask
because the release notes for 3.11 mentions “Feature for
metadata-caching/small file performance is production
ready.” (<a class="m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-freetext" href="https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/" target="_blank">https://gluster.readthedocs.i<wbr>o/en/latest/release-notes/3.11<wbr>.0/</a>).<br>
<br>
Do any of these results suggest anything? If not, what
further tests would be useful?<br>
</div>
</blockquote>
<div><br>
</div>
<div>Group metadata-cache is just a bunch of options one
sets on a volume. So, You can set them manually using
gluster cli. Following are the options and their values:</div>
<div><br>
</div>
<div>performance.md-cache-timeout=<wbr>600<br>
network.inode-lru-limit=50000<br>
<br>
</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> <br>
Thanks<span class="m_-2192927334915852989gmail-HOEnZb"><font color="#888888"><br>
<br>
Pat</font></span>
<div>
<div class="m_-2192927334915852989gmail-h5"><br>
<br>
<br>
<br>
<div class="m_-2192927334915852989gmail-m_-5807630702261925266moz-cite-prefix">On
06/22/2018 07:51 AM, Raghavendra Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Jun 21, 2018
at 8:41 PM, Pat Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> <br>
Hi Raghavendra,<br>
<br>
Thanks for the suggestions. Our
technician will be in on Monday. We'll
test then and let you know the results.<br>
<br>
One question I have, is the "group
metadata-cache" option supposed to
directly impact the performance or is it
to help collect data? If the latter,
where will the data be located?<br>
</div>
</blockquote>
<div><br>
</div>
<div>It impacts performance.</div>
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> <br>
Thanks again.<span class="m_-2192927334915852989gmail-m_-5807630702261925266HOEnZb"><font color="#888888"><br>
<br>
Pat</font></span>
<div>
<div class="m_-2192927334915852989gmail-m_-5807630702261925266h5"><br>
<br>
<br>
<div class="m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-cite-prefix">On
06/21/2018 01:01 AM, Raghavendra
Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Jun 21, 2018 at 10:24
AM, Raghavendra Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>For the
case of writes
to glusterfs
mount,<br>
<br>
</div>
I saw in
earlier
conversations
that there are
too many
lookups, but
small number
of writes.
Since writes
cached in
write-behind
would
invalidate
metadata
cache, lookups
won't be
absorbed by
md-cache. I am
wondering what
would results
look like if
we turn off
performance.write-behind.<br>
<br>
</div>
@Pat,<br>
<br>
</div>
Can you set,<br>
<br>
</div>
# gluster volume
set
<volname>
performance.write-behind
off<br>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Please turn on "group
metadata-cache" for write
tests too.</div>
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div><br>
</div>
and redo the tests
writing to glusterfs
mount? Let us know
about the results
you see.<br>
<br>
</div>
regards,<br>
</div>
Raghavendra<br>
</div>
<div class="m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663HOEnZb">
<div class="m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Jun 21, 2018
at 8:33 AM,
Raghavendra
Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote"><span>On
Thu, Jun 21,
2018 at 8:32
AM,
Raghavendra
Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>For the
case of
reading from
Glusterfs
mount,
read-ahead
should help.
However, we've
known issues
with
read-ahead[1][2].
To work around
these, can you
try with,<br>
<br>
</div>
1. Turn off
performance.open-behind<br>
</div>
#gluster
volume set
<volname>
performance.open-behind off<br>
<br>
<div>2. enable
group meta
metadata-cache<br>
# gluster
volume set
<volname>
group
metadata-cache</div>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>[1] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1084508" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1084508</a></div>
<div>[2] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1214489" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1214489</a></div>
<span>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div class="m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663m_-7085830777103976589m_-1104802400275053683gmail-m_5499637752699686457h5"><br>
<br>
<div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Jun 21,
2018 at 5:00
AM, Pat Haley
<span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Hi,<br>
<br>
We were
recently
revisiting our
problems with
the slowness
of gluster
writes (<a href="http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html" rel="noreferrer" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2017-April<wbr>/030529.html</a>).
Specifically
we were
testing the
suggestions in
a recent post
(<a href="http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html" rel="noreferrer" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2018-March<wbr>/033699.html</a>).
The first two
suggestions
(specifying a
negative-timeout in the mount settings or adding
rpc-auth-allow-insecure to glusterd.vol) did not improve our
performance,
while setting
"disperse.eager-lock off" provided a tiny (5%) speed-up.<br>
<br>
Some of the
various tests
we have tried
earlier can be
seen in the
links below.
Do any of the
above
observations
suggest what
we could try
next to either
improve the
speed or debug
the issue?
Thanks<br>
<br>
<a href="http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-June/0<wbr>31565.html</a><br>
<a href="http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-May/03<wbr>0937.html</a><br>
<br>
Pat<br>
<br>
-- <br>
<br>
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-<br>
Pat Haley
Email:
<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a><br>
Center for
Ocean
Engineering
Phone:
(617) 253-6824<br>
Dept. of
Mechanical
Engineering
Fax: (617)
253-8125<br>
MIT, Room
5-213
<a href="http://web.mit.edu/phaley/www/" rel="noreferrer" target="_blank">http://web.mit.edu/phaley/www/</a><br>
77
Massachusetts
Avenue<br>
Cambridge, MA
02139-4301<br>
<br>
______________________________<wbr>_________________<br>
Gluster-users
mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a></blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</span></div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_-2192927334915852989gmail-m_-5807630702261925266moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_-2192927334915852989moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-2192927334915852989moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-2192927334915852989moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div></div></div>
</blockquote></div><br></div></div>