<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
Hi Raghavendra,<br>
<br>
Setting the performance.write-behind off had a small improvement on
the write speed (~3%),<br>
<br>
We were unable to turn on "group metadata-cache". When we try get
errors like<br>
<br>
# gluster volume set data-volume group metadata-cache<br>
'/var/lib/glusterd/groups/metadata-cache' file format not valid.<br>
<br>
Was metadata-cache available for gluster 3.7.11? We ask because the
release notes for 3.11 mentions “Feature for metadata-caching/small
file performance is production ready.”
(<a class="moz-txt-link-freetext" href="https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/">https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/</a>).<br>
<br>
Do any of these results suggest anything? If not, what further
tests would be useful?<br>
<br>
Thanks<br>
<br>
Pat<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 06/22/2018 07:51 AM, Raghavendra
Gowdappa wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAFkORY97RQ3pniBp38bzDp_+-6xw4ncpbnzRTz-45vimhEv=mA@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Jun 21, 2018 at 8:41 PM, Pat
Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu"
target="_blank" moz-do-not-send="true">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
Hi Raghavendra,<br>
<br>
Thanks for the suggestions. Our technician will be in
on Monday. We'll test then and let you know the
results.<br>
<br>
One question I have, is the "group metadata-cache"
option supposed to directly impact the performance or is
it to help collect data? If the latter, where will the
data be located?<br>
</div>
</blockquote>
<div><br>
</div>
<div>It impacts performance.</div>
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
Thanks again.<span class="HOEnZb"><font color="#888888"><br>
<br>
Pat</font></span>
<div>
<div class="h5"><br>
<br>
<br>
<div class="m_-4085817742671750663moz-cite-prefix">On
06/21/2018 01:01 AM, Raghavendra Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Jun 21, 2018
at 10:24 AM, Raghavendra Gowdappa <span
dir="ltr"><<a
href="mailto:rgowdapp@redhat.com"
target="_blank" moz-do-not-send="true">rgowdapp@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>For the case of writes
to glusterfs mount,<br>
<br>
</div>
I saw in earlier
conversations that there are
too many lookups, but small
number of writes. Since
writes cached in
write-behind would
invalidate metadata cache,
lookups won't be absorbed by
md-cache. I am wondering
what would results look like
if we turn off
performance.write-behind.<br>
<br>
</div>
@Pat,<br>
<br>
</div>
Can you set,<br>
<br>
</div>
# gluster volume set
<volname>
performance.write-behind off<br>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Please turn on "group metadata-cache"
for write tests too.</div>
<div> <br>
</div>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div><br>
</div>
and redo the tests writing to
glusterfs mount? Let us know about
the results you see.<br>
<br>
</div>
regards,<br>
</div>
Raghavendra<br>
</div>
<div class="m_-4085817742671750663HOEnZb">
<div class="m_-4085817742671750663h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Jun
21, 2018 at 8:33 AM, Raghavendra
Gowdappa <span dir="ltr"><<a
href="mailto:rgowdapp@redhat.com"
target="_blank"
moz-do-not-send="true">rgowdapp@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote"><span>On
Thu, Jun 21, 2018 at
8:32 AM, Raghavendra
Gowdappa <span
dir="ltr"><<a
href="mailto:rgowdapp@redhat.com"
target="_blank"
moz-do-not-send="true">rgowdapp@redhat.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px
0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>For the case
of reading from
Glusterfs mount,
read-ahead
should help.
However, we've
known issues
with
read-ahead[1][2].
To work around
these, can you
try with,<br>
<br>
</div>
1. Turn off
performance.open-behind<br>
</div>
#gluster volume set
<volname>
performance.open-behind
off<br>
<br>
<div>2. enable group
meta
metadata-cache<br>
# gluster volume
set
<volname>
group
metadata-cache</div>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>[1] <a
href="https://bugzilla.redhat.com/show_bug.cgi?id=1084508"
target="_blank"
moz-do-not-send="true">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1084508</a></div>
<div>[2] <a
href="https://bugzilla.redhat.com/show_bug.cgi?id=1214489"
target="_blank"
moz-do-not-send="true">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1214489</a></div>
<span>
<div><br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px
0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div
class="m_-4085817742671750663m_-7085830777103976589m_-1104802400275053683gmail-m_5499637752699686457h5"><br>
<br>
<div>
<div>
<div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Thu, Jun 21,
2018 at 5:00
AM, Pat Haley
<span
dir="ltr"><<a
href="mailto:phaley@mit.edu" target="_blank" moz-do-not-send="true">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex"><br>
Hi,<br>
<br>
We were
recently
revisiting our
problems with
the slowness
of gluster
writes (<a
href="http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2017-April<wbr>/030529.html</a>).
Specifically
we were
testing the
suggestions in
a recent post
(<a
href="http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2018-March<wbr>/033699.html</a>).
The first two
suggestions
(specifying a
negative-timeout in the mount settings or adding
rpc-auth-allow-insecure to glusterd.vol) did not improve our
performance,
while setting
"disperse.eager-lock off" provided a tiny (5%) speed-up.<br>
<br>
Some of the
various tests
we have tried
earlier can be
seen in the
links below.
Do any of the
above
observations
suggest what
we could try
next to either
improve the
speed or debug
the issue?
Thanks<br>
<br>
<a
href="http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-June/0<wbr>31565.html</a><br>
<a
href="http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-May/03<wbr>0937.html</a><br>
<br>
Pat<br>
<br>
-- <br>
<br>
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-<br>
Pat Haley
Email:
<a
href="mailto:phaley@mit.edu"
target="_blank" moz-do-not-send="true">phaley@mit.edu</a><br>
Center for
Ocean
Engineering
Phone:
(617) 253-6824<br>
Dept. of
Mechanical
Engineering
Fax: (617)
253-8125<br>
MIT, Room
5-213
<a
href="http://web.mit.edu/phaley/www/" rel="noreferrer" target="_blank"
moz-do-not-send="true">http://web.mit.edu/phaley/www/</a><br>
77
Massachusetts
Avenue<br>
Cambridge, MA
02139-4301<br>
<br>
______________________________<wbr>_________________<br>
Gluster-users
mailing list<br>
<a
href="mailto:Gluster-users@gluster.org"
target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a><br>
<a
href="http://lists.gluster.org/mailman/listinfo/gluster-users"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a></blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</span></div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_-4085817742671750663moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-4085817742671750663moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank" moz-do-not-send="true">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-4085817742671750663moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank" moz-do-not-send="true">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org"
moz-do-not-send="true">Gluster-users@gluster.org</a><br>
<a
href="http://lists.gluster.org/mailman/listinfo/gluster-users"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: <a class="moz-txt-link-abbreviated" href="mailto:phaley@mit.edu">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</body>
</html>