<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 29, 2018 at 10:38 PM, Pat Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<br>
Hi Raghavendra,<br>
<br>
We ran the tests (write tests) and I copied the log files for both
the server and the client to
<a class="m_9072513899500000032moz-txt-link-freetext" href="http://mseas.mit.edu/download/phaley/GlusterUsers/2018/Jun29/" target="_blank">http://mseas.mit.edu/download/<wbr>phaley/GlusterUsers/2018/<wbr>Jun29/</a> . Is
there any additional trace information you need? (If so, where
should I look for it?)<br></div></blockquote><div><br></div><div>Nothing for now. I can see from logs that workaround is not helping. fstat requests are not absorbed by md-cache and read-ahead is witnessing them and flushing its read-ahead cache. I am investigating more on md-cache (It also seems to be invalidating inodes quite frequently which actually might be the root cause of seeing so many fstat requests from kernel). Will post when I find anything relevant.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
<br>
Also the volume information you requested<br>
<br>
[root@mseas-data2 ~]# gluster volume info data-volume<br>
<br>
Volume Name: data-volume<br>
Type: Distribute<br>
Volume ID: c162161e-2a2d-4dac-b015-<wbr>f31fd89ceb18<br>
Status: Started<br>
Number of Bricks: 2<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: mseas-data2:/mnt/brick1<br>
Brick2: mseas-data2:/mnt/brick2<br>
Options Reconfigured:<br>
diagnostics.client-log-level: TRACE<br>
network.inode-lru-limit: 50000<br>
performance.md-cache-timeout: 60<br>
performance.open-behind: off<br>
disperse.eager-lock: off<br>
auth.allow: *<br>
server.allow-insecure: on<br>
nfs.exports-auth-enable: on<br>
diagnostics.brick-sys-log-<wbr>level: WARNING<br>
performance.readdir-ahead: on<br>
nfs.disable: on<br>
nfs.export-volumes: off<br>
[root@mseas-data2 ~]# <br><div><div class="h5">
<br>
<br>
<div class="m_9072513899500000032moz-cite-prefix">On 06/29/2018 12:28 PM, Raghavendra
Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Jun 29, 2018 at 8:24 PM, Pat
Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
Hi Raghavendra,<br>
<br>
Our technician was able to try the manual setting
today. He found that our upper limit for <span>performance.md-cache-timeo<wbr>ut
was 60 not 600, so he used that value, along with the
</span><span>network.inode-lru-limit=50000.<br>
<br>
The result was another small (~1%) increase in speed.
Does this suggest some addition tests/changes we could
try?<br>
</span></div>
</blockquote>
<div><br>
</div>
<div>Can you set gluster option diagnostics.client-log-level
to TRACE and run sequential read tests again (with
md-cache-timeout value of 60)? <br>
</div>
<div><br>
</div>
<div>#gluster volume set <volname>
diagnostics.client-log-level TRACE</div>
<div><br>
</div>
<div>Also are you sure that open-behind was turned off? Can
you give the output of,</div>
<div><br>
</div>
<div># gluster volume info <volname></div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span> <br>
Thanks<span class="m_9072513899500000032HOEnZb"><font color="#888888"><br>
<br>
Pat<br>
<br>
<br>
</font></span></span>
<div>
<div class="m_9072513899500000032h5"><br>
<br>
<div class="m_9072513899500000032m_-2192927334915852989moz-cite-prefix">On
06/25/2018 09:39 PM, Raghavendra Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Jun 26, 2018
at 3:21 AM, Pat Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> <br>
Hi Raghavendra,<br>
<br>
Setting the performance.write-behind off
had a small improvement on the write
speed (~3%),<br>
<br>
We were unable to turn on "group
metadata-cache". When we try get errors
like<br>
<br>
# gluster volume set data-volume group
metadata-cache<br>
'/var/lib/glusterd/groups/meta<wbr>data-cache'
file format not valid.<br>
<br>
Was metadata-cache available for gluster
3.7.11? We ask because the release notes
for 3.11 mentions “Feature for
metadata-caching/small file performance
is production ready.” (<a class="m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-freetext" href="https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/" target="_blank">https://gluster.readthedocs.i<wbr>o/en/latest/release-notes/3.11<wbr>.0/</a>).<br>
<br>
Do any of these results suggest
anything? If not, what further tests
would be useful?<br>
</div>
</blockquote>
<div><br>
</div>
<div>Group metadata-cache is just a bunch of
options one sets on a volume. So, You can
set them manually using gluster cli.
Following are the options and their
values:</div>
<div><br>
</div>
<div>performance.md-cache-timeout=6<wbr>00<br>
network.inode-lru-limit=50000<br>
<br>
</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> <br>
Thanks<span class="m_9072513899500000032m_-2192927334915852989gmail-HOEnZb"><font color="#888888"><br>
<br>
Pat</font></span>
<div>
<div class="m_9072513899500000032m_-2192927334915852989gmail-h5"><br>
<br>
<br>
<br>
<div class="m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-cite-prefix">On
06/22/2018 07:51 AM, Raghavendra
Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Jun 21, 2018 at 8:41
PM, Pat Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> <br>
Hi Raghavendra,<br>
<br>
Thanks for the
suggestions. Our
technician will be in on
Monday. We'll test then
and let you know the
results.<br>
<br>
One question I have, is
the "group
metadata-cache" option
supposed to directly
impact the performance
or is it to help collect
data? If the latter,
where will the data be
located?<br>
</div>
</blockquote>
<div><br>
</div>
<div>It impacts performance.</div>
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> <br>
Thanks again.<span class="m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266HOEnZb"><font color="#888888"><br>
<br>
Pat</font></span>
<div>
<div class="m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266h5"><br>
<br>
<br>
<div class="m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-cite-prefix">On
06/21/2018 01:01
AM, Raghavendra
Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Jun 21,
2018 at 10:24
AM,
Raghavendra
Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>For the
case of writes
to glusterfs
mount,<br>
<br>
</div>
I saw in
earlier
conversations
that there are
too many
lookups, but
small number
of writes.
Since writes
cached in
write-behind
would
invalidate
metadata
cache, lookups
won't be
absorbed by
md-cache. I am
wondering what
would results
look like if
we turn off
performance.write-behind.<br>
<br>
</div>
@Pat,<br>
<br>
</div>
Can you set,<br>
<br>
</div>
# gluster
volume set
<volname>
performance.write-behind off<br>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Please
turn on "group
metadata-cache" for write tests too.</div>
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div><br>
</div>
and redo the
tests writing
to glusterfs
mount? Let us
know about the
results you
see.<br>
<br>
</div>
regards,<br>
</div>
Raghavendra<br>
</div>
<div class="m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663HOEnZb">
<div class="m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Jun 21,
2018 at 8:33
AM,
Raghavendra
Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote"><span>On
Thu, Jun 21,
2018 at 8:32
AM,
Raghavendra
Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>For the
case of
reading from
Glusterfs
mount,
read-ahead
should help.
However, we've
known issues
with
read-ahead[1][2].
To work around
these, can you
try with,<br>
<br>
</div>
1. Turn off
performance.open-behind<br>
</div>
#gluster
volume set
<volname>
performance.open-behind off<br>
<br>
<div>2. enable
group meta
metadata-cache<br>
# gluster
volume set
<volname>
group
metadata-cache</div>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>[1] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1084508" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1084508</a></div>
<div>[2] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1214489" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1214489</a></div>
<span>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div class="m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663m_-7085830777103976589m_-1104802400275053683gmail-m_5499637752699686457h5"><br>
<br>
<div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Jun 21,
2018 at 5:00
AM, Pat Haley
<span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Hi,<br>
<br>
We were
recently
revisiting our
problems with
the slowness
of gluster
writes (<a href="http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html" rel="noreferrer" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2017-April<wbr>/030529.html</a>).
Specifically
we were
testing the
suggestions in
a recent post
(<a href="http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html" rel="noreferrer" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2018-March<wbr>/033699.html</a>).
The first two
suggestions
(specifying a
negative-timeout in the mount settings or adding
rpc-auth-allow-insecure to glusterd.vol) did not improve our
performance,
while setting
"disperse.eager-lock off" provided a tiny (5%) speed-up.<br>
<br>
Some of the
various tests
we have tried
earlier can be
seen in the
links below.
Do any of the
above
observations
suggest what
we could try
next to either
improve the
speed or debug
the issue?
Thanks<br>
<br>
<a href="http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-June/0<wbr>31565.html</a><br>
<a href="http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-May/03<wbr>0937.html</a><br>
<br>
Pat<br>
<br>
-- <br>
<br>
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-<br>
Pat Haley
Email:
<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a><br>
Center for
Ocean
Engineering
Phone:
(617) 253-6824<br>
Dept. of
Mechanical
Engineering
Fax: (617)
253-8125<br>
MIT, Room
5-213
<a href="http://web.mit.edu/phaley/www/" rel="noreferrer" target="_blank">http://web.mit.edu/phaley/www/</a><br>
77
Massachusetts
Avenue<br>
Cambridge, MA
02139-4301<br>
<br>
______________________________<wbr>_________________<br>
Gluster-users
mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a></blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</span></div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_9072513899500000032m_-2192927334915852989moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_9072513899500000032m_-2192927334915852989moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_9072513899500000032m_-2192927334915852989moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_9072513899500000032moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_9072513899500000032moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_9072513899500000032moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div></div></div>
</blockquote></div><br></div></div>