<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 6, 2018 at 5:29 AM, Pat Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<br>
Hi Raghavendra,<br>
<br>
Our technician may have some time to look at this issue tomorrow.
Are there any tests that you'd like to see?<br></div></blockquote><div><br></div><div>Sorry. I've been busy with other things and was away from work for couple of days. It'll take me another 2 days to work on this issue again. So, most likely you'll have an update on this next week.<br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
<br>
Thanks<span class="HOEnZb"><font color="#888888"><br>
<br>
Pat</font></span><div><div class="h5"><br>
<br>
<br>
<div class="m_-1517267327311572646moz-cite-prefix">On 06/29/2018 11:25 PM, Raghavendra
Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Jun 29, 2018 at 10:38 PM, Pat
Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
Hi Raghavendra,<br>
<br>
We ran the tests (write tests) and I copied the log
files for both the server and the client to <a class="m_-1517267327311572646m_9072513899500000032moz-txt-link-freetext" href="http://mseas.mit.edu/download/phaley/GlusterUsers/2018/Jun29/" target="_blank">http://mseas.mit.edu/download/<wbr>phaley/GlusterUsers/2018/Jun29<wbr>/</a>
. Is there any additional trace information you need?
(If so, where should I look for it?)<br>
</div>
</blockquote>
<div><br>
</div>
<div>Nothing for now. I can see from logs that workaround is
not helping. fstat requests are not absorbed by md-cache
and read-ahead is witnessing them and flushing its
read-ahead cache. I am investigating more on md-cache (It
also seems to be invalidating inodes quite frequently
which actually might be the root cause of seeing so many
fstat requests from kernel). Will post when I find
anything relevant.</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
Also the volume information you requested<br>
<br>
[root@mseas-data2 ~]# gluster volume info data-volume<br>
<br>
Volume Name: data-volume<br>
Type: Distribute<br>
Volume ID: c162161e-2a2d-4dac-b015-f31fd8<wbr>9ceb18<br>
Status: Started<br>
Number of Bricks: 2<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: mseas-data2:/mnt/brick1<br>
Brick2: mseas-data2:/mnt/brick2<br>
Options Reconfigured:<br>
diagnostics.client-log-level: TRACE<br>
network.inode-lru-limit: 50000<br>
performance.md-cache-timeout: 60<br>
performance.open-behind: off<br>
disperse.eager-lock: off<br>
auth.allow: *<br>
server.allow-insecure: on<br>
nfs.exports-auth-enable: on<br>
diagnostics.brick-sys-log-leve<wbr>l: WARNING<br>
performance.readdir-ahead: on<br>
nfs.disable: on<br>
nfs.export-volumes: off<br>
[root@mseas-data2 ~]# <br>
<div>
<div class="m_-1517267327311572646h5"> <br>
<br>
<div class="m_-1517267327311572646m_9072513899500000032moz-cite-prefix">On
06/29/2018 12:28 PM, Raghavendra Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Jun 29, 2018
at 8:24 PM, Pat Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
Hi Raghavendra,<br>
<br>
Our technician was able to try the
manual setting today. He found that our
upper limit for <span>performance.md-cache-timeo<wbr>ut
was 60 not 600, so he used that value,
along with the </span><span>network.inode-lru-limit=50000.<br>
<br>
The result was another small (~1%)
increase in speed. Does this suggest
some addition tests/changes we could
try?<br>
</span></div>
</blockquote>
<div><br>
</div>
<div>Can you set gluster option
diagnostics.client-log-level to TRACE and
run sequential read tests again (with
md-cache-timeout value of 60)? <br>
</div>
<div><br>
</div>
<div>#gluster volume set <volname>
diagnostics.client-log-level TRACE</div>
<div><br>
</div>
<div>Also are you sure that open-behind was
turned off? Can you give the output of,</div>
<div><br>
</div>
<div># gluster volume info <volname></div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span>
<br>
Thanks<span class="m_-1517267327311572646m_9072513899500000032HOEnZb"><font color="#888888"><br>
<br>
Pat<br>
<br>
<br>
</font></span></span>
<div>
<div class="m_-1517267327311572646m_9072513899500000032h5"><br>
<br>
<div class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989moz-cite-prefix">On
06/25/2018 09:39 PM, Raghavendra
Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Tue, Jun 26, 2018 at 3:21
AM, Pat Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> <br>
Hi Raghavendra,<br>
<br>
Setting the
performance.write-behind
off had a small
improvement on the write
speed (~3%),<br>
<br>
We were unable to turn
on "group
metadata-cache". When
we try get errors like<br>
<br>
# gluster volume set
data-volume group
metadata-cache<br>
'/var/lib/glusterd/groups/meta<wbr>data-cache' file format not valid.<br>
<br>
Was metadata-cache
available for gluster
3.7.11? We ask because
the release notes for
3.11 mentions “Feature
for
metadata-caching/small
file performance is
production ready.” (<a class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-freetext" href="https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/" target="_blank">https://gluster.readthedocs.i<wbr>o/en/latest/release-notes/3.11<wbr>.0/</a>).<br>
<br>
Do any of these results
suggest anything? If
not, what further tests
would be useful?<br>
</div>
</blockquote>
<div><br>
</div>
<div>Group metadata-cache is
just a bunch of options
one sets on a volume. So,
You can set them manually
using gluster cli.
Following are the options
and their values:</div>
<div><br>
</div>
<div>performance.md-cache-timeout=6<wbr>00<br>
network.inode-lru-limit=50000<br>
<br>
</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> <br>
Thanks<span class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-HOEnZb"><font color="#888888"><br>
<br>
Pat</font></span>
<div>
<div class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-h5"><br>
<br>
<br>
<br>
<div class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-cite-prefix">On
06/22/2018 07:51
AM, Raghavendra
Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Jun 21,
2018 at 8:41
PM, Pat Haley
<span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<br>
Hi
Raghavendra,<br>
<br>
Thanks for the
suggestions.
Our technician
will be in on
Monday. We'll
test then and
let you know
the results.<br>
<br>
One question I
have, is the
"group
metadata-cache"
option
supposed to
directly
impact the
performance or
is it to help
collect data?
If the latter,
where will the
data be
located?<br>
</div>
</blockquote>
<div><br>
</div>
<div>It
impacts
performance.</div>
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<br>
Thanks again.<span class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266HOEnZb"><font color="#888888"><br>
<br>
Pat</font></span>
<div>
<div class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266h5"><br>
<br>
<br>
<div class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-cite-prefix">On
06/21/2018
01:01 AM,
Raghavendra
Gowdappa
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Jun 21,
2018 at 10:24
AM,
Raghavendra
Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>For the
case of writes
to glusterfs
mount,<br>
<br>
</div>
I saw in
earlier
conversations
that there are
too many
lookups, but
small number
of writes.
Since writes
cached in
write-behind
would
invalidate
metadata
cache, lookups
won't be
absorbed by
md-cache. I am
wondering what
would results
look like if
we turn off
performance.write-behind.<br>
<br>
</div>
@Pat,<br>
<br>
</div>
Can you set,<br>
<br>
</div>
# gluster
volume set
<volname>
performance.write-behind off<br>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Please
turn on "group
metadata-cache" for write tests too.</div>
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div><br>
</div>
and redo the
tests writing
to glusterfs
mount? Let us
know about the
results you
see.<br>
<br>
</div>
regards,<br>
</div>
Raghavendra<br>
</div>
<div class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663HOEnZb">
<div class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Jun 21,
2018 at 8:33
AM,
Raghavendra
Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote"><span>On
Thu, Jun 21,
2018 at 8:32
AM,
Raghavendra
Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>For the
case of
reading from
Glusterfs
mount,
read-ahead
should help.
However, we've
known issues
with
read-ahead[1][2].
To work around
these, can you
try with,<br>
<br>
</div>
1. Turn off
performance.open-behind<br>
</div>
#gluster
volume set
<volname>
performance.open-behind off<br>
<br>
<div>2. enable
group meta
metadata-cache<br>
# gluster
volume set
<volname>
group
metadata-cache</div>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>[1] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1084508" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1084508</a></div>
<div>[2] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1214489" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1214489</a></div>
<span>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663m_-7085830777103976589m_-1104802400275053683gmail-m_5499637752699686457h5"><br>
<br>
<div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Jun 21,
2018 at 5:00
AM, Pat Haley
<span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Hi,<br>
<br>
We were
recently
revisiting our
problems with
the slowness
of gluster
writes (<a href="http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html" rel="noreferrer" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2017-April<wbr>/030529.html</a>).
Specifically
we were
testing the
suggestions in
a recent post
(<a href="http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html" rel="noreferrer" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2018-March<wbr>/033699.html</a>).
The first two
suggestions
(specifying a
negative-timeout in the mount settings or adding
rpc-auth-allow-insecure to glusterd.vol) did not improve our
performance,
while setting
"disperse.eager-lock off" provided a tiny (5%) speed-up.<br>
<br>
Some of the
various tests
we have tried
earlier can be
seen in the
links below.
Do any of the
above
observations
suggest what
we could try
next to either
improve the
speed or debug
the issue?
Thanks<br>
<br>
<a href="http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-June/0<wbr>31565.html</a><br>
<a href="http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-May/03<wbr>0937.html</a><br>
<br>
Pat<br>
<br>
-- <br>
<br>
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-<br>
Pat Haley
Email:
<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a><br>
Center for
Ocean
Engineering
Phone:
(617) 253-6824<br>
Dept. of
Mechanical
Engineering
Fax: (617)
253-8125<br>
MIT, Room
5-213
<a href="http://web.mit.edu/phaley/www/" rel="noreferrer" target="_blank">http://web.mit.edu/phaley/www/</a><br>
77
Massachusetts
Avenue<br>
Cambridge, MA
02139-4301<br>
<br>
______________________________<wbr>_________________<br>
Gluster-users
mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a></blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</span></div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Gluster-users
mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_-1517267327311572646m_9072513899500000032moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-1517267327311572646m_9072513899500000032moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-1517267327311572646m_9072513899500000032moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_-1517267327311572646moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-1517267327311572646moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-1517267327311572646moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div></div></div>
</blockquote></div><br></div></div>