<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 13, 2018 at 5:00 AM, Pat Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<br>
Hi Raghavendra,<br>
<br>
We were wondering if you have had a chance to look at this again,
and if so, did you have any further suggestions?<br></div></blockquote><div><br></div><div>Sorry Pat. Too much of work :). I'll be working on a patch today to make sure read-ahead doesn't flush its cache due to fstats making this behavior optional. You can try this patch and let us know about results.</div><div>Will let you know when patch is ready.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
<br>
Thanks<span class="HOEnZb"><font color="#888888"><br>
<br>
Pat</font></span><div><div class="h5"><br>
<br>
<br>
<div class="m_-6455985595820935451moz-cite-prefix">On 07/06/2018 01:27 AM, Raghavendra
Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Jul 6, 2018 at 5:29 AM, Pat
Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
Hi Raghavendra,<br>
<br>
Our technician may have some time to look at this issue
tomorrow. Are there any tests that you'd like to see?<br>
</div>
</blockquote>
<div><br>
</div>
<div>Sorry. I've been busy with other things and was away
from work for couple of days. It'll take me another 2 days
to work on this issue again. So, most likely you'll have
an update on this next week.<br>
</div>
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
Thanks<span class="m_-6455985595820935451HOEnZb"><font color="#888888"><br>
<br>
Pat</font></span>
<div>
<div class="m_-6455985595820935451h5"><br>
<br>
<br>
<div class="m_-6455985595820935451m_-1517267327311572646moz-cite-prefix">On
06/29/2018 11:25 PM, Raghavendra Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Jun 29, 2018
at 10:38 PM, Pat Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
Hi Raghavendra,<br>
<br>
We ran the tests (write tests) and I
copied the log files for both the server
and the client to <a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032moz-txt-link-freetext" href="http://mseas.mit.edu/download/phaley/GlusterUsers/2018/Jun29/" target="_blank">http://mseas.mit.edu/download/<wbr>phaley/GlusterUsers/2018/Jun29<wbr>/</a>
. Is there any additional trace
information you need? (If so, where
should I look for it?)<br>
</div>
</blockquote>
<div><br>
</div>
<div>Nothing for now. I can see from logs
that workaround is not helping. fstat
requests are not absorbed by md-cache and
read-ahead is witnessing them and flushing
its read-ahead cache. I am investigating
more on md-cache (It also seems to be
invalidating inodes quite frequently which
actually might be the root cause of seeing
so many fstat requests from kernel). Will
post when I find anything relevant.</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
Also the volume information you
requested<br>
<br>
[root@mseas-data2 ~]# gluster volume
info data-volume<br>
<br>
Volume Name: data-volume<br>
Type: Distribute<br>
Volume ID:
c162161e-2a2d-4dac-b015-f31fd8<wbr>9ceb18<br>
Status: Started<br>
Number of Bricks: 2<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: mseas-data2:/mnt/brick1<br>
Brick2: mseas-data2:/mnt/brick2<br>
Options Reconfigured:<br>
diagnostics.client-log-level: TRACE<br>
network.inode-lru-limit: 50000<br>
performance.md-cache-timeout: 60<br>
performance.open-behind: off<br>
disperse.eager-lock: off<br>
auth.allow: *<br>
server.allow-insecure: on<br>
nfs.exports-auth-enable: on<br>
diagnostics.brick-sys-log-leve<wbr>l:
WARNING<br>
performance.readdir-ahead: on<br>
nfs.disable: on<br>
nfs.export-volumes: off<br>
[root@mseas-data2 ~]# <br>
<div>
<div class="m_-6455985595820935451m_-1517267327311572646h5">
<br>
<br>
<div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032moz-cite-prefix">On
06/29/2018 12:28 PM, Raghavendra
Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Fri, Jun 29, 2018 at 8:24
PM, Pat Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"> <br>
Hi Raghavendra,<br>
<br>
Our technician was able
to try the manual
setting today. He found
that our upper limit
for <span>performance.md-cache-timeo<wbr>ut
was 60 not 600, so he
used that value, along
with the </span><span>network.inode-lru-limit=50000.<br>
<br>
The result was another
small (~1%) increase
in speed. Does this
suggest some addition
tests/changes we could
try?<br>
</span></div>
</blockquote>
<div><br>
</div>
<div>Can you set gluster
option
diagnostics.client-log-level
to TRACE and run
sequential read tests
again (with
md-cache-timeout value of
60)? <br>
</div>
<div><br>
</div>
<div>#gluster volume set
<volname>
diagnostics.client-log-level
TRACE</div>
<div><br>
</div>
<div>Also are you sure that
open-behind was turned
off? Can you give the
output of,</div>
<div><br>
</div>
<div># gluster volume info
<volname></div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span>
<br>
Thanks<span class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032HOEnZb"><font color="#888888"><br>
<br>
Pat<br>
<br>
<br>
</font></span></span>
<div>
<div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032h5"><br>
<br>
<div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989moz-cite-prefix">On
06/25/2018 09:39
PM, Raghavendra
Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Tue, Jun 26,
2018 at 3:21
AM, Pat Haley
<span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<br>
Hi
Raghavendra,<br>
<br>
Setting the
performance.write-behind
off had a
small
improvement on
the write
speed (~3%),<br>
<br>
We were unable
to turn on
"group
metadata-cache".
When we try
get errors
like<br>
<br>
# gluster
volume set
data-volume
group
metadata-cache<br>
'/var/lib/glusterd/groups/meta<wbr>data-cache' file format not valid.<br>
<br>
Was
metadata-cache
available for
gluster
3.7.11? We ask
because the
release notes
for 3.11
mentions
“Feature for
metadata-caching/small
file
performance is
production
ready.” (<a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-freetext" href="https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/" target="_blank">https://gluster.readthedocs.i<wbr>o/en/latest/release-notes/3.11<wbr>.0/</a>).<br>
<br>
Do any of
these results
suggest
anything? If
not, what
further tests
would be
useful?<br>
</div>
</blockquote>
<div><br>
</div>
<div>Group
metadata-cache
is just a
bunch of
options one
sets on a
volume. So,
You can set
them manually
using gluster
cli. Following
are the
options and
their values:</div>
<div><br>
</div>
<div>performance.md-cache-timeout=6<wbr>00<br>
network.inode-lru-limit=50000<br>
<br>
</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<br>
Thanks<span class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-HOEnZb"><font color="#888888"><br>
<br>
Pat</font></span>
<div>
<div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-h5"><br>
<br>
<br>
<br>
<div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-cite-prefix">On
06/22/2018
07:51 AM,
Raghavendra
Gowdappa
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Jun 21,
2018 at 8:41
PM, Pat Haley
<span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<br>
Hi
Raghavendra,<br>
<br>
Thanks for the
suggestions.
Our technician
will be in on
Monday. We'll
test then and
let you know
the results.<br>
<br>
One question I
have, is the
"group
metadata-cache"
option
supposed to
directly
impact the
performance or
is it to help
collect data?
If the latter,
where will the
data be
located?<br>
</div>
</blockquote>
<div><br>
</div>
<div>It
impacts
performance.</div>
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<br>
Thanks again.<span class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266HOEnZb"><font color="#888888"><br>
<br>
Pat</font></span>
<div>
<div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266h5"><br>
<br>
<br>
<div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-cite-prefix">On
06/21/2018
01:01 AM,
Raghavendra
Gowdappa
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Jun 21,
2018 at 10:24
AM,
Raghavendra
Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>For the
case of writes
to glusterfs
mount,<br>
<br>
</div>
I saw in
earlier
conversations
that there are
too many
lookups, but
small number
of writes.
Since writes
cached in
write-behind
would
invalidate
metadata
cache, lookups
won't be
absorbed by
md-cache. I am
wondering what
would results
look like if
we turn off
performance.write-behind.<br>
<br>
</div>
@Pat,<br>
<br>
</div>
Can you set,<br>
<br>
</div>
# gluster
volume set
<volname>
performance.write-behind off<br>
</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Please
turn on "group
metadata-cache" for write tests too.</div>
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div><br>
</div>
and redo the
tests writing
to glusterfs
mount? Let us
know about the
results you
see.<br>
<br>
</div>
regards,<br>
</div>
Raghavendra<br>
</div>
<div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663HOEnZb">
<div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Jun 21,
2018 at 8:33
AM,
Raghavendra
Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote"><span>On
Thu, Jun 21,
2018 at 8:32
AM,
Raghavendra
Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>For the
case of
reading from
Glusterfs
mount,
read-ahead
should help.
However, we've
known issues
with
read-ahead[1][2].
To work around
these, can you
try with,<br>
<br>
</div>
1. Turn off
performance.open-behind<br>
</div>
#gluster
volume set
<volname>
performance.open-behind off<br>
<br>
<div>2. enable
group meta
metadata-cache<br>
# gluster
volume set
<volname>
group
metadata-cache</div>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>[1] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1084508" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1084508</a></div>
<div>[2] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1214489" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1214489</a></div>
<span>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663m_-7085830777103976589m_-1104802400275053683gmail-m_5499637752699686457h5"><br>
<br>
<div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, Jun 21,
2018 at 5:00
AM, Pat Haley
<span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Hi,<br>
<br>
We were
recently
revisiting our
problems with
the slowness
of gluster
writes (<a href="http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html" rel="noreferrer" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2017-April<wbr>/030529.html</a>).
Specifically
we were
testing the
suggestions in
a recent post
(<a href="http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html" rel="noreferrer" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2018-March<wbr>/033699.html</a>).
The first two
suggestions
(specifying a
negative-timeout in the mount settings or adding
rpc-auth-allow-insecure to glusterd.vol) did not improve our
performance,
while setting
"disperse.eager-lock off" provided a tiny (5%) speed-up.<br>
<br>
Some of the
various tests
we have tried
earlier can be
seen in the
links below.
Do any of the
above
observations
suggest what
we could try
next to either
improve the
speed or debug
the issue?
Thanks<br>
<br>
<a href="http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-June/0<wbr>31565.html</a><br>
<a href="http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-May/03<wbr>0937.html</a><br>
<br>
Pat<br>
<br>
-- <br>
<br>
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-<br>
Pat Haley
Email:
<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a><br>
Center for
Ocean
Engineering
Phone:
(617) 253-6824<br>
Dept. of
Mechanical
Engineering
Fax: (617)
253-8125<br>
MIT, Room
5-213
<a href="http://web.mit.edu/phaley/www/" rel="noreferrer" target="_blank">http://web.mit.edu/phaley/www/</a><br>
77
Massachusetts
Avenue<br>
Cambridge, MA
02139-4301<br>
<br>
______________________________<wbr>_________________<br>
Gluster-users
mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a></blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</span></div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Gluster-users
mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_-6455985595820935451m_-1517267327311572646moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-6455985595820935451m_-1517267327311572646moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-6455985595820935451m_-1517267327311572646moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="m_-6455985595820935451moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-6455985595820935451moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-6455985595820935451moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div></div></div>
</blockquote></div><br></div></div>