<div dir="ltr">Let's start with the same 'dd' test we were testing with to see, what the numbers are. Please provide profile numbers for the same. From there on we will start tuning the volume to see what we can do.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 30, 2017 at 9:16 PM, Pat Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<br>
Hi Pranith,<br>
<br>
Thanks for the tip. We now have the gluster volume mounted under
/home. What tests do you recommend we run?<br>
<br>
Thanks<span class="HOEnZb"><font color="#888888"><br>
<br>
Pat</font></span><div><div class="h5"><br>
<br>
<br>
<div class="m_-3319843143517043073moz-cite-prefix">On 05/17/2017 05:01 AM, Pranith Kumar
Karampuri wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, May 16, 2017 at 9:20 PM, Pat
Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <br>
Hi Pranith,<br>
<br>
Sorry for the delay. I never saw received your reply
(but I did receive Ben Turner's follow-up to your
reply). So we tried to create a gluster volume under
/home using different variations of<br>
<br>
<tt><span>gluster volume create test-volume
mseas-data2:/home/gbrick_test_<wbr>1
mseas-data2:/home/gbrick_test_<wbr>2 transport tcp</span></tt><tt><br>
</tt><br>
However we keep getting errors of the form<br>
<br>
<tt>Wrong brick type: transport, use
<HOSTNAME>:<export-dir-abs-pat<wbr>h></tt><tt><br>
</tt><br>
Any thoughts on what we're doing wrong?<br>
</div>
</blockquote>
<div><br>
</div>
<div>You should give transport tcp at the beginning I think.
Anyways, transport tcp is the default, so no need to
specify so remove those two words from the CLI. <br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <br>
Also do you have a list of the test we should be running
once we get this volume created? Given the time-zone
difference it might help if we can run a small battery
of tests and post the results rather than test-post-new
test-post... .<br>
</div>
</blockquote>
<div><br>
</div>
<div>This is the first time I am doing performance analysis
on users as far as I remember. In our team there are
separate engineers who do these tests. Ben who replied
earlier is one such engineer.<br>
<br>
</div>
<div>Ben,<br>
</div>
<div> Have any suggestions?<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <br>
Thanks<span class="m_-3319843143517043073HOEnZb"><font color="#888888"><br>
<br>
Pat</font></span>
<div>
<div class="m_-3319843143517043073h5"><br>
<br>
<br>
<div class="m_-3319843143517043073m_5903432224896103956moz-cite-prefix">On
05/11/2017 12:06 PM, Pranith Kumar Karampuri
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, May 11, 2017
at 9:32 PM, Pat Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> <br>
Hi Pranith,<br>
<br>
The /home partition is mounted as ext4<br>
<tt>/home ext4
defaults,usrquota,grpquota 1 2</tt><tt><br>
</tt><br>
The brick partitions are mounted ax xfs<br>
<tt>/mnt/brick1 xfs defaults 0 0</tt><tt><br>
</tt><tt>/mnt/brick2 xfs
defaults 0 0</tt><tt><br>
</tt><br>
Will this cause a problem with creating
a volume under /home?<span class="m_-3319843143517043073m_5903432224896103956gmail-HOEnZb"><font color="#888888"><br>
</font></span></div>
</blockquote>
<div><br>
</div>
<div>I don't think the bottleneck is disk.
You can do the same tests you did on your
new volume to confirm?<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><span class="m_-3319843143517043073m_5903432224896103956gmail-HOEnZb"><font color="#888888"> <br>
Pat</font></span>
<div>
<div class="m_-3319843143517043073m_5903432224896103956gmail-h5"><br>
<br>
<br>
<div class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372moz-cite-prefix">On
05/11/2017 11:32 AM, Pranith Kumar
Karampuri wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, May 11, 2017 at 8:57
PM, Pat Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> <br>
Hi Pranith,<br>
<br>
Unfortunately, we don't
have similar hardware
for a small scale test.
All we have is our
production hardware.<span class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372HOEnZb"><font color="#888888"><br>
</font></span></div>
</blockquote>
<div><br>
</div>
<div>You said something
about /home partition
which has lesser disks, we
can create plain
distribute volume inside
one of those directories.
After we are done, we can
remove the setup. What do
you say?<br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><span class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372HOEnZb"><font color="#888888"> <br>
Pat</font></span>
<div>
<div class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372h5"><br>
<br>
<br>
<br>
<div class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417moz-cite-prefix">On
05/11/2017 07:05
AM, Pranith Kumar
Karampuri wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Thu, May 11,
2017 at 2:48
AM, Pat Haley
<span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<br>
Hi Pranith,<br>
<br>
Since we are
mounting the
partitions as
the bricks, I
tried the dd
test writing
to
<brick-path>/.glusterfs/<file-<wbr>to-be-removed-after-test>.
The results
without
oflag=sync
were 1.6 Gb/s
(faster than
gluster but
not as fast as
I was
expecting
given the 1.2
Gb/s to the
no-gluster
area w/ fewer
disks).<span class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417HOEnZb"><font color="#888888"><br>
</font></span></div>
</blockquote>
<div><br>
</div>
<div>Okay,
then 1.6Gb/s
is what we
need to target
for,
considering
your volume is
just
distribute. Is
there any way
you can do
tests on
similar
hardware but
at a small
scale? Just so
we can run the
workload to
learn more
about the
bottlenecks in
the system? We
can probably
try to get the
speed to
1.2Gb/s on
your /home
partition you
were telling
me yesterday.
Let me know if
that is
something you
are okay to
do.<br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><span class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417HOEnZb"><font color="#888888"> <br>
Pat</font></span>
<div>
<div class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417h5"><br>
<br>
<br>
<div class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055moz-cite-prefix">On
05/10/2017
01:27 PM,
Pranith Kumar
Karampuri
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Wed, May 10,
2017 at 10:15
PM, Pat Haley
<span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<br>
Hi Pranith,<br>
<br>
Not entirely
sure (this
isn't my area
of
expertise).
I'll run your
answer by some
other people
who are more
familiar with
this.<br>
<br>
I am also
uncertain
about how to
interpret the
results when
we also add
the dd tests
writing to the
/home area (no
gluster, still
on the same
machine)<br>
<ul>
<li>dd test
without
oflag=sync
(rough average
of multiple
tests)<br>
</li>
<ul>
<li>gluster w/
fuse mount :
570 Mb/s</li>
<li>gluster w/
nfs mount:
390 Mb/s</li>
<li>nfs (no
gluster): 1.2
Gb/s</li>
</ul>
<li>dd test
with
oflag=sync
(rough average
of multiple
tests)</li>
<ul>
<li>gluster w/
fuse mount: 5
Mb/s</li>
<li>gluster w/
nfs mount:
200 Mb/s</li>
<li>nfs (no
gluster): 20
Mb/s<br>
</li>
</ul>
</ul>
Given that the
non-gluster
area is a
RAID-6 of 4
disks while
each brick of
the gluster
area is a
RAID-6 of 32
disks, I would
naively expect
the writes to
the gluster
area to be
roughly 8x
faster than to
the
non-gluster.<br>
</div>
</blockquote>
<div><br>
</div>
<div>I think a
better test is
to try and
write to a
file using nfs
without any
gluster to a
location that
is not inside
the brick but
someother
location that
is on same
disk(s). If
you are
mounting the
partition as
the brick,
then we can
write to a
file inside
.glusterfs
directory,
something like
<brick-path>/.glusterfs/<file-<wbr>to-be-removed-after-test>.
<br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<br>
I still think
we have a
speed issue, I
can't tell if
fuse vs nfs is
part of the
problem.</div>
</blockquote>
<div><br>
</div>
I got
interested in
the post
because I read
that fuse
speed is
lesser than
nfs speed
which is
counter-intuitive
to my
understanding.
So wanted
clarifications.
Now that I got
my
clarifications
where fuse
outperformed
nfs without
sync, we can
resume testing
as described
above and try
to find what
it is. Based
on your
email-id I am
guessing you
are from
Boston and I
am from
Bangalore so
if you are
okay with
doing this
debugging for
multiple days
because of
timezones, I
will be happy
to help.
Please be a
bit patient
with me, I am
under a
release crunch
but I am very
curious with
the problem
you posted.<br>
<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
Was there
anything
useful in the
profiles?<span class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055HOEnZb"><font color="#888888"><br>
</font></span></div>
</blockquote>
<div><br>
</div>
<div>Unfortunately
profiles
didn't help me
much, I think
we are
collecting the
profiles from
an active
volume, so it
has a lot of
information
that is not
pertaining to
dd so it is
difficult to
find the
contributions
of dd. So I
went through
your post
again and
found
something I
didn't pay
much attention
to earlier
i.e.
oflag=sync, so
did my own
tests on my
setup with
FUSE so sent
that reply.<br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><span class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055HOEnZb"><font color="#888888"> <br>
Pat</font></span>
<div>
<div class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055h5"><br>
<br>
<br>
<div class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930moz-cite-prefix">On
05/10/2017
12:15 PM,
Pranith Kumar
Karampuri
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Okay
good. At least
this validates
my doubts.
Handling
O_SYNC in
gluster NFS
and fuse is a
bit different.<br>
</div>
When
application
opens a file
with O_SYNC on
fuse mount
then each
write syscall
has to be
written to
disk as part
of the syscall
where as in
case of NFS,
there is no
concept of
open. NFS
performs write
though a
handle saying
it needs to be
a synchronous
write, so
write()
syscall is
performed
first then it
performs
fsync(). so an
write on an fd
with O_SYNC
becomes
write+fsync. I
am suspecting
that when
multiple
threads do
this
write+fsync()
operation on
the same file,
multiple
writes are
batched
together to be
written do
disk so the
throughput on
the disk is
increasing is
my guess.<br>
<br>
</div>
Does it answer
your doubts?<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Wed, May 10,
2017 at 9:35
PM, Pat Haley
<span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<br>
Without the
oflag=sync and
only a single
test of each,
the FUSE is
going faster
than NFS:<br>
<br>
FUSE:<br>
<tt>mseas-data2(dri_nascar)%
dd
if=/dev/zero
count=4096
bs=1048576
of=zeros.txt
conv=sync</tt><tt><br>
</tt><tt>4096+0
records in</tt><tt><br>
</tt><tt>4096+0
records out</tt><tt><br>
</tt><tt>4294967296
bytes (4.3 GB)
copied,
7.46961 s, 575
MB/s</tt><tt><br>
</tt><tt><br>
<br>
</tt>NFS<br>
<tt>mseas-data2(HYCOM)%
dd
if=/dev/zero
count=4096
bs=1048576
of=zeros.txt
conv=sync</tt><tt><br>
</tt><tt>4096+0
records in</tt><tt><br>
</tt><tt>4096+0
records out</tt><tt><br>
</tt><tt>4294967296
bytes (4.3 GB)
copied,
11.4264 s, 376
MB/s</tt>
<div>
<div class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930h5"><tt><br>
</tt><tt><br>
</tt><tt><br>
</tt>
<div class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365moz-cite-prefix">On
05/10/2017
11:53 AM,
Pranith Kumar
Karampuri
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Could
you let me
know the speed
without
oflag=sync on
both the
mounts? No
need to
collect
profiles.<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Wed, May 10,
2017 at 9:17
PM, Pat Haley
<span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<br>
Here is what I
see now:<br>
<br>
<tt>[root@mseas-data2
~]# gluster
volume info</tt><span><tt><br>
</tt><tt> </tt><tt><br>
</tt><tt>Volume
Name:
data-volume</tt><tt><br>
</tt><tt>Type:
Distribute</tt><tt><br>
</tt><tt>Volume
ID:
c162161e-2a2d-4dac-b015-f31fd8<wbr>9ceb18</tt><tt><br>
</tt><tt>Status:
Started</tt><tt><br>
</tt><tt>Number
of Bricks: 2</tt><tt><br>
</tt><tt>Transport-type:
tcp</tt><tt><br>
</tt><tt>Bricks:</tt><tt><br>
</tt><tt>Brick1:
mseas-data2:/mnt/brick1</tt><tt><br>
</tt><tt>Brick2:
mseas-data2:/mnt/brick2</tt><tt><br>
</tt><tt>Options
Reconfigured:</tt><tt><br>
</tt></span><tt>diagnostics.count-fop-hits:
on</tt><tt><br>
</tt><tt>diagnostics.latency-measuremen<wbr>t:
on</tt><tt><br>
</tt><tt>nfs.exports-auth-enable:
on</tt><tt><br>
</tt><tt>diagnostics.brick-sys-log-leve<wbr>l:
WARNING</tt><span><tt><br>
</tt><tt>performance.readdir-ahead:
on</tt><tt><br>
</tt><tt>nfs.disable:
on</tt><tt><br>
</tt><tt>nfs.export-volumes:
off</tt><tt><br>
</tt><br>
<br>
<br>
</span>
<div>
<div class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365h5">
<div class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365m_5079054141158038028moz-cite-prefix">On
05/10/2017
11:44 AM,
Pranith Kumar
Karampuri
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Is this
the volume
info you have?<br>
<br>
<pre>><i> [<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">root at mseas-data2</a> ~]# gluster volume info
</i>><i>
</i>><i> Volume Name: data-volume
</i>><i> Type: Distribute
</i>><i> Volume ID: c162161e-2a2d-4dac-b015-f31fd8<wbr>9ceb18
</i>><i> Status: Started
</i>><i> Number of Bricks: 2
</i>><i> Transport-type: tcp
</i>><i> Bricks:
</i>><i> Brick1: mseas-data2:/mnt/brick1
</i>><i> Brick2: mseas-data2:/mnt/brick2
</i>><i> Options Reconfigured:
</i>><i> performance.readdir-ahead: on
</i>><i> nfs.disable: on
</i>><i> nfs.export-volumes: off
</i></pre>
</div>
I copied this
from old
thread from
2016. This is
distribute
volume. Did
you change any
of the options
in between? </div>
</blockquote>
</div>
</div>
<span>
<pre class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365m_5079054141158038028moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365m_5079054141158038028moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365m_5079054141158038028moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</span></div>
</blockquote>
</div>
--
<div class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365gmail_signature">
<div dir="ltr">Pranith
</div>
</div>
</div>
</blockquote>
<pre class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
</blockquote>
</div>
--
<div class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930gmail_signature">
<div dir="ltr">Pranith
</div>
</div>
</div>
</blockquote>
<pre class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
</blockquote>
</div>
--
<div class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055gmail_signature">
<div dir="ltr">Pranith
</div>
</div>
</div>
</div>
</blockquote>
<pre class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
</blockquote>
</div>
--
<div class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417gmail_signature">
<div dir="ltr">Pranith
</div>
</div>
</div>
</div>
</blockquote>
<pre class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
</blockquote>
</div>
--
<div class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372gmail_signature">
<div dir="ltr">Pranith
</div>
</div>
</div>
</div>
</blockquote>
<pre class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
</blockquote>
</div>
--
<div class="m_-3319843143517043073m_5903432224896103956gmail_signature">
<div dir="ltr">Pranith
</div>
</div>
</div>
</div>
</blockquote>
<pre class="m_-3319843143517043073m_5903432224896103956moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-3319843143517043073m_5903432224896103956moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-3319843143517043073m_5903432224896103956moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div class="m_-3319843143517043073gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="m_-3319843143517043073moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_-3319843143517043073moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_-3319843143517043073moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div>