<div dir="ltr"><div><div>Thanks this is good information.<br><br></div><div>+Soumya<br><br></div>Soumya,<br></div>       We are trying to find why kNFS is performing way better than plain distribute glusterfs+fuse. What information do you think will benefit us to compare the operations with kNFS vs gluster+fuse? We already have profile output from fuse.<br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 31, 2017 at 7:10 AM, Pat Haley <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div text="#000000" bgcolor="#FFFFFF">
    <br>
    Hi Pranith,<br>
    <br>
    The &quot;dd&quot; command was:<span class=""><br>
    <br>
    <tt>    dd if=/dev/zero count=4096 bs=1048576 of=zeros.txt conv=sync</tt><tt><br>
    </tt><br></span>
    There were 2 instances where dd reported 22 seconds. The output from
    the dd tests are in<br>
    <br>
<a class="m_2414117758133198099moz-txt-link-freetext" href="http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/dd_testvol_gluster.txt" target="_blank">http://mseas.mit.edu/download/<wbr>phaley/GlusterUsers/TestVol/<wbr>dd_testvol_gluster.txt</a><span class="HOEnZb"><font color="#888888"><br>
    <br>
    Pat</font></span><div><div class="h5"><br>
    <br>
    <div class="m_2414117758133198099moz-cite-prefix">On 05/30/2017 09:27 PM, Pranith Kumar
      Karampuri wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">
        <div>Pat,<br>
        </div>
               What is the command you used? As per the following
        output, it seems like at least one write operation took 16
        seconds. Which is really bad.<br>
        <pre>     96.39    1165.10 us      89.00 us <b>16487014.00 us</b>         393212       WRITE</pre>
        <br>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Tue, May 30, 2017 at 10:36 PM, Pat
          Haley <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div text="#000000" bgcolor="#FFFFFF"> <br>
              Hi Pranith,<br>
              <br>
              I ran the same &#39;dd&#39; test both in the gluster test volume
              and in the .glusterfs directory of each brick.  The median
              results (12 dd trials in each test) are similar to before<br>
              <ul>
                <li>gluster test volume: 586.5 MB/s</li>
                <li>bricks (in .glusterfs): 1.4 GB/s<br>
                </li>
              </ul>
              The profile for the gluster test-volume is in<br>
              <br>
              <a class="m_2414117758133198099m_-7360220549516486306moz-txt-link-freetext" href="http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/profile_testvol_gluster.txt" target="_blank">http://mseas.mit.edu/download/<wbr>phaley/GlusterUsers/TestVol/pr<wbr>ofile_testvol_gluster.txt</a><br>
              <br>
              Thanks<span class="m_2414117758133198099HOEnZb"><font color="#888888"><br>
                  <br>
                  Pat</font></span>
              <div>
                <div class="m_2414117758133198099h5"><br>
                  <br>
                  <br>
                  <br>
                  <div class="m_2414117758133198099m_-7360220549516486306moz-cite-prefix">On
                    05/30/2017 12:10 PM, Pranith Kumar Karampuri wrote:<br>
                  </div>
                  <blockquote type="cite">
                    <div dir="ltr">Let&#39;s start with the same &#39;dd&#39; test
                      we were testing with to see, what the numbers are.
                      Please provide profile numbers for the same. From
                      there on we will start tuning the volume to see
                      what we can do.<br>
                    </div>
                    <div class="gmail_extra"><br>
                      <div class="gmail_quote">On Tue, May 30, 2017 at
                        9:16 PM, Pat Haley <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                        wrote:<br>
                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                          <div text="#000000" bgcolor="#FFFFFF"> <br>
                            Hi Pranith,<br>
                            <br>
                            Thanks for the tip.  We now have the gluster
                            volume mounted under /home.  What tests do
                            you recommend we run?<br>
                            <br>
                            Thanks<span class="m_2414117758133198099m_-7360220549516486306HOEnZb"><font color="#888888"><br>
                                <br>
                                Pat</font></span>
                            <div>
                              <div class="m_2414117758133198099m_-7360220549516486306h5"><br>
                                <br>
                                <br>
                                <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073moz-cite-prefix">On
                                  05/17/2017 05:01 AM, Pranith Kumar
                                  Karampuri wrote:<br>
                                </div>
                                <blockquote type="cite">
                                  <div dir="ltr"><br>
                                    <div class="gmail_extra"><br>
                                      <div class="gmail_quote">On Tue,
                                        May 16, 2017 at 9:20 PM, Pat
                                        Haley <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                                        wrote:<br>
                                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                          <div bgcolor="#FFFFFF" text="#000000"> <br>
                                            Hi Pranith,<br>
                                            <br>
                                            Sorry for the delay.  I
                                            never saw received your
                                            reply (but I did receive Ben
                                            Turner&#39;s follow-up to your
                                            reply).  So we tried to
                                            create a gluster volume
                                            under /home using different
                                            variations of<br>
                                            <br>
                                            <tt><span>gluster volume
                                                create test-volume
                                                mseas-data2:/home/gbrick_test_<wbr>1
mseas-data2:/home/gbrick_test_<wbr>2 transport tcp</span></tt><tt><br>
                                            </tt><br>
                                            However we keep getting
                                            errors of the form<br>
                                            <br>
                                            <tt>Wrong brick type:
                                              transport, use
                                              &lt;HOSTNAME&gt;:&lt;export-dir-abs-pat<wbr>h&gt;</tt><tt><br>
                                            </tt><br>
                                            Any thoughts on what we&#39;re
                                            doing wrong?<br>
                                          </div>
                                        </blockquote>
                                        <div><br>
                                        </div>
                                        <div>You should give transport
                                          tcp at the beginning I think.
                                          Anyways, transport tcp is the
                                          default, so no need to specify
                                          so remove those two words from
                                          the CLI. <br>
                                        </div>
                                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                          <div bgcolor="#FFFFFF" text="#000000"> <br>
                                            Also do you have a list of
                                            the test we should be
                                            running once we get this
                                            volume created?  Given the
                                            time-zone difference it
                                            might help if we can run a
                                            small battery of tests and
                                            post the results rather than
                                            test-post-new test-post... .<br>
                                          </div>
                                        </blockquote>
                                        <div><br>
                                        </div>
                                        <div>This is the first time I am
                                          doing performance analysis on
                                          users as far as I remember. In
                                          our team there are separate
                                          engineers who do these tests.
                                          Ben who replied earlier is one
                                          such engineer.<br>
                                          <br>
                                        </div>
                                        <div>Ben,<br>
                                        </div>
                                        <div>    Have any suggestions?<br>
                                        </div>
                                        <div> </div>
                                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                          <div bgcolor="#FFFFFF" text="#000000"> <br>
                                            Thanks<span class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073HOEnZb"><font color="#888888"><br>
                                                <br>
                                                Pat</font></span>
                                            <div>
                                              <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073h5"><br>
                                                <br>
                                                <br>
                                                <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956moz-cite-prefix">On
                                                  05/11/2017 12:06 PM,
                                                  Pranith Kumar
                                                  Karampuri wrote:<br>
                                                </div>
                                                <blockquote type="cite">
                                                  <div dir="ltr"><br>
                                                    <div class="gmail_extra"><br>
                                                      <div class="gmail_quote">On
                                                        Thu, May 11,
                                                        2017 at 9:32 PM,
                                                        Pat Haley <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                                                        wrote:<br>
                                                        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <br>
                                                          Hi Pranith,<br>
                                                          <br>
                                                          The /home
                                                          partition is
                                                          mounted as
                                                          ext4<br>
                                                          <tt>/home   
                                                                   
                                                          ext4   
                                                          defaults,usrquota,grpquota   
                                                            1 2</tt><tt><br>
                                                          </tt><br>
                                                          The brick
                                                          partitions are
                                                          mounted ax xfs<br>
                                                          <tt>/mnt/brick1 
                                                          xfs
                                                          defaults       
                                                          0 0</tt><tt><br>
                                                          </tt><tt>/mnt/brick2 
                                                          xfs
                                                          defaults       
                                                          0 0</tt><tt><br>
                                                          </tt><br>
                                                          Will this
                                                          cause a
                                                          problem with
                                                          creating a
                                                          volume under
                                                          /home?<span class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-HOEnZb"><font color="#888888"><br>
                                                          </font></span></div>
                                                        </blockquote>
                                                        <div><br>
                                                        </div>
                                                        <div>I don&#39;t
                                                          think the
                                                          bottleneck is
                                                          disk. You can
                                                          do the same
                                                          tests you did
                                                          on your new
                                                          volume to
                                                          confirm?<br>
                                                        </div>
                                                        <div> </div>
                                                        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF"><span class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-HOEnZb"><font color="#888888"> <br>
                                                          Pat</font></span>
                                                          <div>
                                                          <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-h5"><br>
                                                          <br>
                                                          <br>
                                                          <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372moz-cite-prefix">On
                                                          05/11/2017
                                                          11:32 AM,
                                                          Pranith Kumar
                                                          Karampuri
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr"><br>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Thu, May 11,
                                                          2017 at 8:57
                                                          PM, Pat Haley
                                                          <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <br>
                                                          Hi Pranith,<br>
                                                          <br>
                                                          Unfortunately,
                                                          we don&#39;t have
                                                          similar
                                                          hardware for a
                                                          small scale
                                                          test.  All we
                                                          have is our
                                                          production
                                                          hardware.<span class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372HOEnZb"><font color="#888888"><br>
                                                          </font></span></div>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          <div>You said
                                                          something
                                                          about /home
                                                          partition
                                                          which has
                                                          lesser disks,
                                                          we can create
                                                          plain
                                                          distribute
                                                          volume inside
                                                          one of those
                                                          directories.
                                                          After we are
                                                          done, we can
                                                          remove the
                                                          setup. What do
                                                          you say?<br>
                                                           <br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF"><span class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372HOEnZb"><font color="#888888"> <br>
                                                          Pat</font></span>
                                                          <div>
                                                          <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372h5"><br>
                                                          <br>
                                                          <br>
                                                          <br>
                                                          <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417moz-cite-prefix">On
                                                          05/11/2017
                                                          07:05 AM,
                                                          Pranith Kumar
                                                          Karampuri
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr"><br>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Thu, May 11,
                                                          2017 at 2:48
                                                          AM, Pat Haley
                                                          <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <br>
                                                          Hi Pranith,<br>
                                                          <br>
                                                          Since we are
                                                          mounting the
                                                          partitions as
                                                          the bricks, I
                                                          tried the dd
                                                          test writing
                                                          to
                                                          &lt;brick-path&gt;/.glusterfs/&lt;file-<wbr>to-be-removed-after-test&gt;.
                                                          The results
                                                          without
                                                          oflag=sync
                                                          were 1.6 Gb/s
                                                          (faster than
                                                          gluster but
                                                          not as fast as
                                                          I was
                                                          expecting
                                                          given the 1.2
                                                          Gb/s to the
                                                          no-gluster
                                                          area w/ fewer
                                                          disks).<span class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417HOEnZb"><font color="#888888"><br>
                                                          </font></span></div>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          <div>Okay,
                                                          then 1.6Gb/s
                                                          is what we
                                                          need to target
                                                          for,
                                                          considering
                                                          your volume is
                                                          just
                                                          distribute. Is
                                                          there any way
                                                          you can do
                                                          tests on
                                                          similar
                                                          hardware but
                                                          at a small
                                                          scale? Just so
                                                          we can run the
                                                          workload to
                                                          learn more
                                                          about the
                                                          bottlenecks in
                                                          the system? We
                                                          can probably
                                                          try to get the
                                                          speed to
                                                          1.2Gb/s on
                                                          your /home
                                                          partition you
                                                          were telling
                                                          me yesterday.
                                                          Let me know if
                                                          that is
                                                          something you
                                                          are okay to
                                                          do.<br>
                                                          </div>
                                                          <div> </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF"><span class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417HOEnZb"><font color="#888888"> <br>
                                                          Pat</font></span>
                                                          <div>
                                                          <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417h5"><br>
                                                          <br>
                                                          <br>
                                                          <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055moz-cite-prefix">On
                                                          05/10/2017
                                                          01:27 PM,
                                                          Pranith Kumar
                                                          Karampuri
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr"><br>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Wed, May 10,
                                                          2017 at 10:15
                                                          PM, Pat Haley
                                                          <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <br>
                                                          Hi Pranith,<br>
                                                          <br>
                                                          Not entirely
                                                          sure (this
                                                          isn&#39;t my area
                                                          of
                                                          expertise). 
                                                          I&#39;ll run your
                                                          answer by some
                                                          other people
                                                          who are more
                                                          familiar with
                                                          this.<br>
                                                          <br>
                                                          I am also
                                                          uncertain
                                                          about how to
                                                          interpret the
                                                          results when
                                                          we also add
                                                          the dd tests
                                                          writing to the
                                                          /home area (no
                                                          gluster, still
                                                          on the same
                                                          machine)<br>
                                                          <ul>
                                                          <li>dd test
                                                          without
                                                          oflag=sync
                                                          (rough average
                                                          of multiple
                                                          tests)<br>
                                                          </li>
                                                          <ul>
                                                          <li>gluster w/
                                                          fuse mount :
                                                          570 Mb/s</li>
                                                          <li>gluster w/
                                                          nfs mount: 
                                                          390 Mb/s</li>
                                                          <li>nfs (no
                                                          gluster):  1.2
                                                          Gb/s</li>
                                                          </ul>
                                                          <li>dd test
                                                          with
                                                          oflag=sync
                                                          (rough average
                                                          of multiple
                                                          tests)</li>
                                                          <ul>
                                                          <li>gluster w/
                                                          fuse mount:  5
                                                          Mb/s</li>
                                                          <li>gluster w/
                                                          nfs mount: 
                                                          200 Mb/s</li>
                                                          <li>nfs (no
                                                          gluster): 20
                                                          Mb/s<br>
                                                          </li>
                                                          </ul>
                                                          </ul>
                                                          Given that the
                                                          non-gluster
                                                          area is a
                                                          RAID-6 of 4
                                                          disks while
                                                          each brick of
                                                          the gluster
                                                          area is a
                                                          RAID-6 of 32
                                                          disks, I would
                                                          naively expect
                                                          the writes to
                                                          the gluster
                                                          area to be
                                                          roughly 8x
                                                          faster than to
                                                          the
                                                          non-gluster.<br>
                                                          </div>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          <div>I think a
                                                          better test is
                                                          to try and
                                                          write to a
                                                          file using nfs
                                                          without any
                                                          gluster to a
                                                          location that
                                                          is not inside
                                                          the brick but
                                                          someother
                                                          location that
                                                          is on same
                                                          disk(s). If
                                                          you are
                                                          mounting the
                                                          partition as
                                                          the brick,
                                                          then we can
                                                          write to a
                                                          file inside
                                                          .glusterfs
                                                          directory,
                                                          something like
&lt;brick-path&gt;/.glusterfs/&lt;file-<wbr>to-be-removed-after-test&gt;.
                                                          <br>
                                                          <br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <br>
                                                          I still think
                                                          we have a
                                                          speed issue, I
                                                          can&#39;t tell if
                                                          fuse vs nfs is
                                                          part of the
                                                          problem.</div>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          I got
                                                          interested in
                                                          the post
                                                          because I read
                                                          that fuse
                                                          speed is
                                                          lesser than
                                                          nfs speed
                                                          which is
                                                          counter-intuitive
                                                          to my
                                                          understanding.
                                                          So wanted
                                                          clarifications.
                                                          Now that I got
                                                          my
                                                          clarifications
                                                          where fuse
                                                          outperformed
                                                          nfs without
                                                          sync, we can
                                                          resume testing
                                                          as described
                                                          above and try
                                                          to find what
                                                          it is. Based
                                                          on your
                                                          email-id I am
                                                          guessing you
                                                          are from
                                                          Boston and I
                                                          am from
                                                          Bangalore so
                                                          if you are
                                                          okay with
                                                          doing this
                                                          debugging for
                                                          multiple days
                                                          because of
                                                          timezones, I
                                                          will be happy
                                                          to help.
                                                          Please be a
                                                          bit patient
                                                          with me, I am
                                                          under a
                                                          release crunch
                                                          but I am very
                                                          curious with
                                                          the problem
                                                          you posted.<br>
                                                          <br>
                                                          </div>
                                                          <div class="gmail_quote">
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF"> 
                                                          Was there
                                                          anything
                                                          useful in the
                                                          profiles?<span class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055HOEnZb"><font color="#888888"><br>
                                                          </font></span></div>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          <div>Unfortunately
                                                          profiles
                                                          didn&#39;t help me
                                                          much, I think
                                                          we are
                                                          collecting the
                                                          profiles from
                                                          an active
                                                          volume, so it
                                                          has a lot of
                                                          information
                                                          that is not
                                                          pertaining to
                                                          dd so it is
                                                          difficult to
                                                          find the
                                                          contributions
                                                          of dd. So I
                                                          went through
                                                          your post
                                                          again and
                                                          found
                                                          something I
                                                          didn&#39;t pay
                                                          much attention
                                                          to earlier
                                                          i.e.
                                                          oflag=sync, so
                                                          did my own
                                                          tests on my
                                                          setup with
                                                          FUSE so sent
                                                          that reply.<br>
                                                           <br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF"><span class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055HOEnZb"><font color="#888888"> <br>
                                                          Pat</font></span>
                                                          <div>
                                                          <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055h5"><br>
                                                          <br>
                                                          <br>
                                                          <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930moz-cite-prefix">On
                                                          05/10/2017
                                                          12:15 PM,
                                                          Pranith Kumar
                                                          Karampuri
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>Okay
                                                          good. At least
                                                          this validates
                                                          my doubts.
                                                          Handling
                                                          O_SYNC in
                                                          gluster NFS
                                                          and fuse is a
                                                          bit different.<br>
                                                          </div>
                                                          When
                                                          application
                                                          opens a file
                                                          with O_SYNC on
                                                          fuse mount
                                                          then each
                                                          write syscall
                                                          has to be
                                                          written to
                                                          disk as part
                                                          of the syscall
                                                          where as in
                                                          case of NFS,
                                                          there is no
                                                          concept of
                                                          open. NFS
                                                          performs write
                                                          though a
                                                          handle saying
                                                          it needs to be
                                                          a synchronous
                                                          write, so
                                                          write()
                                                          syscall is
                                                          performed
                                                          first then it
                                                          performs
                                                          fsync(). so an
                                                          write on an fd
                                                          with O_SYNC
                                                          becomes
                                                          write+fsync. I
                                                          am suspecting
                                                          that when
                                                          multiple
                                                          threads do
                                                          this
                                                          write+fsync()
                                                          operation on
                                                          the same file,
                                                          multiple
                                                          writes are
                                                          batched
                                                          together to be
                                                          written do
                                                          disk so the
                                                          throughput on
                                                          the disk is
                                                          increasing is
                                                          my guess.<br>
                                                          <br>
                                                          </div>
                                                          Does it answer
                                                          your doubts?<br>
                                                          </div>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Wed, May 10,
                                                          2017 at 9:35
                                                          PM, Pat Haley
                                                          <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <br>
                                                          Without the
                                                          oflag=sync and
                                                          only a single
                                                          test of each,
                                                          the FUSE is
                                                          going faster
                                                          than NFS:<br>
                                                          <br>
                                                          FUSE:<br>
                                                          <tt>mseas-data2(dri_nascar)%
                                                          dd
                                                          if=/dev/zero
                                                          count=4096
                                                          bs=1048576
                                                          of=zeros.txt
                                                          conv=sync</tt><tt><br>
                                                          </tt><tt>4096+0
                                                          records in</tt><tt><br>
                                                          </tt><tt>4096+0
                                                          records out</tt><tt><br>
                                                          </tt><tt>4294967296
                                                          bytes (4.3 GB)
                                                          copied,
                                                          7.46961 s, 575
                                                          MB/s</tt><tt><br>
                                                          </tt><tt><br>
                                                          <br>
                                                          </tt>NFS<br>
                                                          <tt>mseas-data2(HYCOM)%
                                                          dd
                                                          if=/dev/zero
                                                          count=4096
                                                          bs=1048576
                                                          of=zeros.txt
                                                          conv=sync</tt><tt><br>
                                                          </tt><tt>4096+0
                                                          records in</tt><tt><br>
                                                          </tt><tt>4096+0
                                                          records out</tt><tt><br>
                                                          </tt><tt>4294967296
                                                          bytes (4.3 GB)
                                                          copied,
                                                          11.4264 s, 376
                                                          MB/s</tt>
                                                          <div>
                                                          <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930h5"><tt><br>
                                                          </tt><tt><br>
                                                          </tt><tt><br>
                                                          </tt>
                                                          <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365moz-cite-prefix">On
                                                          05/10/2017
                                                          11:53 AM,
                                                          Pranith Kumar
                                                          Karampuri
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">Could
                                                          you let me
                                                          know the speed
                                                          without
                                                          oflag=sync on
                                                          both the
                                                          mounts? No
                                                          need to
                                                          collect
                                                          profiles.<br>
                                                          </div>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Wed, May 10,
                                                          2017 at 9:17
                                                          PM, Pat Haley
                                                          <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <br>
                                                          Here is what I
                                                          see now:<br>
                                                          <br>
                                                          <tt>[root@mseas-data2
                                                          ~]# gluster
                                                          volume info</tt><span><tt><br>
                                                          </tt><tt> </tt><tt><br>
                                                          </tt><tt>Volume
                                                          Name:
                                                          data-volume</tt><tt><br>
                                                          </tt><tt>Type:
                                                          Distribute</tt><tt><br>
                                                          </tt><tt>Volume
                                                          ID:
                                                          c162161e-2a2d-4dac-b015-f31fd8<wbr>9ceb18</tt><tt><br>
                                                          </tt><tt>Status:
                                                          Started</tt><tt><br>
                                                          </tt><tt>Number
                                                          of Bricks: 2</tt><tt><br>
                                                          </tt><tt>Transport-type:
                                                          tcp</tt><tt><br>
                                                          </tt><tt>Bricks:</tt><tt><br>
                                                          </tt><tt>Brick1:
mseas-data2:/mnt/brick1</tt><tt><br>
                                                          </tt><tt>Brick2:
mseas-data2:/mnt/brick2</tt><tt><br>
                                                          </tt><tt>Options
                                                          Reconfigured:</tt><tt><br>
                                                          </tt></span><tt>diagnostics.count-fop-hits:
                                                          on</tt><tt><br>
                                                          </tt><tt>diagnostics.latency-measuremen<wbr>t:
                                                          on</tt><tt><br>
                                                          </tt><tt>nfs.exports-auth-enable:
                                                          on</tt><tt><br>
                                                          </tt><tt>diagnostics.brick-sys-log-leve<wbr>l:
                                                          WARNING</tt><span><tt><br>
                                                          </tt><tt>performance.readdir-ahead:
                                                          on</tt><tt><br>
                                                          </tt><tt>nfs.disable:
                                                          on</tt><tt><br>
                                                          </tt><tt>nfs.export-volumes:
                                                          off</tt><tt><br>
                                                          </tt><br>
                                                          <br>
                                                          <br>
                                                          </span>
                                                          <div>
                                                          <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365h5">
                                                          <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365m_5079054141158038028moz-cite-prefix">On
                                                          05/10/2017
                                                          11:44 AM,
                                                          Pranith Kumar
                                                          Karampuri
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">
                                                          <div>Is this
                                                          the volume
                                                          info you have?<br>
                                                          <br>
                                                          <pre>&gt;<i>     [<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">root at mseas-data2</a> ~]# gluster volume info
</i>&gt;<i>
</i>&gt;<i>     Volume Name: data-volume
</i>&gt;<i>     Type: Distribute
</i>&gt;<i>     Volume ID: c162161e-2a2d-4dac-b015-f31fd8<wbr>9ceb18
</i>&gt;<i>     Status: Started
</i>&gt;<i>     Number of Bricks: 2
</i>&gt;<i>     Transport-type: tcp
</i>&gt;<i>     Bricks:
</i>&gt;<i>     Brick1: mseas-data2:/mnt/brick1
</i>&gt;<i>     Brick2: mseas-data2:/mnt/brick2
</i>&gt;<i>     Options Reconfigured:
</i>&gt;<i>     performance.readdir-ahead: on
</i>&gt;<i>     nfs.disable: on
</i>&gt;<i>     nfs.export-volumes: off

</i></pre>
                                                          </div>
                                                          ​I copied this
                                                          from old
                                                          thread from
                                                          2016. This is
                                                          distribute
                                                          volume. Did
                                                          you change any
                                                          of the options
                                                          in between? </div>
                                                          </blockquote>
                                                          </div>
                                                          </div>
                                                          <span>
                                                          <pre class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365m_5079054141158038028moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365m_5079054141158038028moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365m_5079054141158038028moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                                                          </span></div>
                                                          </blockquote>
                                                          </div>
                                                          --
                                                          <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365gmail_signature">
                                                          <div dir="ltr">Pranith
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <pre class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930m_7039630128565981365moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          --
                                                          <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930gmail_signature">
                                                          <div dir="ltr">Pranith
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <pre class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055m_-1330903101597946930moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          --
                                                          <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055gmail_signature">
                                                          <div dir="ltr">Pranith
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <pre class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417m_-7935477196994992055moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          --
                                                          <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417gmail_signature">
                                                          <div dir="ltr">Pranith
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <pre class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372m_6626346771705009417moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          --
                                                          <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372gmail_signature">
                                                          <div dir="ltr">Pranith
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <pre class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail-m_-1677399571678699372moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                                                          </div>
                                                          </div>
                                                          </div>
                                                        </blockquote>
                                                      </div>
                                                      --
                                                      <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956gmail_signature">
                                                        <div dir="ltr">Pranith
                                                        </div>
                                                      </div>
                                                    </div>
                                                  </div>
                                                </blockquote>
                                                <pre class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073m_5903432224896103956moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                                              </div>
                                            </div>
                                          </div>
                                        </blockquote>
                                      </div>
                                      <br>
                                      <br clear="all">
                                      <br>
                                      -- <br>
                                      <div class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073gmail_signature" data-smartmail="gmail_signature">
                                        <div dir="ltr">Pranith<br>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                </blockquote>
                                <br>
                                <pre class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_2414117758133198099m_-7360220549516486306m_-3319843143517043073moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                              </div>
                            </div>
                          </div>
                        </blockquote>
                      </div>
                      <br>
                      <br clear="all">
                      <br>
                      -- <br>
                      <div class="m_2414117758133198099m_-7360220549516486306gmail_signature" data-smartmail="gmail_signature">
                        <div dir="ltr">Pranith<br>
                        </div>
                      </div>
                    </div>
                  </blockquote>
                  <br>
                  <pre class="m_2414117758133198099m_-7360220549516486306moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_2414117758133198099m_-7360220549516486306moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_2414117758133198099m_-7360220549516486306moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
        <br clear="all">
        <br>
        -- <br>
        <div class="m_2414117758133198099gmail_signature" data-smartmail="gmail_signature">
          <div dir="ltr">Pranith<br>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
    <pre class="m_2414117758133198099moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_2414117758133198099moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_2414117758133198099moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
  </div></div></div>

</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div>