<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 13, 2018 at 5:00 AM, Pat Haley <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div text="#000000" bgcolor="#FFFFFF">
    <br>
    Hi Raghavendra,<br>
    <br>
    We were wondering if you have had a chance to look at this again,
    and if so, did you have any further suggestions?<br></div></blockquote><div><br></div><div>Sorry Pat. Too much of work :). I&#39;ll be working on a patch today to make sure read-ahead doesn&#39;t flush its cache due to fstats making this behavior optional. You can try this patch and let us know about results.</div><div>Will let you know when patch is ready.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
    <br>
    Thanks<span class="HOEnZb"><font color="#888888"><br>
    <br>
    Pat</font></span><div><div class="h5"><br>
    <br>
    <br>
    <div class="m_-6455985595820935451moz-cite-prefix">On 07/06/2018 01:27 AM, Raghavendra
      Gowdappa wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr"><br>
        <div class="gmail_extra"><br>
          <div class="gmail_quote">On Fri, Jul 6, 2018 at 5:29 AM, Pat
            Haley <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div text="#000000" bgcolor="#FFFFFF"> <br>
                Hi Raghavendra,<br>
                <br>
                Our technician may have some time to look at this issue
                tomorrow.  Are there any tests that you&#39;d like to see?<br>
              </div>
            </blockquote>
            <div><br>
            </div>
            <div>Sorry. I&#39;ve been busy with other things and was away
              from work for couple of days. It&#39;ll take me another 2 days
              to work on this issue again. So, most likely you&#39;ll have
              an update on this next week.<br>
            </div>
            <div> <br>
            </div>
            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div text="#000000" bgcolor="#FFFFFF"> <br>
                Thanks<span class="m_-6455985595820935451HOEnZb"><font color="#888888"><br>
                    <br>
                    Pat</font></span>
                <div>
                  <div class="m_-6455985595820935451h5"><br>
                    <br>
                    <br>
                    <div class="m_-6455985595820935451m_-1517267327311572646moz-cite-prefix">On
                      06/29/2018 11:25 PM, Raghavendra Gowdappa wrote:<br>
                    </div>
                    <blockquote type="cite">
                      <div dir="ltr"><br>
                        <div class="gmail_extra"><br>
                          <div class="gmail_quote">On Fri, Jun 29, 2018
                            at 10:38 PM, Pat Haley <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                            wrote:<br>
                            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                              <div text="#000000" bgcolor="#FFFFFF"> <br>
                                Hi Raghavendra,<br>
                                <br>
                                We ran the tests (write tests) and I
                                copied the log files for both the server
                                and the client to <a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032moz-txt-link-freetext" href="http://mseas.mit.edu/download/phaley/GlusterUsers/2018/Jun29/" target="_blank">http://mseas.mit.edu/download/<wbr>phaley/GlusterUsers/2018/Jun29<wbr>/</a>
                                .  Is there any additional trace
                                information you need?  (If so, where
                                should I look for it?)<br>
                              </div>
                            </blockquote>
                            <div><br>
                            </div>
                            <div>Nothing for now. I can see from logs
                              that workaround is not helping. fstat
                              requests are not absorbed by md-cache and
                              read-ahead is witnessing them and flushing
                              its read-ahead cache. I am investigating
                              more on md-cache (It also seems to be
                              invalidating inodes quite frequently which
                              actually might be the root cause of seeing
                              so many fstat requests from kernel). Will
                              post when I find anything relevant.</div>
                            <div><br>
                            </div>
                            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                              <div text="#000000" bgcolor="#FFFFFF"> <br>
                                Also the volume information you
                                requested<br>
                                <br>
                                [root@mseas-data2 ~]# gluster volume
                                info data-volume<br>
                                 <br>
                                Volume Name: data-volume<br>
                                Type: Distribute<br>
                                Volume ID:
                                c162161e-2a2d-4dac-b015-f31fd8<wbr>9ceb18<br>
                                Status: Started<br>
                                Number of Bricks: 2<br>
                                Transport-type: tcp<br>
                                Bricks:<br>
                                Brick1: mseas-data2:/mnt/brick1<br>
                                Brick2: mseas-data2:/mnt/brick2<br>
                                Options Reconfigured:<br>
                                diagnostics.client-log-level: TRACE<br>
                                network.inode-lru-limit: 50000<br>
                                performance.md-cache-timeout: 60<br>
                                performance.open-behind: off<br>
                                disperse.eager-lock: off<br>
                                auth.allow: *<br>
                                server.allow-insecure: on<br>
                                nfs.exports-auth-enable: on<br>
                                diagnostics.brick-sys-log-leve<wbr>l:
                                WARNING<br>
                                performance.readdir-ahead: on<br>
                                nfs.disable: on<br>
                                nfs.export-volumes: off<br>
                                [root@mseas-data2 ~]# <br>
                                <div>
                                  <div class="m_-6455985595820935451m_-1517267327311572646h5">
                                    <br>
                                    <br>
                                    <div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032moz-cite-prefix">On
                                      06/29/2018 12:28 PM, Raghavendra
                                      Gowdappa wrote:<br>
                                    </div>
                                    <blockquote type="cite">
                                      <div dir="ltr"><br>
                                        <div class="gmail_extra"><br>
                                          <div class="gmail_quote">On
                                            Fri, Jun 29, 2018 at 8:24
                                            PM, Pat Haley <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                                            wrote:<br>
                                            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                              <div text="#000000" bgcolor="#FFFFFF"> <br>
                                                Hi Raghavendra,<br>
                                                <br>
                                                Our technician was able
                                                to try the manual
                                                setting today.  He found
                                                that our upper limit
                                                for <span>performance.md-cache-timeo<wbr>ut
                                                  was 60 not 600, so he
                                                  used that value, along
                                                  with the </span><span>network.inode-lru-limit=50000.<br>
                                                  <br>
                                                  The result was another
                                                  small (~1%) increase
                                                  in speed.  Does this
                                                  suggest some addition
                                                  tests/changes we could
                                                  try?<br>
                                                </span></div>
                                            </blockquote>
                                            <div><br>
                                            </div>
                                            <div>Can you set gluster
                                              option
                                              diagnostics.client-log-level
                                              to TRACE  and run
                                              sequential read tests
                                              again (with
                                              md-cache-timeout value of
                                              60)? <br>
                                            </div>
                                            <div><br>
                                            </div>
                                            <div>#gluster volume set
                                              &lt;volname&gt;
                                              diagnostics.client-log-level
                                              TRACE</div>
                                            <div><br>
                                            </div>
                                            <div>Also are you sure that
                                              open-behind was turned
                                              off? Can you give the
                                              output of,</div>
                                            <div><br>
                                            </div>
                                            <div># gluster volume info
                                              &lt;volname&gt;</div>
                                            <div><br>
                                            </div>
                                            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                              <div text="#000000" bgcolor="#FFFFFF"><span>
                                                  <br>
                                                  Thanks<span class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032HOEnZb"><font color="#888888"><br>
                                                      <br>
                                                      Pat<br>
                                                      <br>
                                                      <br>
                                                    </font></span></span>
                                                <div>
                                                  <div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032h5"><br>
                                                    <br>
                                                    <div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989moz-cite-prefix">On
                                                      06/25/2018 09:39
                                                      PM, Raghavendra
                                                      Gowdappa wrote:<br>
                                                    </div>
                                                    <blockquote type="cite">
                                                      <div dir="ltr"><br>
                                                        <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Tue, Jun 26,
                                                          2018 at 3:21
                                                          AM, Pat Haley
                                                          <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <br>
                                                          Hi
                                                          Raghavendra,<br>
                                                          <br>
                                                          Setting the
                                                          performance.write-behind
                                                          off had a
                                                          small
                                                          improvement on
                                                          the write
                                                          speed (~3%),<br>
                                                          <br>
                                                          We were unable
                                                          to turn on
                                                          &quot;group
                                                          metadata-cache&quot;. 
                                                          When we try
                                                          get errors
                                                          like<br>
                                                          <br>
                                                          # gluster
                                                          volume set
                                                          data-volume
                                                          group
                                                          metadata-cache<br>
&#39;/var/lib/glusterd/groups/meta<wbr>data-cache&#39; file format not valid.<br>
                                                          <br>
                                                          Was
                                                          metadata-cache
                                                          available for
                                                          gluster
                                                          3.7.11? We ask
                                                          because the
                                                          release notes
                                                          for 3.11
                                                          mentions
                                                          “Feature for
                                                          metadata-caching/small
                                                          file
                                                          performance is
                                                          production
                                                          ready.” (<a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-freetext" href="https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/" target="_blank">https://gluster.readthedocs.i<wbr>o/en/latest/release-notes/3.11<wbr>.0/</a>).<br>
                                                          <br>
                                                          Do any of
                                                          these results
                                                          suggest
                                                          anything?  If
                                                          not, what
                                                          further tests
                                                          would be
                                                          useful?<br>
                                                          </div>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          <div>Group
                                                          metadata-cache
                                                          is just a
                                                          bunch of
                                                          options one
                                                          sets on a
                                                          volume. So,
                                                          You can set
                                                          them manually
                                                          using gluster
                                                          cli. Following
                                                          are the
                                                          options and
                                                          their values:</div>
                                                          <div><br>
                                                          </div>
                                                          <div>performance.md-cache-timeout=6<wbr>00<br>
network.inode-lru-limit=50000<br>
                                                          <br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <br>
                                                          Thanks<span class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-HOEnZb"><font color="#888888"><br>
                                                          <br>
                                                          Pat</font></span>
                                                          <div>
                                                          <div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-h5"><br>
                                                          <br>
                                                           <br>
                                                          <br>
                                                          <div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-cite-prefix">On
                                                          06/22/2018
                                                          07:51 AM,
                                                          Raghavendra
                                                          Gowdappa
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr"><br>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Thu, Jun 21,
                                                          2018 at 8:41
                                                          PM, Pat Haley
                                                          <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <br>
                                                          Hi
                                                          Raghavendra,<br>
                                                          <br>
                                                          Thanks for the
                                                          suggestions. 
                                                          Our technician
                                                          will be in on
                                                          Monday.  We&#39;ll
                                                          test then and
                                                          let you know
                                                          the results.<br>
                                                          <br>
                                                          One question I
                                                          have, is the
                                                          &quot;group
                                                          metadata-cache&quot;
                                                          option
                                                          supposed to
                                                          directly
                                                          impact the
                                                          performance or
                                                          is it to help
                                                          collect data? 
                                                          If the latter,
                                                          where will the
                                                          data be
                                                          located?<br>
                                                          </div>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          <div>It
                                                          impacts
                                                          performance.</div>
                                                          <div> <br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <br>
                                                          Thanks again.<span class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266HOEnZb"><font color="#888888"><br>
                                                          <br>
                                                          Pat</font></span>
                                                          <div>
                                                          <div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266h5"><br>
                                                          <br>
                                                          <br>
                                                          <div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-cite-prefix">On
                                                          06/21/2018
                                                          01:01 AM,
                                                          Raghavendra
                                                          Gowdappa
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr"><br>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Thu, Jun 21,
                                                          2018 at 10:24
                                                          AM,
                                                          Raghavendra
                                                          Gowdappa <span dir="ltr">&lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>
                                                          <div>
                                                          <div>
                                                          <div>
                                                          <div>
                                                          <div>For the
                                                          case of writes
                                                          to glusterfs
                                                          mount,<br>
                                                          <br>
                                                          </div>
                                                          I saw in
                                                          earlier
                                                          conversations
                                                          that there are
                                                          too many
                                                          lookups, but
                                                          small number
                                                          of writes.
                                                          Since writes
                                                          cached in
                                                          write-behind
                                                          would
                                                          invalidate
                                                          metadata
                                                          cache, lookups
                                                          won&#39;t be
                                                          absorbed by
                                                          md-cache. I am
                                                          wondering what
                                                          would results
                                                          look like if
                                                          we turn off
                                                          performance.write-behind.<br>
                                                          <br>
                                                          </div>
                                                          @Pat,<br>
                                                          <br>
                                                          </div>
                                                          Can you set,<br>
                                                          <br>
                                                          </div>
                                                          # gluster
                                                          volume set
                                                          &lt;volname&gt;
performance.write-behind off<br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          <div>Please
                                                          turn on &quot;group
metadata-cache&quot; for write tests too.</div>
                                                          <div> <br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>
                                                          <div><br>
                                                          </div>
                                                          and redo the
                                                          tests writing
                                                          to glusterfs
                                                          mount? Let us
                                                          know about the
                                                          results you
                                                          see.<br>
                                                          <br>
                                                          </div>
                                                          regards,<br>
                                                          </div>
                                                          Raghavendra<br>
                                                          </div>
                                                          <div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663HOEnZb">
                                                          <div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663h5">
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Thu, Jun 21,
                                                          2018 at 8:33
                                                          AM,
                                                          Raghavendra
                                                          Gowdappa <span dir="ltr">&lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr"><br>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote"><span>On
                                                          Thu, Jun 21,
                                                          2018 at 8:32
                                                          AM,
                                                          Raghavendra
                                                          Gowdappa <span dir="ltr">&lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>For the
                                                          case of
                                                          reading from
                                                          Glusterfs
                                                          mount,
                                                          read-ahead
                                                          should help.
                                                          However, we&#39;ve
                                                          known issues
                                                          with
                                                          read-ahead[1][2].
                                                          To work around
                                                          these, can you
                                                          try with,<br>
                                                          <br>
                                                          </div>
                                                          1. Turn off
                                                          performance.open-behind<br>
                                                          </div>
                                                          #gluster
                                                          volume set
                                                          &lt;volname&gt;
performance.open-behind off<br>
                                                          <br>
                                                          <div>2. enable
                                                          group meta
                                                          metadata-cache<br>
                                                          # gluster
                                                          volume set
                                                          &lt;volname&gt;
                                                          group
                                                          metadata-cache</div>
                                                          </div>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          </span>
                                                          <div>[1]  <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1084508" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1084508</a></div>
                                                          <div>[2] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1214489" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1214489</a></div>
                                                          <span>
                                                          <div><br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>
                                                          <div class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663m_-7085830777103976589m_-1104802400275053683gmail-m_5499637752699686457h5"><br>
                                                          <br>
                                                          <div>
                                                          <div>
                                                          <div>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Thu, Jun 21,
                                                          2018 at 5:00
                                                          AM, Pat Haley
                                                          <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
                                                          Hi,<br>
                                                          <br>
                                                          We were
                                                          recently
                                                          revisiting our
                                                          problems with
                                                          the slowness
                                                          of gluster
                                                          writes (<a href="http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html" rel="noreferrer" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2017-April<wbr>/030529.html</a>).
                                                          Specifically
                                                          we were
                                                          testing the
                                                          suggestions in
                                                          a recent post
                                                          (<a href="http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html" rel="noreferrer" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2018-March<wbr>/033699.html</a>).
                                                          The first two
                                                          suggestions
                                                          (specifying a
negative-timeout in the mount settings or adding
rpc-auth-allow-insecure to glusterd.vol) did not improve our
                                                          performance,
                                                          while setting
&quot;disperse.eager-lock off&quot; provided a tiny (5%) speed-up.<br>
                                                          <br>
                                                          Some of the
                                                          various tests
                                                          we have tried
                                                          earlier can be
                                                          seen in the
                                                          links below. 
                                                          Do any of the
                                                          above
                                                          observations
                                                          suggest what
                                                          we could try
                                                          next to either
                                                          improve the
                                                          speed or debug
                                                          the issue? 
                                                          Thanks<br>
                                                          <br>
                                                          <a href="http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-June/0<wbr>31565.html</a><br>
                                                          <a href="http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-May/03<wbr>0937.html</a><br>
                                                          <br>
                                                          Pat<br>
                                                          <br>
                                                          -- <br>
                                                          <br>
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-<br>
                                                          Pat Haley     
                                                                       
                                                                Email: 
                                                          <a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a><br>
                                                          Center for
                                                          Ocean
                                                          Engineering   
                                                             Phone: 
                                                          (617) 253-6824<br>
                                                          Dept. of
                                                          Mechanical
                                                          Engineering   
                                                          Fax:    (617)
                                                          253-8125<br>
                                                          MIT, Room
                                                          5-213         
                                                                    <a href="http://web.mit.edu/phaley/www/" rel="noreferrer" target="_blank">http://web.mit.edu/phaley/www/</a><br>
                                                          77
                                                          Massachusetts
                                                          Avenue<br>
                                                          Cambridge, MA 
                                                          02139-4301<br>
                                                          <br>
______________________________<wbr>_________________<br>
                                                          Gluster-users
                                                          mailing list<br>
                                                          <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
                                                          <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a></blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </span></div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          <pre class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
______________________________<wbr>_________________<br>
                                                          Gluster-users
                                                          mailing list<br>
                                                          <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
                                                          <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          <pre class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                        </div>
                                                      </div>
                                                    </blockquote>
                                                    <br>
                                                    <pre class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032m_-2192927334915852989moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                                                  </div>
                                                </div>
                                              </div>
                                            </blockquote>
                                          </div>
                                          <br>
                                        </div>
                                      </div>
                                    </blockquote>
                                    <br>
                                    <pre class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_-6455985595820935451m_-1517267327311572646m_9072513899500000032moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                                  </div>
                                </div>
                              </div>
                            </blockquote>
                          </div>
                          <br>
                        </div>
                      </div>
                    </blockquote>
                    <br>
                    <pre class="m_-6455985595820935451m_-1517267327311572646moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_-6455985595820935451m_-1517267327311572646moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_-6455985595820935451m_-1517267327311572646moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                  </div>
                </div>
              </div>
            </blockquote>
          </div>
          <br>
        </div>
      </div>
    </blockquote>
    <br>
    <pre class="m_-6455985595820935451moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_-6455985595820935451moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_-6455985595820935451moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
  </div></div></div>

</blockquote></div><br></div></div>