<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 6, 2018 at 5:29 AM, Pat Haley <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div text="#000000" bgcolor="#FFFFFF">
    <br>
    Hi Raghavendra,<br>
    <br>
    Our technician may have some time to look at this issue tomorrow. 
    Are there any tests that you&#39;d like to see?<br></div></blockquote><div><br></div><div>Sorry. I&#39;ve been busy with other things and was away from work for couple of days. It&#39;ll take me another 2 days to work on this issue again. So, most likely you&#39;ll have an update on this next week.<br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
    <br>
    Thanks<span class="HOEnZb"><font color="#888888"><br>
    <br>
    Pat</font></span><div><div class="h5"><br>
    <br>
    <br>
    <div class="m_-1517267327311572646moz-cite-prefix">On 06/29/2018 11:25 PM, Raghavendra
      Gowdappa wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr"><br>
        <div class="gmail_extra"><br>
          <div class="gmail_quote">On Fri, Jun 29, 2018 at 10:38 PM, Pat
            Haley <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div text="#000000" bgcolor="#FFFFFF"> <br>
                Hi Raghavendra,<br>
                <br>
                We ran the tests (write tests) and I copied the log
                files for both the server and the client to <a class="m_-1517267327311572646m_9072513899500000032moz-txt-link-freetext" href="http://mseas.mit.edu/download/phaley/GlusterUsers/2018/Jun29/" target="_blank">http://mseas.mit.edu/download/<wbr>phaley/GlusterUsers/2018/Jun29<wbr>/</a>
                .  Is there any additional trace information you need? 
                (If so, where should I look for it?)<br>
              </div>
            </blockquote>
            <div><br>
            </div>
            <div>Nothing for now. I can see from logs that workaround is
              not helping. fstat requests are not absorbed by md-cache
              and read-ahead is witnessing them and flushing its
              read-ahead cache. I am investigating more on md-cache (It
              also seems to be invalidating inodes quite frequently
              which actually might be the root cause of seeing so many
              fstat requests from kernel). Will post when I find
              anything relevant.</div>
            <div><br>
            </div>
            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div text="#000000" bgcolor="#FFFFFF"> <br>
                Also the volume information you requested<br>
                <br>
                [root@mseas-data2 ~]# gluster volume info data-volume<br>
                 <br>
                Volume Name: data-volume<br>
                Type: Distribute<br>
                Volume ID: c162161e-2a2d-4dac-b015-f31fd8<wbr>9ceb18<br>
                Status: Started<br>
                Number of Bricks: 2<br>
                Transport-type: tcp<br>
                Bricks:<br>
                Brick1: mseas-data2:/mnt/brick1<br>
                Brick2: mseas-data2:/mnt/brick2<br>
                Options Reconfigured:<br>
                diagnostics.client-log-level: TRACE<br>
                network.inode-lru-limit: 50000<br>
                performance.md-cache-timeout: 60<br>
                performance.open-behind: off<br>
                disperse.eager-lock: off<br>
                auth.allow: *<br>
                server.allow-insecure: on<br>
                nfs.exports-auth-enable: on<br>
                diagnostics.brick-sys-log-leve<wbr>l: WARNING<br>
                performance.readdir-ahead: on<br>
                nfs.disable: on<br>
                nfs.export-volumes: off<br>
                [root@mseas-data2 ~]# <br>
                <div>
                  <div class="m_-1517267327311572646h5"> <br>
                    <br>
                    <div class="m_-1517267327311572646m_9072513899500000032moz-cite-prefix">On
                      06/29/2018 12:28 PM, Raghavendra Gowdappa wrote:<br>
                    </div>
                    <blockquote type="cite">
                      <div dir="ltr"><br>
                        <div class="gmail_extra"><br>
                          <div class="gmail_quote">On Fri, Jun 29, 2018
                            at 8:24 PM, Pat Haley <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                            wrote:<br>
                            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                              <div text="#000000" bgcolor="#FFFFFF"> <br>
                                Hi Raghavendra,<br>
                                <br>
                                Our technician was able to try the
                                manual setting today.  He found that our
                                upper limit for <span>performance.md-cache-timeo<wbr>ut
                                  was 60 not 600, so he used that value,
                                  along with the </span><span>network.inode-lru-limit=50000.<br>
                                  <br>
                                  The result was another small (~1%)
                                  increase in speed.  Does this suggest
                                  some addition tests/changes we could
                                  try?<br>
                                </span></div>
                            </blockquote>
                            <div><br>
                            </div>
                            <div>Can you set gluster option
                              diagnostics.client-log-level to TRACE  and
                              run sequential read tests again (with
                              md-cache-timeout value of 60)? <br>
                            </div>
                            <div><br>
                            </div>
                            <div>#gluster volume set &lt;volname&gt;
                              diagnostics.client-log-level TRACE</div>
                            <div><br>
                            </div>
                            <div>Also are you sure that open-behind was
                              turned off? Can you give the output of,</div>
                            <div><br>
                            </div>
                            <div># gluster volume info &lt;volname&gt;</div>
                            <div><br>
                            </div>
                            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                              <div text="#000000" bgcolor="#FFFFFF"><span>
                                  <br>
                                  Thanks<span class="m_-1517267327311572646m_9072513899500000032HOEnZb"><font color="#888888"><br>
                                      <br>
                                      Pat<br>
                                      <br>
                                      <br>
                                    </font></span></span>
                                <div>
                                  <div class="m_-1517267327311572646m_9072513899500000032h5"><br>
                                    <br>
                                    <div class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989moz-cite-prefix">On
                                      06/25/2018 09:39 PM, Raghavendra
                                      Gowdappa wrote:<br>
                                    </div>
                                    <blockquote type="cite">
                                      <div dir="ltr"><br>
                                        <div class="gmail_extra"><br>
                                          <div class="gmail_quote">On
                                            Tue, Jun 26, 2018 at 3:21
                                            AM, Pat Haley <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                                            wrote:<br>
                                            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                              <div bgcolor="#FFFFFF"> <br>
                                                Hi Raghavendra,<br>
                                                <br>
                                                Setting the
                                                performance.write-behind
                                                off had a small
                                                improvement on the write
                                                speed (~3%),<br>
                                                <br>
                                                We were unable to turn
                                                on &quot;group
                                                metadata-cache&quot;.  When
                                                we try get errors like<br>
                                                <br>
                                                # gluster volume set
                                                data-volume group
                                                metadata-cache<br>
&#39;/var/lib/glusterd/groups/meta<wbr>data-cache&#39; file format not valid.<br>
                                                <br>
                                                Was metadata-cache
                                                available for gluster
                                                3.7.11? We ask because
                                                the release notes for
                                                3.11 mentions “Feature
                                                for
                                                metadata-caching/small
                                                file performance is
                                                production ready.” (<a class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-freetext" href="https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/" target="_blank">https://gluster.readthedocs.i<wbr>o/en/latest/release-notes/3.11<wbr>.0/</a>).<br>
                                                <br>
                                                Do any of these results
                                                suggest anything?  If
                                                not, what further tests
                                                would be useful?<br>
                                              </div>
                                            </blockquote>
                                            <div><br>
                                            </div>
                                            <div>Group metadata-cache is
                                              just a bunch of options
                                              one sets on a volume. So,
                                              You can set them manually
                                              using gluster cli.
                                              Following are the options
                                              and their values:</div>
                                            <div><br>
                                            </div>
                                            <div>performance.md-cache-timeout=6<wbr>00<br>
network.inode-lru-limit=50000<br>
                                              <br>
                                            </div>
                                            <div><br>
                                            </div>
                                            <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                              <div bgcolor="#FFFFFF"> <br>
                                                Thanks<span class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-HOEnZb"><font color="#888888"><br>
                                                    <br>
                                                    Pat</font></span>
                                                <div>
                                                  <div class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-h5"><br>
                                                    <br>
                                                     <br>
                                                    <br>
                                                    <div class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-cite-prefix">On
                                                      06/22/2018 07:51
                                                      AM, Raghavendra
                                                      Gowdappa wrote:<br>
                                                    </div>
                                                    <blockquote type="cite">
                                                      <div dir="ltr"><br>
                                                        <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Thu, Jun 21,
                                                          2018 at 8:41
                                                          PM, Pat Haley
                                                          <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <br>
                                                          Hi
                                                          Raghavendra,<br>
                                                          <br>
                                                          Thanks for the
                                                          suggestions. 
                                                          Our technician
                                                          will be in on
                                                          Monday.  We&#39;ll
                                                          test then and
                                                          let you know
                                                          the results.<br>
                                                          <br>
                                                          One question I
                                                          have, is the
                                                          &quot;group
                                                          metadata-cache&quot;
                                                          option
                                                          supposed to
                                                          directly
                                                          impact the
                                                          performance or
                                                          is it to help
                                                          collect data? 
                                                          If the latter,
                                                          where will the
                                                          data be
                                                          located?<br>
                                                          </div>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          <div>It
                                                          impacts
                                                          performance.</div>
                                                          <div> <br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div bgcolor="#FFFFFF">
                                                          <br>
                                                          Thanks again.<span class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266HOEnZb"><font color="#888888"><br>
                                                          <br>
                                                          Pat</font></span>
                                                          <div>
                                                          <div class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266h5"><br>
                                                          <br>
                                                          <br>
                                                          <div class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-cite-prefix">On
                                                          06/21/2018
                                                          01:01 AM,
                                                          Raghavendra
                                                          Gowdappa
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr"><br>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Thu, Jun 21,
                                                          2018 at 10:24
                                                          AM,
                                                          Raghavendra
                                                          Gowdappa <span dir="ltr">&lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>
                                                          <div>
                                                          <div>
                                                          <div>
                                                          <div>
                                                          <div>For the
                                                          case of writes
                                                          to glusterfs
                                                          mount,<br>
                                                          <br>
                                                          </div>
                                                          I saw in
                                                          earlier
                                                          conversations
                                                          that there are
                                                          too many
                                                          lookups, but
                                                          small number
                                                          of writes.
                                                          Since writes
                                                          cached in
                                                          write-behind
                                                          would
                                                          invalidate
                                                          metadata
                                                          cache, lookups
                                                          won&#39;t be
                                                          absorbed by
                                                          md-cache. I am
                                                          wondering what
                                                          would results
                                                          look like if
                                                          we turn off
                                                          performance.write-behind.<br>
                                                          <br>
                                                          </div>
                                                          @Pat,<br>
                                                          <br>
                                                          </div>
                                                          Can you set,<br>
                                                          <br>
                                                          </div>
                                                          # gluster
                                                          volume set
                                                          &lt;volname&gt;
performance.write-behind off<br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          <div>Please
                                                          turn on &quot;group
metadata-cache&quot; for write tests too.</div>
                                                          <div> <br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>
                                                          <div><br>
                                                          </div>
                                                          and redo the
                                                          tests writing
                                                          to glusterfs
                                                          mount? Let us
                                                          know about the
                                                          results you
                                                          see.<br>
                                                          <br>
                                                          </div>
                                                          regards,<br>
                                                          </div>
                                                          Raghavendra<br>
                                                          </div>
                                                          <div class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663HOEnZb">
                                                          <div class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663h5">
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Thu, Jun 21,
                                                          2018 at 8:33
                                                          AM,
                                                          Raghavendra
                                                          Gowdappa <span dir="ltr">&lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr"><br>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote"><span>On
                                                          Thu, Jun 21,
                                                          2018 at 8:32
                                                          AM,
                                                          Raghavendra
                                                          Gowdappa <span dir="ltr">&lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>For the
                                                          case of
                                                          reading from
                                                          Glusterfs
                                                          mount,
                                                          read-ahead
                                                          should help.
                                                          However, we&#39;ve
                                                          known issues
                                                          with
                                                          read-ahead[1][2].
                                                          To work around
                                                          these, can you
                                                          try with,<br>
                                                          <br>
                                                          </div>
                                                          1. Turn off
                                                          performance.open-behind<br>
                                                          </div>
                                                          #gluster
                                                          volume set
                                                          &lt;volname&gt;
performance.open-behind off<br>
                                                          <br>
                                                          <div>2. enable
                                                          group meta
                                                          metadata-cache<br>
                                                          # gluster
                                                          volume set
                                                          &lt;volname&gt;
                                                          group
                                                          metadata-cache</div>
                                                          </div>
                                                          </blockquote>
                                                          <div><br>
                                                          </div>
                                                          </span>
                                                          <div>[1]  <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1084508" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1084508</a></div>
                                                          <div>[2] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1214489" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1214489</a></div>
                                                          <span>
                                                          <div><br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>
                                                          <div class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663m_-7085830777103976589m_-1104802400275053683gmail-m_5499637752699686457h5"><br>
                                                          <br>
                                                          <div>
                                                          <div>
                                                          <div>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Thu, Jun 21,
                                                          2018 at 5:00
                                                          AM, Pat Haley
                                                          <span dir="ltr">&lt;<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
                                                          Hi,<br>
                                                          <br>
                                                          We were
                                                          recently
                                                          revisiting our
                                                          problems with
                                                          the slowness
                                                          of gluster
                                                          writes (<a href="http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html" rel="noreferrer" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2017-April<wbr>/030529.html</a>).
                                                          Specifically
                                                          we were
                                                          testing the
                                                          suggestions in
                                                          a recent post
                                                          (<a href="http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html" rel="noreferrer" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2018-March<wbr>/033699.html</a>).
                                                          The first two
                                                          suggestions
                                                          (specifying a
negative-timeout in the mount settings or adding
rpc-auth-allow-insecure to glusterd.vol) did not improve our
                                                          performance,
                                                          while setting
&quot;disperse.eager-lock off&quot; provided a tiny (5%) speed-up.<br>
                                                          <br>
                                                          Some of the
                                                          various tests
                                                          we have tried
                                                          earlier can be
                                                          seen in the
                                                          links below. 
                                                          Do any of the
                                                          above
                                                          observations
                                                          suggest what
                                                          we could try
                                                          next to either
                                                          improve the
                                                          speed or debug
                                                          the issue? 
                                                          Thanks<br>
                                                          <br>
                                                          <a href="http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-June/0<wbr>31565.html</a><br>
                                                          <a href="http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2017-May/03<wbr>0937.html</a><br>
                                                          <br>
                                                          Pat<br>
                                                          <br>
                                                          -- <br>
                                                          <br>
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-<br>
                                                          Pat Haley     
                                                                       
                                                                Email: 
                                                          <a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a><br>
                                                          Center for
                                                          Ocean
                                                          Engineering   
                                                             Phone: 
                                                          (617) 253-6824<br>
                                                          Dept. of
                                                          Mechanical
                                                          Engineering   
                                                          Fax:    (617)
                                                          253-8125<br>
                                                          MIT, Room
                                                          5-213         
                                                                    <a href="http://web.mit.edu/phaley/www/" rel="noreferrer" target="_blank">http://web.mit.edu/phaley/www/</a><br>
                                                          77
                                                          Massachusetts
                                                          Avenue<br>
                                                          Cambridge, MA 
                                                          02139-4301<br>
                                                          <br>
______________________________<wbr>_________________<br>
                                                          Gluster-users
                                                          mailing list<br>
                                                          <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
                                                          <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a></blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </span></div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          <pre class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266m_-4085817742671750663moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
______________________________<wbr>_________________<br>
                                                          Gluster-users
                                                          mailing list<br>
                                                          <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
                                                          <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                        </div>
                                                      </div>
                                                    </blockquote>
                                                    <br>
                                                    <pre class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989gmail-m_-5807630702261925266moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                                                  </div>
                                                </div>
                                              </div>
                                            </blockquote>
                                          </div>
                                          <br>
                                        </div>
                                      </div>
                                    </blockquote>
                                    <br>
                                    <pre class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_-1517267327311572646m_9072513899500000032m_-2192927334915852989moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                                  </div>
                                </div>
                              </div>
                            </blockquote>
                          </div>
                          <br>
                        </div>
                      </div>
                    </blockquote>
                    <br>
                    <pre class="m_-1517267327311572646m_9072513899500000032moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_-1517267327311572646m_9072513899500000032moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_-1517267327311572646m_9072513899500000032moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
                  </div>
                </div>
              </div>
            </blockquote>
          </div>
          <br>
        </div>
      </div>
    </blockquote>
    <br>
    <pre class="m_-1517267327311572646moz-signature" cols="72">-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley                          Email:  <a class="m_-1517267327311572646moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    <a class="m_-1517267327311572646moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA  02139-4301
</pre>
  </div></div></div>

</blockquote></div><br></div></div>