[Gluster-users] Slow write times to gluster disk

Pat Haley phaley at mit.edu
Mon Jun 12 18:35:41 UTC 2017


Hi Guys,

I was wondering what our next steps should be to solve the slow write times.

Recently I was debugging a large code and writing a lot of output at 
every time step.  When I tried writing to our gluster disks, it was 
taking over a day to do a single time step whereas if I had the same 
program (same hardware, network) write to our nfs disk the time per 
time-step was about 45 minutes. What we are shooting for here would be 
to have similar times to either gluster of nfs.

Thanks

Pat


On 06/02/2017 01:07 AM, Ben Turner wrote:
> Are you sure using conv=sync is what you want?  I normally use conv=fdatasync, I'll look up the difference between the two and see if it affects your test.
>
>
> -b
>
> ----- Original Message -----
>> From: "Pat Haley" <phaley at mit.edu>
>> To: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
>> Cc: "Ravishankar N" <ravishankar at redhat.com>, gluster-users at gluster.org, "Steve Postma" <SPostma at ztechnet.com>, "Ben
>> Turner" <bturner at redhat.com>
>> Sent: Tuesday, May 30, 2017 9:40:34 PM
>> Subject: Re: [Gluster-users] Slow write times to gluster disk
>>
>>
>> Hi Pranith,
>>
>> The "dd" command was:
>>
>>       dd if=/dev/zero count=4096 bs=1048576 of=zeros.txt conv=sync
>>
>> There were 2 instances where dd reported 22 seconds. The output from the
>> dd tests are in
>>
>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/dd_testvol_gluster.txt
>>
>> Pat
>>
>> On 05/30/2017 09:27 PM, Pranith Kumar Karampuri wrote:
>>> Pat,
>>>         What is the command you used? As per the following output, it
>>> seems like at least one write operation took 16 seconds. Which is
>>> really bad.
>>>        96.39    1165.10 us      89.00 us*16487014.00 us*          393212
>>>        WRITE
>>>
>>>
>>> On Tue, May 30, 2017 at 10:36 PM, Pat Haley <phaley at mit.edu
>>> <mailto:phaley at mit.edu>> wrote:
>>>
>>>
>>>      Hi Pranith,
>>>
>>>      I ran the same 'dd' test both in the gluster test volume and in
>>>      the .glusterfs directory of each brick.  The median results (12 dd
>>>      trials in each test) are similar to before
>>>
>>>        * gluster test volume: 586.5 MB/s
>>>        * bricks (in .glusterfs): 1.4 GB/s
>>>
>>>      The profile for the gluster test-volume is in
>>>
>>>      http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/profile_testvol_gluster.txt
>>>      <http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/profile_testvol_gluster.txt>
>>>
>>>      Thanks
>>>
>>>      Pat
>>>
>>>
>>>
>>>
>>>      On 05/30/2017 12:10 PM, Pranith Kumar Karampuri wrote:
>>>>      Let's start with the same 'dd' test we were testing with to see,
>>>>      what the numbers are. Please provide profile numbers for the
>>>>      same. From there on we will start tuning the volume to see what
>>>>      we can do.
>>>>
>>>>      On Tue, May 30, 2017 at 9:16 PM, Pat Haley <phaley at mit.edu
>>>>      <mailto:phaley at mit.edu>> wrote:
>>>>
>>>>
>>>>          Hi Pranith,
>>>>
>>>>          Thanks for the tip.  We now have the gluster volume mounted
>>>>          under /home.  What tests do you recommend we run?
>>>>
>>>>          Thanks
>>>>
>>>>          Pat
>>>>
>>>>
>>>>
>>>>          On 05/17/2017 05:01 AM, Pranith Kumar Karampuri wrote:
>>>>>
>>>>>          On Tue, May 16, 2017 at 9:20 PM, Pat Haley <phaley at mit.edu
>>>>>          <mailto:phaley at mit.edu>> wrote:
>>>>>
>>>>>
>>>>>              Hi Pranith,
>>>>>
>>>>>              Sorry for the delay.  I never saw received your reply
>>>>>              (but I did receive Ben Turner's follow-up to your
>>>>>              reply).  So we tried to create a gluster volume under
>>>>>              /home using different variations of
>>>>>
>>>>>              gluster volume create test-volume
>>>>>              mseas-data2:/home/gbrick_test_1
>>>>>              mseas-data2:/home/gbrick_test_2 transport tcp
>>>>>
>>>>>              However we keep getting errors of the form
>>>>>
>>>>>              Wrong brick type: transport, use
>>>>>              <HOSTNAME>:<export-dir-abs-path>
>>>>>
>>>>>              Any thoughts on what we're doing wrong?
>>>>>
>>>>>
>>>>>          You should give transport tcp at the beginning I think.
>>>>>          Anyways, transport tcp is the default, so no need to specify
>>>>>          so remove those two words from the CLI.
>>>>>
>>>>>
>>>>>              Also do you have a list of the test we should be running
>>>>>              once we get this volume created?  Given the time-zone
>>>>>              difference it might help if we can run a small battery
>>>>>              of tests and post the results rather than test-post-new
>>>>>              test-post... .
>>>>>
>>>>>
>>>>>          This is the first time I am doing performance analysis on
>>>>>          users as far as I remember. In our team there are separate
>>>>>          engineers who do these tests. Ben who replied earlier is one
>>>>>          such engineer.
>>>>>
>>>>>          Ben,
>>>>>              Have any suggestions?
>>>>>
>>>>>
>>>>>              Thanks
>>>>>
>>>>>              Pat
>>>>>
>>>>>
>>>>>
>>>>>              On 05/11/2017 12:06 PM, Pranith Kumar Karampuri wrote:
>>>>>>
>>>>>>              On Thu, May 11, 2017 at 9:32 PM, Pat Haley
>>>>>>              <phaley at mit.edu <mailto:phaley at mit.edu>> wrote:
>>>>>>
>>>>>>
>>>>>>                  Hi Pranith,
>>>>>>
>>>>>>                  The /home partition is mounted as ext4
>>>>>>                  /home ext4 defaults,usrquota,grpquota   1 2
>>>>>>
>>>>>>                  The brick partitions are mounted ax xfs
>>>>>>                  /mnt/brick1 xfs defaults 0 0
>>>>>>                  /mnt/brick2 xfs defaults 0 0
>>>>>>
>>>>>>                  Will this cause a problem with creating a volume
>>>>>>                  under /home?
>>>>>>
>>>>>>
>>>>>>              I don't think the bottleneck is disk. You can do the
>>>>>>              same tests you did on your new volume to confirm?
>>>>>>
>>>>>>
>>>>>>                  Pat
>>>>>>
>>>>>>
>>>>>>
>>>>>>                  On 05/11/2017 11:32 AM, Pranith Kumar Karampuri wrote:
>>>>>>>
>>>>>>>                  On Thu, May 11, 2017 at 8:57 PM, Pat Haley
>>>>>>>                  <phaley at mit.edu <mailto:phaley at mit.edu>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>                      Hi Pranith,
>>>>>>>
>>>>>>>                      Unfortunately, we don't have similar hardware
>>>>>>>                      for a small scale test.  All we have is our
>>>>>>>                      production hardware.
>>>>>>>
>>>>>>>
>>>>>>>                  You said something about /home partition which has
>>>>>>>                  lesser disks, we can create plain distribute
>>>>>>>                  volume inside one of those directories. After we
>>>>>>>                  are done, we can remove the setup. What do you say?
>>>>>>>
>>>>>>>
>>>>>>>                      Pat
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                      On 05/11/2017 07:05 AM, Pranith Kumar
>>>>>>>                      Karampuri wrote:
>>>>>>>>
>>>>>>>>                      On Thu, May 11, 2017 at 2:48 AM, Pat Haley
>>>>>>>>                      <phaley at mit.edu <mailto:phaley at mit.edu>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>                          Hi Pranith,
>>>>>>>>
>>>>>>>>                          Since we are mounting the partitions as
>>>>>>>>                          the bricks, I tried the dd test writing
>>>>>>>>                          to
>>>>>>>>                          <brick-path>/.glusterfs/<file-to-be-removed-after-test>.
>>>>>>>>                          The results without oflag=sync were 1.6
>>>>>>>>                          Gb/s (faster than gluster but not as fast
>>>>>>>>                          as I was expecting given the 1.2 Gb/s to
>>>>>>>>                          the no-gluster area w/ fewer disks).
>>>>>>>>
>>>>>>>>
>>>>>>>>                      Okay, then 1.6Gb/s is what we need to target
>>>>>>>>                      for, considering your volume is just
>>>>>>>>                      distribute. Is there any way you can do tests
>>>>>>>>                      on similar hardware but at a small scale?
>>>>>>>>                      Just so we can run the workload to learn more
>>>>>>>>                      about the bottlenecks in the system? We can
>>>>>>>>                      probably try to get the speed to 1.2Gb/s on
>>>>>>>>                      your /home partition you were telling me
>>>>>>>>                      yesterday. Let me know if that is something
>>>>>>>>                      you are okay to do.
>>>>>>>>
>>>>>>>>
>>>>>>>>                          Pat
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                          On 05/10/2017 01:27 PM, Pranith Kumar
>>>>>>>>                          Karampuri wrote:
>>>>>>>>>
>>>>>>>>>                          On Wed, May 10, 2017 at 10:15 PM, Pat
>>>>>>>>>                          Haley <phaley at mit.edu
>>>>>>>>>                          <mailto:phaley at mit.edu>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                              Hi Pranith,
>>>>>>>>>
>>>>>>>>>                              Not entirely sure (this isn't my
>>>>>>>>>                              area of expertise). I'll run your
>>>>>>>>>                              answer by some other people who are
>>>>>>>>>                              more familiar with this.
>>>>>>>>>
>>>>>>>>>                              I am also uncertain about how to
>>>>>>>>>                              interpret the results when we also
>>>>>>>>>                              add the dd tests writing to the
>>>>>>>>>                              /home area (no gluster, still on the
>>>>>>>>>                              same machine)
>>>>>>>>>
>>>>>>>>>                                * dd test without oflag=sync
>>>>>>>>>                                  (rough average of multiple tests)
>>>>>>>>>                                    o gluster w/ fuse mount : 570 Mb/s
>>>>>>>>>                                    o gluster w/ nfs mount: 390 Mb/s
>>>>>>>>>                                    o nfs (no gluster):  1.2 Gb/s
>>>>>>>>>                                * dd test with oflag=sync (rough
>>>>>>>>>                                  average of multiple tests)
>>>>>>>>>                                    o gluster w/ fuse mount:  5 Mb/s
>>>>>>>>>                                    o gluster w/ nfs mount: 200 Mb/s
>>>>>>>>>                                    o nfs (no gluster): 20 Mb/s
>>>>>>>>>
>>>>>>>>>                              Given that the non-gluster area is a
>>>>>>>>>                              RAID-6 of 4 disks while each brick
>>>>>>>>>                              of the gluster area is a RAID-6 of
>>>>>>>>>                              32 disks, I would naively expect the
>>>>>>>>>                              writes to the gluster area to be
>>>>>>>>>                              roughly 8x faster than to the
>>>>>>>>>                              non-gluster.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                          I think a better test is to try and
>>>>>>>>>                          write to a file using nfs without any
>>>>>>>>>                          gluster to a location that is not inside
>>>>>>>>>                          the brick but someother location that is
>>>>>>>>>                          on same disk(s). If you are mounting the
>>>>>>>>>                          partition as the brick, then we can
>>>>>>>>>                          write to a file inside .glusterfs
>>>>>>>>>                          directory, something like
>>>>>>>>>                          <brick-path>/.glusterfs/<file-to-be-removed-after-test>.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                              I still think we have a speed issue,
>>>>>>>>>                              I can't tell if fuse vs nfs is part
>>>>>>>>>                              of the problem.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                          I got interested in the post because I
>>>>>>>>>                          read that fuse speed is lesser than nfs
>>>>>>>>>                          speed which is counter-intuitive to my
>>>>>>>>>                          understanding. So wanted clarifications.
>>>>>>>>>                          Now that I got my clarifications where
>>>>>>>>>                          fuse outperformed nfs without sync, we
>>>>>>>>>                          can resume testing as described above
>>>>>>>>>                          and try to find what it is. Based on
>>>>>>>>>                          your email-id I am guessing you are from
>>>>>>>>>                          Boston and I am from Bangalore so if you
>>>>>>>>>                          are okay with doing this debugging for
>>>>>>>>>                          multiple days because of timezones, I
>>>>>>>>>                          will be happy to help. Please be a bit
>>>>>>>>>                          patient with me, I am under a release
>>>>>>>>>                          crunch but I am very curious with the
>>>>>>>>>                          problem you posted.
>>>>>>>>>
>>>>>>>>>                              Was there anything useful in the
>>>>>>>>>                              profiles?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                          Unfortunately profiles didn't help me
>>>>>>>>>                          much, I think we are collecting the
>>>>>>>>>                          profiles from an active volume, so it
>>>>>>>>>                          has a lot of information that is not
>>>>>>>>>                          pertaining to dd so it is difficult to
>>>>>>>>>                          find the contributions of dd. So I went
>>>>>>>>>                          through your post again and found
>>>>>>>>>                          something I didn't pay much attention to
>>>>>>>>>                          earlier i.e. oflag=sync, so did my own
>>>>>>>>>                          tests on my setup with FUSE so sent that
>>>>>>>>>                          reply.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                              Pat
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                              On 05/10/2017 12:15 PM, Pranith
>>>>>>>>>                              Kumar Karampuri wrote:
>>>>>>>>>>                              Okay good. At least this validates
>>>>>>>>>>                              my doubts. Handling O_SYNC in
>>>>>>>>>>                              gluster NFS and fuse is a bit
>>>>>>>>>>                              different.
>>>>>>>>>>                              When application opens a file with
>>>>>>>>>>                              O_SYNC on fuse mount then each
>>>>>>>>>>                              write syscall has to be written to
>>>>>>>>>>                              disk as part of the syscall where
>>>>>>>>>>                              as in case of NFS, there is no
>>>>>>>>>>                              concept of open. NFS performs write
>>>>>>>>>>                              though a handle saying it needs to
>>>>>>>>>>                              be a synchronous write, so write()
>>>>>>>>>>                              syscall is performed first then it
>>>>>>>>>>                              performs fsync(). so an write on an
>>>>>>>>>>                              fd with O_SYNC becomes write+fsync.
>>>>>>>>>>                              I am suspecting that when multiple
>>>>>>>>>>                              threads do this write+fsync()
>>>>>>>>>>                              operation on the same file,
>>>>>>>>>>                              multiple writes are batched
>>>>>>>>>>                              together to be written do disk so
>>>>>>>>>>                              the throughput on the disk is
>>>>>>>>>>                              increasing is my guess.
>>>>>>>>>>
>>>>>>>>>>                              Does it answer your doubts?
>>>>>>>>>>
>>>>>>>>>>                              On Wed, May 10, 2017 at 9:35 PM,
>>>>>>>>>>                              Pat Haley <phaley at mit.edu
>>>>>>>>>>                              <mailto:phaley at mit.edu>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                                  Without the oflag=sync and only
>>>>>>>>>>                                  a single test of each, the FUSE
>>>>>>>>>>                                  is going faster than NFS:
>>>>>>>>>>
>>>>>>>>>>                                  FUSE:
>>>>>>>>>>                                  mseas-data2(dri_nascar)% dd
>>>>>>>>>>                                  if=/dev/zero count=4096
>>>>>>>>>>                                  bs=1048576 of=zeros.txt conv=sync
>>>>>>>>>>                                  4096+0 records in
>>>>>>>>>>                                  4096+0 records out
>>>>>>>>>>                                  4294967296 bytes (4.3 GB)
>>>>>>>>>>                                  copied, 7.46961 s, 575 MB/s
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                                  NFS
>>>>>>>>>>                                  mseas-data2(HYCOM)% dd
>>>>>>>>>>                                  if=/dev/zero count=4096
>>>>>>>>>>                                  bs=1048576 of=zeros.txt conv=sync
>>>>>>>>>>                                  4096+0 records in
>>>>>>>>>>                                  4096+0 records out
>>>>>>>>>>                                  4294967296 bytes (4.3 GB)
>>>>>>>>>>                                  copied, 11.4264 s, 376 MB/s
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                                  On 05/10/2017 11:53 AM, Pranith
>>>>>>>>>>                                  Kumar Karampuri wrote:
>>>>>>>>>>>                                  Could you let me know the
>>>>>>>>>>>                                  speed without oflag=sync on
>>>>>>>>>>>                                  both the mounts? No need to
>>>>>>>>>>>                                  collect profiles.
>>>>>>>>>>>
>>>>>>>>>>>                                  On Wed, May 10, 2017 at 9:17
>>>>>>>>>>>                                  PM, Pat Haley <phaley at mit.edu
>>>>>>>>>>>                                  <mailto:phaley at mit.edu>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                                      Here is what I see now:
>>>>>>>>>>>
>>>>>>>>>>>                                      [root at mseas-data2 ~]#
>>>>>>>>>>>                                      gluster volume info
>>>>>>>>>>>
>>>>>>>>>>>                                      Volume Name: data-volume
>>>>>>>>>>>                                      Type: Distribute
>>>>>>>>>>>                                      Volume ID:
>>>>>>>>>>>                                      c162161e-2a2d-4dac-b015-f31fd89ceb18
>>>>>>>>>>>                                      Status: Started
>>>>>>>>>>>                                      Number of Bricks: 2
>>>>>>>>>>>                                      Transport-type: tcp
>>>>>>>>>>>                                      Bricks:
>>>>>>>>>>>                                      Brick1:
>>>>>>>>>>>                                      mseas-data2:/mnt/brick1
>>>>>>>>>>>                                      Brick2:
>>>>>>>>>>>                                      mseas-data2:/mnt/brick2
>>>>>>>>>>>                                      Options Reconfigured:
>>>>>>>>>>>                                      diagnostics.count-fop-hits: on
>>>>>>>>>>>                                      diagnostics.latency-measurement:
>>>>>>>>>>>                                      on
>>>>>>>>>>>                                      nfs.exports-auth-enable: on
>>>>>>>>>>>                                      diagnostics.brick-sys-log-level:
>>>>>>>>>>>                                      WARNING
>>>>>>>>>>>                                      performance.readdir-ahead: on
>>>>>>>>>>>                                      nfs.disable: on
>>>>>>>>>>>                                      nfs.export-volumes: off
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                                      On 05/10/2017 11:44 AM,
>>>>>>>>>>>                                      Pranith Kumar Karampuri wrote:
>>>>>>>>>>>>                                      Is this the volume info
>>>>>>>>>>>>                                      you have?
>>>>>>>>>>>>
>>>>>>>>>>>>                                      >/[root at mseas-data2
>>>>>>>>>>>>                                      <http://www.gluster.org/mailman/listinfo/gluster-users>
>>>>>>>>>>>>                                      ~]# gluster volume info
>>>>>>>>>>>>                                      />//>/Volume Name:
>>>>>>>>>>>>                                      data-volume />/Type:
>>>>>>>>>>>>                                      Distribute />/Volume ID:
>>>>>>>>>>>>                                      c162161e-2a2d-4dac-b015-f31fd89ceb18
>>>>>>>>>>>>                                      />/Status: Started />/Number
>>>>>>>>>>>>                                      of Bricks: 2
>>>>>>>>>>>>                                      />/Transport-type: tcp
>>>>>>>>>>>>                                      />/Bricks: />/Brick1:
>>>>>>>>>>>>                                      mseas-data2:/mnt/brick1
>>>>>>>>>>>>                                      />/Brick2:
>>>>>>>>>>>>                                      mseas-data2:/mnt/brick2
>>>>>>>>>>>>                                      />/Options Reconfigured:
>>>>>>>>>>>>                                      />/performance.readdir-ahead:
>>>>>>>>>>>>                                      on />/nfs.disable: on
>>>>>>>>>>>>                                      />/nfs.export-volumes: off /
>>>>>>>>>>>>                                      ​I copied this from old
>>>>>>>>>>>>                                      thread from 2016. This is
>>>>>>>>>>>>                                      distribute volume. Did
>>>>>>>>>>>>                                      you change any of the
>>>>>>>>>>>>                                      options in between?
>>>>>>>>>>>                                      --
>>>>>>>>>>>
>>>>>>>>>>>                                      -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>>>                                      Pat Haley
>>>>>>>>>>>                                      Email:phaley at mit.edu
>>>>>>>>>>>                                      <mailto:phaley at mit.edu>
>>>>>>>>>>>                                      Center for Ocean Engineering
>>>>>>>>>>>                                      Phone:  (617) 253-6824
>>>>>>>>>>>                                      Dept. of Mechanical Engineering
>>>>>>>>>>>                                      Fax:    (617) 253-8125
>>>>>>>>>>>                                      MIT, Room
>>>>>>>>>>>                                      5-213http://web.mit.edu/phaley/www/
>>>>>>>>>>>                                      77 Massachusetts Avenue
>>>>>>>>>>>                                      Cambridge, MA  02139-4301
>>>>>>>>>>>
>>>>>>>>>>>                                  --
>>>>>>>>>>>                                  Pranith
>>>>>>>>>>                                  --
>>>>>>>>>>
>>>>>>>>>>                                  -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>>                                  Pat Haley
>>>>>>>>>>                                  Email:phaley at mit.edu
>>>>>>>>>>                                  <mailto:phaley at mit.edu>
>>>>>>>>>>                                  Center for Ocean Engineering
>>>>>>>>>>                                  Phone:  (617) 253-6824
>>>>>>>>>>                                  Dept. of Mechanical Engineering
>>>>>>>>>>                                  Fax:    (617) 253-8125
>>>>>>>>>>                                  MIT, Room
>>>>>>>>>>                                  5-213http://web.mit.edu/phaley/www/
>>>>>>>>>>                                  77 Massachusetts Avenue
>>>>>>>>>>                                  Cambridge, MA  02139-4301
>>>>>>>>>>
>>>>>>>>>>                              --
>>>>>>>>>>                              Pranith
>>>>>>>>>                              --
>>>>>>>>>
>>>>>>>>>                              -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>                              Pat Haley
>>>>>>>>>                              Email:phaley at mit.edu
>>>>>>>>>                              <mailto:phaley at mit.edu>
>>>>>>>>>                              Center for Ocean Engineering       Phone:
>>>>>>>>>                              (617) 253-6824
>>>>>>>>>                              Dept. of Mechanical Engineering    Fax:
>>>>>>>>>                              (617) 253-8125
>>>>>>>>>                              MIT, Room
>>>>>>>>>                              5-213http://web.mit.edu/phaley/www/
>>>>>>>>>                              77 Massachusetts Avenue
>>>>>>>>>                              Cambridge, MA  02139-4301
>>>>>>>>>
>>>>>>>>>                          --
>>>>>>>>>                          Pranith
>>>>>>>>                          --
>>>>>>>>
>>>>>>>>                          -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>                          Pat Haley
>>>>>>>>                          Email:phaley at mit.edu
>>>>>>>>                          <mailto:phaley at mit.edu>
>>>>>>>>                          Center for Ocean Engineering       Phone:
>>>>>>>>                          (617) 253-6824
>>>>>>>>                          Dept. of Mechanical Engineering    Fax:
>>>>>>>>                          (617) 253-8125
>>>>>>>>                          MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>>>>>                          77 Massachusetts Avenue
>>>>>>>>                          Cambridge, MA  02139-4301
>>>>>>>>
>>>>>>>>                      --
>>>>>>>>                      Pranith
>>>>>>>                      --
>>>>>>>
>>>>>>>                      -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>                      Pat Haley
>>>>>>>                      Email:phaley at mit.edu
>>>>>>>                      <mailto:phaley at mit.edu>
>>>>>>>                      Center for Ocean Engineering       Phone:  (617)
>>>>>>>                      253-6824
>>>>>>>                      Dept. of Mechanical Engineering    Fax:    (617)
>>>>>>>                      253-8125
>>>>>>>                      MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>>>>                      77 Massachusetts Avenue
>>>>>>>                      Cambridge, MA  02139-4301
>>>>>>>
>>>>>>>                  --
>>>>>>>                  Pranith
>>>>>>                  --
>>>>>>
>>>>>>                  -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>                  Pat Haley                          Email:phaley at mit.edu
>>>>>>                  <mailto:phaley at mit.edu>
>>>>>>                  Center for Ocean Engineering       Phone:  (617)
>>>>>>                  253-6824
>>>>>>                  Dept. of Mechanical Engineering    Fax:    (617)
>>>>>>                  253-8125
>>>>>>                  MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>>>                  77 Massachusetts Avenue
>>>>>>                  Cambridge, MA  02139-4301
>>>>>>
>>>>>>              --
>>>>>>              Pranith
>>>>>              --
>>>>>
>>>>>              -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>              Pat Haley                          Email:phaley at mit.edu
>>>>>              <mailto:phaley at mit.edu>
>>>>>              Center for Ocean Engineering       Phone:  (617) 253-6824
>>>>>              Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>>>>              MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>>              77 Massachusetts Avenue
>>>>>              Cambridge, MA  02139-4301
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>          --
>>>>>          Pranith
>>>>          --
>>>>
>>>>          -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>          Pat Haley                          Email:phaley at mit.edu
>>>>          <mailto:phaley at mit.edu>
>>>>          Center for Ocean Engineering       Phone:  (617) 253-6824
>>>>          Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>>>          MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>          77 Massachusetts Avenue
>>>>          Cambridge, MA  02139-4301
>>>>
>>>>
>>>>
>>>>
>>>>      --
>>>>      Pranith
>>>      --
>>>
>>>      -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>      Pat Haley                          Email:phaley at mit.edu
>>>      <mailto:phaley at mit.edu>
>>>      Center for Ocean Engineering       Phone:  (617) 253-6824
>>>      Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>>      MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>      77 Massachusetts Avenue
>>>      Cambridge, MA  02139-4301
>>>
>>>
>>>
>>>
>>> --
>>> Pranith
>> --
>>
>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> Pat Haley                          Email:  phaley at mit.edu
>> Center for Ocean Engineering       Phone:  (617) 253-6824
>> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>> MIT, Room 5-213                    http://web.mit.edu/phaley/www/
>> 77 Massachusetts Avenue
>> Cambridge, MA  02139-4301
>>
>>

-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301



More information about the Gluster-users mailing list