[Gluster-users] Slow write times to gluster disk

Ben Turner bturner at redhat.com
Fri Jun 2 05:07:28 UTC 2017


Are you sure using conv=sync is what you want?  I normally use conv=fdatasync, I'll look up the difference between the two and see if it affects your test.


-b

----- Original Message -----
> From: "Pat Haley" <phaley at mit.edu>
> To: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> Cc: "Ravishankar N" <ravishankar at redhat.com>, gluster-users at gluster.org, "Steve Postma" <SPostma at ztechnet.com>, "Ben
> Turner" <bturner at redhat.com>
> Sent: Tuesday, May 30, 2017 9:40:34 PM
> Subject: Re: [Gluster-users] Slow write times to gluster disk
> 
> 
> Hi Pranith,
> 
> The "dd" command was:
> 
>      dd if=/dev/zero count=4096 bs=1048576 of=zeros.txt conv=sync
> 
> There were 2 instances where dd reported 22 seconds. The output from the
> dd tests are in
> 
> http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/dd_testvol_gluster.txt
> 
> Pat
> 
> On 05/30/2017 09:27 PM, Pranith Kumar Karampuri wrote:
> > Pat,
> >        What is the command you used? As per the following output, it
> > seems like at least one write operation took 16 seconds. Which is
> > really bad.
> >       96.39    1165.10 us      89.00 us*16487014.00 us*          393212
> >       WRITE
> >
> >
> > On Tue, May 30, 2017 at 10:36 PM, Pat Haley <phaley at mit.edu
> > <mailto:phaley at mit.edu>> wrote:
> >
> >
> >     Hi Pranith,
> >
> >     I ran the same 'dd' test both in the gluster test volume and in
> >     the .glusterfs directory of each brick.  The median results (12 dd
> >     trials in each test) are similar to before
> >
> >       * gluster test volume: 586.5 MB/s
> >       * bricks (in .glusterfs): 1.4 GB/s
> >
> >     The profile for the gluster test-volume is in
> >
> >     http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/profile_testvol_gluster.txt
> >     <http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/profile_testvol_gluster.txt>
> >
> >     Thanks
> >
> >     Pat
> >
> >
> >
> >
> >     On 05/30/2017 12:10 PM, Pranith Kumar Karampuri wrote:
> >>     Let's start with the same 'dd' test we were testing with to see,
> >>     what the numbers are. Please provide profile numbers for the
> >>     same. From there on we will start tuning the volume to see what
> >>     we can do.
> >>
> >>     On Tue, May 30, 2017 at 9:16 PM, Pat Haley <phaley at mit.edu
> >>     <mailto:phaley at mit.edu>> wrote:
> >>
> >>
> >>         Hi Pranith,
> >>
> >>         Thanks for the tip.  We now have the gluster volume mounted
> >>         under /home.  What tests do you recommend we run?
> >>
> >>         Thanks
> >>
> >>         Pat
> >>
> >>
> >>
> >>         On 05/17/2017 05:01 AM, Pranith Kumar Karampuri wrote:
> >>>
> >>>
> >>>         On Tue, May 16, 2017 at 9:20 PM, Pat Haley <phaley at mit.edu
> >>>         <mailto:phaley at mit.edu>> wrote:
> >>>
> >>>
> >>>             Hi Pranith,
> >>>
> >>>             Sorry for the delay.  I never saw received your reply
> >>>             (but I did receive Ben Turner's follow-up to your
> >>>             reply).  So we tried to create a gluster volume under
> >>>             /home using different variations of
> >>>
> >>>             gluster volume create test-volume
> >>>             mseas-data2:/home/gbrick_test_1
> >>>             mseas-data2:/home/gbrick_test_2 transport tcp
> >>>
> >>>             However we keep getting errors of the form
> >>>
> >>>             Wrong brick type: transport, use
> >>>             <HOSTNAME>:<export-dir-abs-path>
> >>>
> >>>             Any thoughts on what we're doing wrong?
> >>>
> >>>
> >>>         You should give transport tcp at the beginning I think.
> >>>         Anyways, transport tcp is the default, so no need to specify
> >>>         so remove those two words from the CLI.
> >>>
> >>>
> >>>             Also do you have a list of the test we should be running
> >>>             once we get this volume created?  Given the time-zone
> >>>             difference it might help if we can run a small battery
> >>>             of tests and post the results rather than test-post-new
> >>>             test-post... .
> >>>
> >>>
> >>>         This is the first time I am doing performance analysis on
> >>>         users as far as I remember. In our team there are separate
> >>>         engineers who do these tests. Ben who replied earlier is one
> >>>         such engineer.
> >>>
> >>>         Ben,
> >>>             Have any suggestions?
> >>>
> >>>
> >>>             Thanks
> >>>
> >>>             Pat
> >>>
> >>>
> >>>
> >>>             On 05/11/2017 12:06 PM, Pranith Kumar Karampuri wrote:
> >>>>
> >>>>
> >>>>             On Thu, May 11, 2017 at 9:32 PM, Pat Haley
> >>>>             <phaley at mit.edu <mailto:phaley at mit.edu>> wrote:
> >>>>
> >>>>
> >>>>                 Hi Pranith,
> >>>>
> >>>>                 The /home partition is mounted as ext4
> >>>>                 /home ext4 defaults,usrquota,grpquota   1 2
> >>>>
> >>>>                 The brick partitions are mounted ax xfs
> >>>>                 /mnt/brick1 xfs defaults 0 0
> >>>>                 /mnt/brick2 xfs defaults 0 0
> >>>>
> >>>>                 Will this cause a problem with creating a volume
> >>>>                 under /home?
> >>>>
> >>>>
> >>>>             I don't think the bottleneck is disk. You can do the
> >>>>             same tests you did on your new volume to confirm?
> >>>>
> >>>>
> >>>>                 Pat
> >>>>
> >>>>
> >>>>
> >>>>                 On 05/11/2017 11:32 AM, Pranith Kumar Karampuri wrote:
> >>>>>
> >>>>>
> >>>>>                 On Thu, May 11, 2017 at 8:57 PM, Pat Haley
> >>>>>                 <phaley at mit.edu <mailto:phaley at mit.edu>> wrote:
> >>>>>
> >>>>>
> >>>>>                     Hi Pranith,
> >>>>>
> >>>>>                     Unfortunately, we don't have similar hardware
> >>>>>                     for a small scale test.  All we have is our
> >>>>>                     production hardware.
> >>>>>
> >>>>>
> >>>>>                 You said something about /home partition which has
> >>>>>                 lesser disks, we can create plain distribute
> >>>>>                 volume inside one of those directories. After we
> >>>>>                 are done, we can remove the setup. What do you say?
> >>>>>
> >>>>>
> >>>>>                     Pat
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>                     On 05/11/2017 07:05 AM, Pranith Kumar
> >>>>>                     Karampuri wrote:
> >>>>>>
> >>>>>>
> >>>>>>                     On Thu, May 11, 2017 at 2:48 AM, Pat Haley
> >>>>>>                     <phaley at mit.edu <mailto:phaley at mit.edu>> wrote:
> >>>>>>
> >>>>>>
> >>>>>>                         Hi Pranith,
> >>>>>>
> >>>>>>                         Since we are mounting the partitions as
> >>>>>>                         the bricks, I tried the dd test writing
> >>>>>>                         to
> >>>>>>                         <brick-path>/.glusterfs/<file-to-be-removed-after-test>.
> >>>>>>                         The results without oflag=sync were 1.6
> >>>>>>                         Gb/s (faster than gluster but not as fast
> >>>>>>                         as I was expecting given the 1.2 Gb/s to
> >>>>>>                         the no-gluster area w/ fewer disks).
> >>>>>>
> >>>>>>
> >>>>>>                     Okay, then 1.6Gb/s is what we need to target
> >>>>>>                     for, considering your volume is just
> >>>>>>                     distribute. Is there any way you can do tests
> >>>>>>                     on similar hardware but at a small scale?
> >>>>>>                     Just so we can run the workload to learn more
> >>>>>>                     about the bottlenecks in the system? We can
> >>>>>>                     probably try to get the speed to 1.2Gb/s on
> >>>>>>                     your /home partition you were telling me
> >>>>>>                     yesterday. Let me know if that is something
> >>>>>>                     you are okay to do.
> >>>>>>
> >>>>>>
> >>>>>>                         Pat
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>                         On 05/10/2017 01:27 PM, Pranith Kumar
> >>>>>>                         Karampuri wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>                         On Wed, May 10, 2017 at 10:15 PM, Pat
> >>>>>>>                         Haley <phaley at mit.edu
> >>>>>>>                         <mailto:phaley at mit.edu>> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>                             Hi Pranith,
> >>>>>>>
> >>>>>>>                             Not entirely sure (this isn't my
> >>>>>>>                             area of expertise). I'll run your
> >>>>>>>                             answer by some other people who are
> >>>>>>>                             more familiar with this.
> >>>>>>>
> >>>>>>>                             I am also uncertain about how to
> >>>>>>>                             interpret the results when we also
> >>>>>>>                             add the dd tests writing to the
> >>>>>>>                             /home area (no gluster, still on the
> >>>>>>>                             same machine)
> >>>>>>>
> >>>>>>>                               * dd test without oflag=sync
> >>>>>>>                                 (rough average of multiple tests)
> >>>>>>>                                   o gluster w/ fuse mount : 570 Mb/s
> >>>>>>>                                   o gluster w/ nfs mount: 390 Mb/s
> >>>>>>>                                   o nfs (no gluster):  1.2 Gb/s
> >>>>>>>                               * dd test with oflag=sync (rough
> >>>>>>>                                 average of multiple tests)
> >>>>>>>                                   o gluster w/ fuse mount:  5 Mb/s
> >>>>>>>                                   o gluster w/ nfs mount: 200 Mb/s
> >>>>>>>                                   o nfs (no gluster): 20 Mb/s
> >>>>>>>
> >>>>>>>                             Given that the non-gluster area is a
> >>>>>>>                             RAID-6 of 4 disks while each brick
> >>>>>>>                             of the gluster area is a RAID-6 of
> >>>>>>>                             32 disks, I would naively expect the
> >>>>>>>                             writes to the gluster area to be
> >>>>>>>                             roughly 8x faster than to the
> >>>>>>>                             non-gluster.
> >>>>>>>
> >>>>>>>
> >>>>>>>                         I think a better test is to try and
> >>>>>>>                         write to a file using nfs without any
> >>>>>>>                         gluster to a location that is not inside
> >>>>>>>                         the brick but someother location that is
> >>>>>>>                         on same disk(s). If you are mounting the
> >>>>>>>                         partition as the brick, then we can
> >>>>>>>                         write to a file inside .glusterfs
> >>>>>>>                         directory, something like
> >>>>>>>                         <brick-path>/.glusterfs/<file-to-be-removed-after-test>.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>                             I still think we have a speed issue,
> >>>>>>>                             I can't tell if fuse vs nfs is part
> >>>>>>>                             of the problem.
> >>>>>>>
> >>>>>>>
> >>>>>>>                         I got interested in the post because I
> >>>>>>>                         read that fuse speed is lesser than nfs
> >>>>>>>                         speed which is counter-intuitive to my
> >>>>>>>                         understanding. So wanted clarifications.
> >>>>>>>                         Now that I got my clarifications where
> >>>>>>>                         fuse outperformed nfs without sync, we
> >>>>>>>                         can resume testing as described above
> >>>>>>>                         and try to find what it is. Based on
> >>>>>>>                         your email-id I am guessing you are from
> >>>>>>>                         Boston and I am from Bangalore so if you
> >>>>>>>                         are okay with doing this debugging for
> >>>>>>>                         multiple days because of timezones, I
> >>>>>>>                         will be happy to help. Please be a bit
> >>>>>>>                         patient with me, I am under a release
> >>>>>>>                         crunch but I am very curious with the
> >>>>>>>                         problem you posted.
> >>>>>>>
> >>>>>>>                             Was there anything useful in the
> >>>>>>>                             profiles?
> >>>>>>>
> >>>>>>>
> >>>>>>>                         Unfortunately profiles didn't help me
> >>>>>>>                         much, I think we are collecting the
> >>>>>>>                         profiles from an active volume, so it
> >>>>>>>                         has a lot of information that is not
> >>>>>>>                         pertaining to dd so it is difficult to
> >>>>>>>                         find the contributions of dd. So I went
> >>>>>>>                         through your post again and found
> >>>>>>>                         something I didn't pay much attention to
> >>>>>>>                         earlier i.e. oflag=sync, so did my own
> >>>>>>>                         tests on my setup with FUSE so sent that
> >>>>>>>                         reply.
> >>>>>>>
> >>>>>>>
> >>>>>>>                             Pat
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>                             On 05/10/2017 12:15 PM, Pranith
> >>>>>>>                             Kumar Karampuri wrote:
> >>>>>>>>                             Okay good. At least this validates
> >>>>>>>>                             my doubts. Handling O_SYNC in
> >>>>>>>>                             gluster NFS and fuse is a bit
> >>>>>>>>                             different.
> >>>>>>>>                             When application opens a file with
> >>>>>>>>                             O_SYNC on fuse mount then each
> >>>>>>>>                             write syscall has to be written to
> >>>>>>>>                             disk as part of the syscall where
> >>>>>>>>                             as in case of NFS, there is no
> >>>>>>>>                             concept of open. NFS performs write
> >>>>>>>>                             though a handle saying it needs to
> >>>>>>>>                             be a synchronous write, so write()
> >>>>>>>>                             syscall is performed first then it
> >>>>>>>>                             performs fsync(). so an write on an
> >>>>>>>>                             fd with O_SYNC becomes write+fsync.
> >>>>>>>>                             I am suspecting that when multiple
> >>>>>>>>                             threads do this write+fsync()
> >>>>>>>>                             operation on the same file,
> >>>>>>>>                             multiple writes are batched
> >>>>>>>>                             together to be written do disk so
> >>>>>>>>                             the throughput on the disk is
> >>>>>>>>                             increasing is my guess.
> >>>>>>>>
> >>>>>>>>                             Does it answer your doubts?
> >>>>>>>>
> >>>>>>>>                             On Wed, May 10, 2017 at 9:35 PM,
> >>>>>>>>                             Pat Haley <phaley at mit.edu
> >>>>>>>>                             <mailto:phaley at mit.edu>> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>                                 Without the oflag=sync and only
> >>>>>>>>                                 a single test of each, the FUSE
> >>>>>>>>                                 is going faster than NFS:
> >>>>>>>>
> >>>>>>>>                                 FUSE:
> >>>>>>>>                                 mseas-data2(dri_nascar)% dd
> >>>>>>>>                                 if=/dev/zero count=4096
> >>>>>>>>                                 bs=1048576 of=zeros.txt conv=sync
> >>>>>>>>                                 4096+0 records in
> >>>>>>>>                                 4096+0 records out
> >>>>>>>>                                 4294967296 bytes (4.3 GB)
> >>>>>>>>                                 copied, 7.46961 s, 575 MB/s
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>                                 NFS
> >>>>>>>>                                 mseas-data2(HYCOM)% dd
> >>>>>>>>                                 if=/dev/zero count=4096
> >>>>>>>>                                 bs=1048576 of=zeros.txt conv=sync
> >>>>>>>>                                 4096+0 records in
> >>>>>>>>                                 4096+0 records out
> >>>>>>>>                                 4294967296 bytes (4.3 GB)
> >>>>>>>>                                 copied, 11.4264 s, 376 MB/s
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>                                 On 05/10/2017 11:53 AM, Pranith
> >>>>>>>>                                 Kumar Karampuri wrote:
> >>>>>>>>>                                 Could you let me know the
> >>>>>>>>>                                 speed without oflag=sync on
> >>>>>>>>>                                 both the mounts? No need to
> >>>>>>>>>                                 collect profiles.
> >>>>>>>>>
> >>>>>>>>>                                 On Wed, May 10, 2017 at 9:17
> >>>>>>>>>                                 PM, Pat Haley <phaley at mit.edu
> >>>>>>>>>                                 <mailto:phaley at mit.edu>> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>                                     Here is what I see now:
> >>>>>>>>>
> >>>>>>>>>                                     [root at mseas-data2 ~]#
> >>>>>>>>>                                     gluster volume info
> >>>>>>>>>
> >>>>>>>>>                                     Volume Name: data-volume
> >>>>>>>>>                                     Type: Distribute
> >>>>>>>>>                                     Volume ID:
> >>>>>>>>>                                     c162161e-2a2d-4dac-b015-f31fd89ceb18
> >>>>>>>>>                                     Status: Started
> >>>>>>>>>                                     Number of Bricks: 2
> >>>>>>>>>                                     Transport-type: tcp
> >>>>>>>>>                                     Bricks:
> >>>>>>>>>                                     Brick1:
> >>>>>>>>>                                     mseas-data2:/mnt/brick1
> >>>>>>>>>                                     Brick2:
> >>>>>>>>>                                     mseas-data2:/mnt/brick2
> >>>>>>>>>                                     Options Reconfigured:
> >>>>>>>>>                                     diagnostics.count-fop-hits: on
> >>>>>>>>>                                     diagnostics.latency-measurement:
> >>>>>>>>>                                     on
> >>>>>>>>>                                     nfs.exports-auth-enable: on
> >>>>>>>>>                                     diagnostics.brick-sys-log-level:
> >>>>>>>>>                                     WARNING
> >>>>>>>>>                                     performance.readdir-ahead: on
> >>>>>>>>>                                     nfs.disable: on
> >>>>>>>>>                                     nfs.export-volumes: off
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>                                     On 05/10/2017 11:44 AM,
> >>>>>>>>>                                     Pranith Kumar Karampuri wrote:
> >>>>>>>>>>                                     Is this the volume info
> >>>>>>>>>>                                     you have?
> >>>>>>>>>>
> >>>>>>>>>>                                     >/[root at mseas-data2
> >>>>>>>>>>                                     <http://www.gluster.org/mailman/listinfo/gluster-users>
> >>>>>>>>>>                                     ~]# gluster volume info
> >>>>>>>>>>                                     />//>/Volume Name:
> >>>>>>>>>>                                     data-volume />/Type:
> >>>>>>>>>>                                     Distribute />/Volume ID:
> >>>>>>>>>>                                     c162161e-2a2d-4dac-b015-f31fd89ceb18
> >>>>>>>>>>                                     />/Status: Started />/Number
> >>>>>>>>>>                                     of Bricks: 2
> >>>>>>>>>>                                     />/Transport-type: tcp
> >>>>>>>>>>                                     />/Bricks: />/Brick1:
> >>>>>>>>>>                                     mseas-data2:/mnt/brick1
> >>>>>>>>>>                                     />/Brick2:
> >>>>>>>>>>                                     mseas-data2:/mnt/brick2
> >>>>>>>>>>                                     />/Options Reconfigured:
> >>>>>>>>>>                                     />/performance.readdir-ahead:
> >>>>>>>>>>                                     on />/nfs.disable: on
> >>>>>>>>>>                                     />/nfs.export-volumes: off /
> >>>>>>>>>>                                     ​I copied this from old
> >>>>>>>>>>                                     thread from 2016. This is
> >>>>>>>>>>                                     distribute volume. Did
> >>>>>>>>>>                                     you change any of the
> >>>>>>>>>>                                     options in between?
> >>>>>>>>>
> >>>>>>>>>                                     --
> >>>>>>>>>
> >>>>>>>>>                                     -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> >>>>>>>>>                                     Pat Haley
> >>>>>>>>>                                     Email:phaley at mit.edu
> >>>>>>>>>                                     <mailto:phaley at mit.edu>
> >>>>>>>>>                                     Center for Ocean Engineering
> >>>>>>>>>                                     Phone:  (617) 253-6824
> >>>>>>>>>                                     Dept. of Mechanical Engineering
> >>>>>>>>>                                     Fax:    (617) 253-8125
> >>>>>>>>>                                     MIT, Room
> >>>>>>>>>                                     5-213http://web.mit.edu/phaley/www/
> >>>>>>>>>                                     77 Massachusetts Avenue
> >>>>>>>>>                                     Cambridge, MA  02139-4301
> >>>>>>>>>
> >>>>>>>>>                                 --
> >>>>>>>>>                                 Pranith
> >>>>>>>>
> >>>>>>>>                                 --
> >>>>>>>>
> >>>>>>>>                                 -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> >>>>>>>>                                 Pat Haley
> >>>>>>>>                                 Email:phaley at mit.edu
> >>>>>>>>                                 <mailto:phaley at mit.edu>
> >>>>>>>>                                 Center for Ocean Engineering
> >>>>>>>>                                 Phone:  (617) 253-6824
> >>>>>>>>                                 Dept. of Mechanical Engineering
> >>>>>>>>                                 Fax:    (617) 253-8125
> >>>>>>>>                                 MIT, Room
> >>>>>>>>                                 5-213http://web.mit.edu/phaley/www/
> >>>>>>>>                                 77 Massachusetts Avenue
> >>>>>>>>                                 Cambridge, MA  02139-4301
> >>>>>>>>
> >>>>>>>>                             --
> >>>>>>>>                             Pranith
> >>>>>>>
> >>>>>>>                             --
> >>>>>>>
> >>>>>>>                             -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> >>>>>>>                             Pat Haley
> >>>>>>>                             Email:phaley at mit.edu
> >>>>>>>                             <mailto:phaley at mit.edu>
> >>>>>>>                             Center for Ocean Engineering       Phone:
> >>>>>>>                             (617) 253-6824
> >>>>>>>                             Dept. of Mechanical Engineering    Fax:
> >>>>>>>                             (617) 253-8125
> >>>>>>>                             MIT, Room
> >>>>>>>                             5-213http://web.mit.edu/phaley/www/
> >>>>>>>                             77 Massachusetts Avenue
> >>>>>>>                             Cambridge, MA  02139-4301
> >>>>>>>
> >>>>>>>                         --
> >>>>>>>                         Pranith
> >>>>>>
> >>>>>>                         --
> >>>>>>
> >>>>>>                         -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> >>>>>>                         Pat Haley
> >>>>>>                         Email:phaley at mit.edu
> >>>>>>                         <mailto:phaley at mit.edu>
> >>>>>>                         Center for Ocean Engineering       Phone:
> >>>>>>                         (617) 253-6824
> >>>>>>                         Dept. of Mechanical Engineering    Fax:
> >>>>>>                         (617) 253-8125
> >>>>>>                         MIT, Room 5-213http://web.mit.edu/phaley/www/
> >>>>>>                         77 Massachusetts Avenue
> >>>>>>                         Cambridge, MA  02139-4301
> >>>>>>
> >>>>>>                     --
> >>>>>>                     Pranith
> >>>>>
> >>>>>                     --
> >>>>>
> >>>>>                     -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> >>>>>                     Pat Haley
> >>>>>                     Email:phaley at mit.edu
> >>>>>                     <mailto:phaley at mit.edu>
> >>>>>                     Center for Ocean Engineering       Phone:  (617)
> >>>>>                     253-6824
> >>>>>                     Dept. of Mechanical Engineering    Fax:    (617)
> >>>>>                     253-8125
> >>>>>                     MIT, Room 5-213http://web.mit.edu/phaley/www/
> >>>>>                     77 Massachusetts Avenue
> >>>>>                     Cambridge, MA  02139-4301
> >>>>>
> >>>>>                 --
> >>>>>                 Pranith
> >>>>
> >>>>                 --
> >>>>
> >>>>                 -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> >>>>                 Pat Haley                          Email:phaley at mit.edu
> >>>>                 <mailto:phaley at mit.edu>
> >>>>                 Center for Ocean Engineering       Phone:  (617)
> >>>>                 253-6824
> >>>>                 Dept. of Mechanical Engineering    Fax:    (617)
> >>>>                 253-8125
> >>>>                 MIT, Room 5-213http://web.mit.edu/phaley/www/
> >>>>                 77 Massachusetts Avenue
> >>>>                 Cambridge, MA  02139-4301
> >>>>
> >>>>             --
> >>>>             Pranith
> >>>
> >>>             --
> >>>
> >>>             -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> >>>             Pat Haley                          Email:phaley at mit.edu
> >>>             <mailto:phaley at mit.edu>
> >>>             Center for Ocean Engineering       Phone:  (617) 253-6824
> >>>             Dept. of Mechanical Engineering    Fax:    (617) 253-8125
> >>>             MIT, Room 5-213http://web.mit.edu/phaley/www/
> >>>             77 Massachusetts Avenue
> >>>             Cambridge, MA  02139-4301
> >>>
> >>>
> >>>
> >>>
> >>>         --
> >>>         Pranith
> >>
> >>         --
> >>
> >>         -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> >>         Pat Haley                          Email:phaley at mit.edu
> >>         <mailto:phaley at mit.edu>
> >>         Center for Ocean Engineering       Phone:  (617) 253-6824
> >>         Dept. of Mechanical Engineering    Fax:    (617) 253-8125
> >>         MIT, Room 5-213http://web.mit.edu/phaley/www/
> >>         77 Massachusetts Avenue
> >>         Cambridge, MA  02139-4301
> >>
> >>
> >>
> >>
> >>     --
> >>     Pranith
> >
> >     --
> >
> >     -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> >     Pat Haley                          Email:phaley at mit.edu
> >     <mailto:phaley at mit.edu>
> >     Center for Ocean Engineering       Phone:  (617) 253-6824
> >     Dept. of Mechanical Engineering    Fax:    (617) 253-8125
> >     MIT, Room 5-213http://web.mit.edu/phaley/www/
> >     77 Massachusetts Avenue
> >     Cambridge, MA  02139-4301
> >
> >
> >
> >
> > --
> > Pranith
> 
> --
> 
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley                          Email:  phaley at mit.edu
> Center for Ocean Engineering       Phone:  (617) 253-6824
> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
> MIT, Room 5-213                    http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA  02139-4301
> 
> 


More information about the Gluster-users mailing list