[Gluster-users] Slow write times to gluster disk

Pat Haley phaley at mit.edu
Thu Jun 22 20:53:42 UTC 2017


Hi,

Today we experimented with some of the FUSE options that we found in the 
list.

Changing these options had no effect:

gluster volume set test-volume performance.cache-max-file-size 2MB
gluster volume set test-volume performance.cache-refresh-timeout 4
gluster volume set test-volume performance.cache-size 256MB
gluster volume set test-volume performance.write-behind-window-size 4MB
gluster volume set test-volume performance.write-behind-window-size 8MB

Changing the following option from its default value made the speed slower

gluster volume set test-volume performance.write-behind off (on by default)

Changing the following options initially appeared to give a 10% increase 
in speed, but this vanished in subsequent tests (we think the apparent 
increase may have been to a lighter workload on the computer from other 
users)

gluster volume set test-volume performance.stat-prefetch on
gluster volume set test-volume client.event-threads 4
gluster volume set test-volume server.event-threads 4


Can anything be gleaned from these observations?  Are there other things 
we can try?

Thanks

Pat


On 06/20/2017 12:06 PM, Pat Haley wrote:
>
> Hi Ben,
>
> Sorry this took so long, but we had a real-time forecasting exercise 
> last week and I could only get to this now.
>
> Backend Hardware/OS:
>
>   * Much of the information on our back end system is included at the
>     top of
>     http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html
>   * The specific model of the hard disks is SeaGate ENTERPRISE
>     CAPACITY V.4 6TB (ST6000NM0024). The rated speed is 6Gb/s.
>   * Note: there is one physical server that hosts both the NFS and the
>     GlusterFS areas
>
> Latest tests
>
> I have had time to run the tests for one of the dd tests you requested 
> to the underlying XFS FS.  The median rate was 170 MB/s.  The dd 
> results and iostat record are in
>
> http://mseas.mit.edu/download/phaley/GlusterUsers/TestXFS/
>
> I'll add tests for the other brick and to the NFS area later.
>
> Thanks
>
> Pat
>
>
> On 06/12/2017 06:06 PM, Ben Turner wrote:
>> Ok you are correct, you have a pure distributed volume.  IE no replication overhead.  So normally for pure dist I use:
>>
>> throughput = slowest of disks / NIC * .6-.7
>>
>> In your case we have:
>>
>> 1200 * .6 = 720
>>
>> So you are seeing a little less throughput than I would expect in your configuration.  What I like to do here is:
>>
>> -First tell me more about your back end storage, will it sustain 1200 MB / sec?  What kind of HW?  How many disks?  What type and specs are the disks?  What kind of RAID are you using?
>>
>> -Second can you refresh me on your workload?  Are you doing reads / writes or both?  If both what mix?  Since we are using DD I assume you are working iwth large file sequential I/O, is this correct?
>>
>> -Run some DD tests on the back end XFS FS.  I normally have /xfs-mount/gluster-brick, if you have something similar just mkdir on the XFS -> /xfs-mount/my-test-dir.  Inside the test dir run:
>>
>> If you are focusing on a write workload run:
>>
>> # dd if=/dev/zero of=/xfs-mount/file bs=1024k count=10000 conv=fdatasync
>>
>> If you are focusing on a read workload run:
>>
>> # echo 3 > /proc/sys/vm/drop_caches
>> # dd if=/gluster-mount/file of=/dev/null bs=1024k count=10000
>>
>> ** MAKE SURE TO DROP CACHE IN BETWEEN READS!! **
>>
>> Run this in a loop similar to how you did in:
>>
>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/dd_testvol_gluster.txt
>>
>> Run this on both servers one at a time and if you are running on a SAN then run again on both at the same time.  While this is running gather iostat for me:
>>
>> # iostat -c -m -x 1 > iostat-$(hostname).txt
>>
>> Lets see how the back end performs on both servers while capturing iostat, then see how the same workload / data looks on gluster.
>>
>> -Last thing, when you run your kernel NFS tests are you using the same filesystem / storage you are using for the gluster bricks?  I want to be sure we have an apples to apples comparison here.
>>
>> -b
>>
>>
>>
>> ----- Original Message -----
>>> From: "Pat Haley"<phaley at mit.edu>
>>> To: "Ben Turner"<bturner at redhat.com>
>>> Sent: Monday, June 12, 2017 5:18:07 PM
>>> Subject: Re: [Gluster-users] Slow write times to gluster disk
>>>
>>>
>>> Hi Ben,
>>>
>>> Here is the output:
>>>
>>> [root at mseas-data2 ~]# gluster volume info
>>>
>>> Volume Name: data-volume
>>> Type: Distribute
>>> Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
>>> Status: Started
>>> Number of Bricks: 2
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: mseas-data2:/mnt/brick1
>>> Brick2: mseas-data2:/mnt/brick2
>>> Options Reconfigured:
>>> nfs.exports-auth-enable: on
>>> diagnostics.brick-sys-log-level: WARNING
>>> performance.readdir-ahead: on
>>> nfs.disable: on
>>> nfs.export-volumes: off
>>>
>>>
>>> On 06/12/2017 05:01 PM, Ben Turner wrote:
>>>> What is the output of gluster v info?  That will tell us more about your
>>>> config.
>>>>
>>>> -b
>>>>
>>>> ----- Original Message -----
>>>>> From: "Pat Haley"<phaley at mit.edu>
>>>>> To: "Ben Turner"<bturner at redhat.com>
>>>>> Sent: Monday, June 12, 2017 4:54:00 PM
>>>>> Subject: Re: [Gluster-users] Slow write times to gluster disk
>>>>>
>>>>>
>>>>> Hi Ben,
>>>>>
>>>>> I guess I'm confused about what you mean by replication.  If I look at
>>>>> the underlying bricks I only ever have a single copy of any file.  It
>>>>> either resides on one brick or the other  (directories exist on both
>>>>> bricks but not files).  We are not using gluster for redundancy (or at
>>>>> least that wasn't our intent).   Is that what you meant by replication
>>>>> or is it something else?
>>>>>
>>>>> Thanks
>>>>>
>>>>> Pat
>>>>>
>>>>> On 06/12/2017 04:28 PM, Ben Turner wrote:
>>>>>> ----- Original Message -----
>>>>>>> From: "Pat Haley"<phaley at mit.edu>
>>>>>>> To: "Ben Turner"<bturner at redhat.com>, "Pranith Kumar Karampuri"
>>>>>>> <pkarampu at redhat.com>
>>>>>>> Cc: "Ravishankar N"<ravishankar at redhat.com>,gluster-users at gluster.org,
>>>>>>> "Steve Postma"<SPostma at ztechnet.com>
>>>>>>> Sent: Monday, June 12, 2017 2:35:41 PM
>>>>>>> Subject: Re: [Gluster-users] Slow write times to gluster disk
>>>>>>>
>>>>>>>
>>>>>>> Hi Guys,
>>>>>>>
>>>>>>> I was wondering what our next steps should be to solve the slow write
>>>>>>> times.
>>>>>>>
>>>>>>> Recently I was debugging a large code and writing a lot of output at
>>>>>>> every time step.  When I tried writing to our gluster disks, it was
>>>>>>> taking over a day to do a single time step whereas if I had the same
>>>>>>> program (same hardware, network) write to our nfs disk the time per
>>>>>>> time-step was about 45 minutes. What we are shooting for here would be
>>>>>>> to have similar times to either gluster of nfs.
>>>>>> I can see in your test:
>>>>>>
>>>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/dd_testvol_gluster.txt
>>>>>>
>>>>>> You averaged ~600 MB / sec(expected for replica 2 with 10G, {~1200 MB /
>>>>>> sec} / #replicas{2} = 600).  Gluster does client side replication so with
>>>>>> replica 2 you will only ever see 1/2 the speed of your slowest part of
>>>>>> the
>>>>>> stack(NW, disk, RAM, CPU).  This is usually NW or disk and 600 is
>>>>>> normally
>>>>>> a best case.  Now in your output I do see the instances where you went
>>>>>> down to 200 MB / sec.  I can only explain this in three ways:
>>>>>>
>>>>>> 1.  You are not using conv=fdatasync and writes are actually going to
>>>>>> page
>>>>>> cache and then being flushed to disk.  During the fsync the memory is not
>>>>>> yet available and the disks are busy flushing dirty pages.
>>>>>> 2.  Your storage RAID group is shared across multiple LUNS(like in a SAN)
>>>>>> and when write times are slow the RAID group is busy serviceing other
>>>>>> LUNs.
>>>>>> 3.  Gluster bug / config issue / some other unknown unknown.
>>>>>>
>>>>>> So I see 2 issues here:
>>>>>>
>>>>>> 1.  NFS does in 45 minutes what gluster can do in 24 hours.
>>>>>> 2.  Sometimes your throughput drops dramatically.
>>>>>>
>>>>>> WRT #1 - have a look at my estimates above.  My formula for guestimating
>>>>>> gluster perf is: throughput = NIC throughput or storage(whatever is
>>>>>> slower) / # replicas * overhead(figure .7 or .8).  Also the larger the
>>>>>> record size the better for glusterfs mounts, I normally like to be at
>>>>>> LEAST 64k up to 1024k:
>>>>>>
>>>>>> # dd if=/dev/zero of=/gluster-mount/file bs=1024k count=10000
>>>>>> conv=fdatasync
>>>>>>
>>>>>> WRT #2 - Again, I question your testing and your storage config.  Try
>>>>>> using
>>>>>> conv=fdatasync for your DDs, use a larger record size, and make sure that
>>>>>> your back end storage is not causing your slowdowns.  Also remember that
>>>>>> with replica 2 you will take ~50% hit on writes because the client uses
>>>>>> 50% of its bandwidth to write to one replica and 50% to the other.
>>>>>>
>>>>>> -b
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Pat
>>>>>>>
>>>>>>>
>>>>>>> On 06/02/2017 01:07 AM, Ben Turner wrote:
>>>>>>>> Are you sure using conv=sync is what you want?  I normally use
>>>>>>>> conv=fdatasync, I'll look up the difference between the two and see if
>>>>>>>> it
>>>>>>>> affects your test.
>>>>>>>>
>>>>>>>>
>>>>>>>> -b
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Pat Haley"<phaley at mit.edu>
>>>>>>>>> To: "Pranith Kumar Karampuri"<pkarampu at redhat.com>
>>>>>>>>> Cc: "Ravishankar N"<ravishankar at redhat.com>,
>>>>>>>>> gluster-users at gluster.org,
>>>>>>>>> "Steve Postma"<SPostma at ztechnet.com>, "Ben
>>>>>>>>> Turner"<bturner at redhat.com>
>>>>>>>>> Sent: Tuesday, May 30, 2017 9:40:34 PM
>>>>>>>>> Subject: Re: [Gluster-users] Slow write times to gluster disk
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Pranith,
>>>>>>>>>
>>>>>>>>> The "dd" command was:
>>>>>>>>>
>>>>>>>>>          dd if=/dev/zero count=4096 bs=1048576 of=zeros.txt conv=sync
>>>>>>>>>
>>>>>>>>> There were 2 instances where dd reported 22 seconds. The output from
>>>>>>>>> the
>>>>>>>>> dd tests are in
>>>>>>>>>
>>>>>>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/dd_testvol_gluster.txt
>>>>>>>>>
>>>>>>>>> Pat
>>>>>>>>>
>>>>>>>>> On 05/30/2017 09:27 PM, Pranith Kumar Karampuri wrote:
>>>>>>>>>> Pat,
>>>>>>>>>>            What is the command you used? As per the following output,
>>>>>>>>>>            it
>>>>>>>>>> seems like at least one write operation took 16 seconds. Which is
>>>>>>>>>> really bad.
>>>>>>>>>>           96.39    1165.10 us      89.00 us*16487014.00 us*
>>>>>>>>>>           393212
>>>>>>>>>>           WRITE
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, May 30, 2017 at 10:36 PM, Pat Haley <phaley at mit.edu
>>>>>>>>>> <mailto:phaley at mit.edu>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>         Hi Pranith,
>>>>>>>>>>
>>>>>>>>>>         I ran the same 'dd' test both in the gluster test volume and
>>>>>>>>>>         in
>>>>>>>>>>         the .glusterfs directory of each brick.  The median results
>>>>>>>>>>         (12
>>>>>>>>>>         dd
>>>>>>>>>>         trials in each test) are similar to before
>>>>>>>>>>
>>>>>>>>>>           * gluster test volume: 586.5 MB/s
>>>>>>>>>>           * bricks (in .glusterfs): 1.4 GB/s
>>>>>>>>>>
>>>>>>>>>>         The profile for the gluster test-volume is in
>>>>>>>>>>
>>>>>>>>>>         http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/profile_testvol_gluster.txt
>>>>>>>>>>         <http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/profile_testvol_gluster.txt>
>>>>>>>>>>
>>>>>>>>>>         Thanks
>>>>>>>>>>
>>>>>>>>>>         Pat
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>         On 05/30/2017 12:10 PM, Pranith Kumar Karampuri wrote:
>>>>>>>>>>>         Let's start with the same 'dd' test we were testing with to
>>>>>>>>>>>         see,
>>>>>>>>>>>         what the numbers are. Please provide profile numbers for the
>>>>>>>>>>>         same. From there on we will start tuning the volume to see
>>>>>>>>>>>         what
>>>>>>>>>>>         we can do.
>>>>>>>>>>>
>>>>>>>>>>>         On Tue, May 30, 2017 at 9:16 PM, Pat Haley <phaley at mit.edu
>>>>>>>>>>>         <mailto:phaley at mit.edu>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>             Hi Pranith,
>>>>>>>>>>>
>>>>>>>>>>>             Thanks for the tip.  We now have the gluster volume
>>>>>>>>>>>             mounted
>>>>>>>>>>>             under /home.  What tests do you recommend we run?
>>>>>>>>>>>
>>>>>>>>>>>             Thanks
>>>>>>>>>>>
>>>>>>>>>>>             Pat
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>             On 05/17/2017 05:01 AM, Pranith Kumar Karampuri wrote:
>>>>>>>>>>>>             On Tue, May 16, 2017 at 9:20 PM, Pat Haley
>>>>>>>>>>>>             <phaley at mit.edu
>>>>>>>>>>>>             <mailto:phaley at mit.edu>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                 Hi Pranith,
>>>>>>>>>>>>
>>>>>>>>>>>>                 Sorry for the delay.  I never saw received your
>>>>>>>>>>>>                 reply
>>>>>>>>>>>>                 (but I did receive Ben Turner's follow-up to your
>>>>>>>>>>>>                 reply).  So we tried to create a gluster volume
>>>>>>>>>>>>                 under
>>>>>>>>>>>>                 /home using different variations of
>>>>>>>>>>>>
>>>>>>>>>>>>                 gluster volume create test-volume
>>>>>>>>>>>>                 mseas-data2:/home/gbrick_test_1
>>>>>>>>>>>>                 mseas-data2:/home/gbrick_test_2 transport tcp
>>>>>>>>>>>>
>>>>>>>>>>>>                 However we keep getting errors of the form
>>>>>>>>>>>>
>>>>>>>>>>>>                 Wrong brick type: transport, use
>>>>>>>>>>>>                 <HOSTNAME>:<export-dir-abs-path>
>>>>>>>>>>>>
>>>>>>>>>>>>                 Any thoughts on what we're doing wrong?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>             You should give transport tcp at the beginning I think.
>>>>>>>>>>>>             Anyways, transport tcp is the default, so no need to
>>>>>>>>>>>>             specify
>>>>>>>>>>>>             so remove those two words from the CLI.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                 Also do you have a list of the test we should be
>>>>>>>>>>>>                 running
>>>>>>>>>>>>                 once we get this volume created?  Given the
>>>>>>>>>>>>                 time-zone
>>>>>>>>>>>>                 difference it might help if we can run a small
>>>>>>>>>>>>                 battery
>>>>>>>>>>>>                 of tests and post the results rather than
>>>>>>>>>>>>                 test-post-new
>>>>>>>>>>>>                 test-post... .
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>             This is the first time I am doing performance analysis
>>>>>>>>>>>>             on
>>>>>>>>>>>>             users as far as I remember. In our team there are
>>>>>>>>>>>>             separate
>>>>>>>>>>>>             engineers who do these tests. Ben who replied earlier is
>>>>>>>>>>>>             one
>>>>>>>>>>>>             such engineer.
>>>>>>>>>>>>
>>>>>>>>>>>>             Ben,
>>>>>>>>>>>>                 Have any suggestions?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                 Thanks
>>>>>>>>>>>>
>>>>>>>>>>>>                 Pat
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                 On 05/11/2017 12:06 PM, Pranith Kumar Karampuri
>>>>>>>>>>>>                 wrote:
>>>>>>>>>>>>>                 On Thu, May 11, 2017 at 9:32 PM, Pat Haley
>>>>>>>>>>>>>                 <phaley at mit.edu  <mailto:phaley at mit.edu>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                     Hi Pranith,
>>>>>>>>>>>>>
>>>>>>>>>>>>>                     The /home partition is mounted as ext4
>>>>>>>>>>>>>                     /home ext4 defaults,usrquota,grpquota   1 2
>>>>>>>>>>>>>
>>>>>>>>>>>>>                     The brick partitions are mounted ax xfs
>>>>>>>>>>>>>                     /mnt/brick1 xfs defaults 0 0
>>>>>>>>>>>>>                     /mnt/brick2 xfs defaults 0 0
>>>>>>>>>>>>>
>>>>>>>>>>>>>                     Will this cause a problem with creating a
>>>>>>>>>>>>>                     volume
>>>>>>>>>>>>>                     under /home?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                 I don't think the bottleneck is disk. You can do
>>>>>>>>>>>>>                 the
>>>>>>>>>>>>>                 same tests you did on your new volume to confirm?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                     Pat
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                     On 05/11/2017 11:32 AM, Pranith Kumar Karampuri
>>>>>>>>>>>>>                     wrote:
>>>>>>>>>>>>>>                     On Thu, May 11, 2017 at 8:57 PM, Pat Haley
>>>>>>>>>>>>>>                     <phaley at mit.edu  <mailto:phaley at mit.edu>>
>>>>>>>>>>>>>>                     wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         Hi Pranith,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         Unfortunately, we don't have similar
>>>>>>>>>>>>>>                         hardware
>>>>>>>>>>>>>>                         for a small scale test.  All we have is
>>>>>>>>>>>>>>                         our
>>>>>>>>>>>>>>                         production hardware.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                     You said something about /home partition which
>>>>>>>>>>>>>>                     has
>>>>>>>>>>>>>>                     lesser disks, we can create plain distribute
>>>>>>>>>>>>>>                     volume inside one of those directories. After
>>>>>>>>>>>>>>                     we
>>>>>>>>>>>>>>                     are done, we can remove the setup. What do you
>>>>>>>>>>>>>>                     say?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         Pat
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         On 05/11/2017 07:05 AM, Pranith Kumar
>>>>>>>>>>>>>>                         Karampuri wrote:
>>>>>>>>>>>>>>>                         On Thu, May 11, 2017 at 2:48 AM, Pat
>>>>>>>>>>>>>>>                         Haley
>>>>>>>>>>>>>>>                         <phaley at mit.edu  <mailto:phaley at mit.edu>>
>>>>>>>>>>>>>>>                         wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                             Hi Pranith,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                             Since we are mounting the partitions
>>>>>>>>>>>>>>>                             as
>>>>>>>>>>>>>>>                             the bricks, I tried the dd test
>>>>>>>>>>>>>>>                             writing
>>>>>>>>>>>>>>>                             to
>>>>>>>>>>>>>>>                             <brick-path>/.glusterfs/<file-to-be-removed-after-test>.
>>>>>>>>>>>>>>>                             The results without oflag=sync were
>>>>>>>>>>>>>>>                             1.6
>>>>>>>>>>>>>>>                             Gb/s (faster than gluster but not as
>>>>>>>>>>>>>>>                             fast
>>>>>>>>>>>>>>>                             as I was expecting given the 1.2 Gb/s
>>>>>>>>>>>>>>>                             to
>>>>>>>>>>>>>>>                             the no-gluster area w/ fewer disks).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                         Okay, then 1.6Gb/s is what we need to
>>>>>>>>>>>>>>>                         target
>>>>>>>>>>>>>>>                         for, considering your volume is just
>>>>>>>>>>>>>>>                         distribute. Is there any way you can do
>>>>>>>>>>>>>>>                         tests
>>>>>>>>>>>>>>>                         on similar hardware but at a small scale?
>>>>>>>>>>>>>>>                         Just so we can run the workload to learn
>>>>>>>>>>>>>>>                         more
>>>>>>>>>>>>>>>                         about the bottlenecks in the system? We
>>>>>>>>>>>>>>>                         can
>>>>>>>>>>>>>>>                         probably try to get the speed to 1.2Gb/s
>>>>>>>>>>>>>>>                         on
>>>>>>>>>>>>>>>                         your /home partition you were telling me
>>>>>>>>>>>>>>>                         yesterday. Let me know if that is
>>>>>>>>>>>>>>>                         something
>>>>>>>>>>>>>>>                         you are okay to do.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                             Pat
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                             On 05/10/2017 01:27 PM, Pranith Kumar
>>>>>>>>>>>>>>>                             Karampuri wrote:
>>>>>>>>>>>>>>>>                             On Wed, May 10, 2017 at 10:15 PM,
>>>>>>>>>>>>>>>>                             Pat
>>>>>>>>>>>>>>>>                             Haley <phaley at mit.edu
>>>>>>>>>>>>>>>>                             <mailto:phaley at mit.edu>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                 Hi Pranith,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                 Not entirely sure (this isn't my
>>>>>>>>>>>>>>>>                                 area of expertise). I'll run
>>>>>>>>>>>>>>>>                                 your
>>>>>>>>>>>>>>>>                                 answer by some other people who
>>>>>>>>>>>>>>>>                                 are
>>>>>>>>>>>>>>>>                                 more familiar with this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                 I am also uncertain about how to
>>>>>>>>>>>>>>>>                                 interpret the results when we
>>>>>>>>>>>>>>>>                                 also
>>>>>>>>>>>>>>>>                                 add the dd tests writing to the
>>>>>>>>>>>>>>>>                                 /home area (no gluster, still on
>>>>>>>>>>>>>>>>                                 the
>>>>>>>>>>>>>>>>                                 same machine)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                   * dd test without oflag=sync
>>>>>>>>>>>>>>>>                                     (rough average of multiple
>>>>>>>>>>>>>>>>                                     tests)
>>>>>>>>>>>>>>>>                                       o gluster w/ fuse mount :
>>>>>>>>>>>>>>>>                                       570
>>>>>>>>>>>>>>>>                                       Mb/s
>>>>>>>>>>>>>>>>                                       o gluster w/ nfs mount:
>>>>>>>>>>>>>>>>                                       390
>>>>>>>>>>>>>>>>                                       Mb/s
>>>>>>>>>>>>>>>>                                       o nfs (no gluster):  1.2
>>>>>>>>>>>>>>>>                                       Gb/s
>>>>>>>>>>>>>>>>                                   * dd test with oflag=sync
>>>>>>>>>>>>>>>>                                   (rough
>>>>>>>>>>>>>>>>                                     average of multiple tests)
>>>>>>>>>>>>>>>>                                       o gluster w/ fuse mount:
>>>>>>>>>>>>>>>>                                       5
>>>>>>>>>>>>>>>>                                       Mb/s
>>>>>>>>>>>>>>>>                                       o gluster w/ nfs mount:
>>>>>>>>>>>>>>>>                                       200
>>>>>>>>>>>>>>>>                                       Mb/s
>>>>>>>>>>>>>>>>                                       o nfs (no gluster): 20
>>>>>>>>>>>>>>>>                                       Mb/s
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                 Given that the non-gluster area
>>>>>>>>>>>>>>>>                                 is
>>>>>>>>>>>>>>>>                                 a
>>>>>>>>>>>>>>>>                                 RAID-6 of 4 disks while each
>>>>>>>>>>>>>>>>                                 brick
>>>>>>>>>>>>>>>>                                 of the gluster area is a RAID-6
>>>>>>>>>>>>>>>>                                 of
>>>>>>>>>>>>>>>>                                 32 disks, I would naively expect
>>>>>>>>>>>>>>>>                                 the
>>>>>>>>>>>>>>>>                                 writes to the gluster area to be
>>>>>>>>>>>>>>>>                                 roughly 8x faster than to the
>>>>>>>>>>>>>>>>                                 non-gluster.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                             I think a better test is to try and
>>>>>>>>>>>>>>>>                             write to a file using nfs without
>>>>>>>>>>>>>>>>                             any
>>>>>>>>>>>>>>>>                             gluster to a location that is not
>>>>>>>>>>>>>>>>                             inside
>>>>>>>>>>>>>>>>                             the brick but someother location
>>>>>>>>>>>>>>>>                             that
>>>>>>>>>>>>>>>>                             is
>>>>>>>>>>>>>>>>                             on same disk(s). If you are mounting
>>>>>>>>>>>>>>>>                             the
>>>>>>>>>>>>>>>>                             partition as the brick, then we can
>>>>>>>>>>>>>>>>                             write to a file inside .glusterfs
>>>>>>>>>>>>>>>>                             directory, something like
>>>>>>>>>>>>>>>>                             <brick-path>/.glusterfs/<file-to-be-removed-after-test>.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                 I still think we have a speed
>>>>>>>>>>>>>>>>                                 issue,
>>>>>>>>>>>>>>>>                                 I can't tell if fuse vs nfs is
>>>>>>>>>>>>>>>>                                 part
>>>>>>>>>>>>>>>>                                 of the problem.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                             I got interested in the post because
>>>>>>>>>>>>>>>>                             I
>>>>>>>>>>>>>>>>                             read that fuse speed is lesser than
>>>>>>>>>>>>>>>>                             nfs
>>>>>>>>>>>>>>>>                             speed which is counter-intuitive to
>>>>>>>>>>>>>>>>                             my
>>>>>>>>>>>>>>>>                             understanding. So wanted
>>>>>>>>>>>>>>>>                             clarifications.
>>>>>>>>>>>>>>>>                             Now that I got my clarifications
>>>>>>>>>>>>>>>>                             where
>>>>>>>>>>>>>>>>                             fuse outperformed nfs without sync,
>>>>>>>>>>>>>>>>                             we
>>>>>>>>>>>>>>>>                             can resume testing as described
>>>>>>>>>>>>>>>>                             above
>>>>>>>>>>>>>>>>                             and try to find what it is. Based on
>>>>>>>>>>>>>>>>                             your email-id I am guessing you are
>>>>>>>>>>>>>>>>                             from
>>>>>>>>>>>>>>>>                             Boston and I am from Bangalore so if
>>>>>>>>>>>>>>>>                             you
>>>>>>>>>>>>>>>>                             are okay with doing this debugging
>>>>>>>>>>>>>>>>                             for
>>>>>>>>>>>>>>>>                             multiple days because of timezones,
>>>>>>>>>>>>>>>>                             I
>>>>>>>>>>>>>>>>                             will be happy to help. Please be a
>>>>>>>>>>>>>>>>                             bit
>>>>>>>>>>>>>>>>                             patient with me, I am under a
>>>>>>>>>>>>>>>>                             release
>>>>>>>>>>>>>>>>                             crunch but I am very curious with
>>>>>>>>>>>>>>>>                             the
>>>>>>>>>>>>>>>>                             problem you posted.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                 Was there anything useful in the
>>>>>>>>>>>>>>>>                                 profiles?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                             Unfortunately profiles didn't help
>>>>>>>>>>>>>>>>                             me
>>>>>>>>>>>>>>>>                             much, I think we are collecting the
>>>>>>>>>>>>>>>>                             profiles from an active volume, so
>>>>>>>>>>>>>>>>                             it
>>>>>>>>>>>>>>>>                             has a lot of information that is not
>>>>>>>>>>>>>>>>                             pertaining to dd so it is difficult
>>>>>>>>>>>>>>>>                             to
>>>>>>>>>>>>>>>>                             find the contributions of dd. So I
>>>>>>>>>>>>>>>>                             went
>>>>>>>>>>>>>>>>                             through your post again and found
>>>>>>>>>>>>>>>>                             something I didn't pay much
>>>>>>>>>>>>>>>>                             attention
>>>>>>>>>>>>>>>>                             to
>>>>>>>>>>>>>>>>                             earlier i.e. oflag=sync, so did my
>>>>>>>>>>>>>>>>                             own
>>>>>>>>>>>>>>>>                             tests on my setup with FUSE so sent
>>>>>>>>>>>>>>>>                             that
>>>>>>>>>>>>>>>>                             reply.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                 Pat
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                 On 05/10/2017 12:15 PM, Pranith
>>>>>>>>>>>>>>>>                                 Kumar Karampuri wrote:
>>>>>>>>>>>>>>>>>                                 Okay good. At least this
>>>>>>>>>>>>>>>>>                                 validates
>>>>>>>>>>>>>>>>>                                 my doubts. Handling O_SYNC in
>>>>>>>>>>>>>>>>>                                 gluster NFS and fuse is a bit
>>>>>>>>>>>>>>>>>                                 different.
>>>>>>>>>>>>>>>>>                                 When application opens a file
>>>>>>>>>>>>>>>>>                                 with
>>>>>>>>>>>>>>>>>                                 O_SYNC on fuse mount then each
>>>>>>>>>>>>>>>>>                                 write syscall has to be written
>>>>>>>>>>>>>>>>>                                 to
>>>>>>>>>>>>>>>>>                                 disk as part of the syscall
>>>>>>>>>>>>>>>>>                                 where
>>>>>>>>>>>>>>>>>                                 as in case of NFS, there is no
>>>>>>>>>>>>>>>>>                                 concept of open. NFS performs
>>>>>>>>>>>>>>>>>                                 write
>>>>>>>>>>>>>>>>>                                 though a handle saying it needs
>>>>>>>>>>>>>>>>>                                 to
>>>>>>>>>>>>>>>>>                                 be a synchronous write, so
>>>>>>>>>>>>>>>>>                                 write()
>>>>>>>>>>>>>>>>>                                 syscall is performed first then
>>>>>>>>>>>>>>>>>                                 it
>>>>>>>>>>>>>>>>>                                 performs fsync(). so an write
>>>>>>>>>>>>>>>>>                                 on
>>>>>>>>>>>>>>>>>                                 an
>>>>>>>>>>>>>>>>>                                 fd with O_SYNC becomes
>>>>>>>>>>>>>>>>>                                 write+fsync.
>>>>>>>>>>>>>>>>>                                 I am suspecting that when
>>>>>>>>>>>>>>>>>                                 multiple
>>>>>>>>>>>>>>>>>                                 threads do this write+fsync()
>>>>>>>>>>>>>>>>>                                 operation on the same file,
>>>>>>>>>>>>>>>>>                                 multiple writes are batched
>>>>>>>>>>>>>>>>>                                 together to be written do disk
>>>>>>>>>>>>>>>>>                                 so
>>>>>>>>>>>>>>>>>                                 the throughput on the disk is
>>>>>>>>>>>>>>>>>                                 increasing is my guess.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                 Does it answer your doubts?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                 On Wed, May 10, 2017 at 9:35
>>>>>>>>>>>>>>>>>                                 PM,
>>>>>>>>>>>>>>>>>                                 Pat Haley <phaley at mit.edu
>>>>>>>>>>>>>>>>>                                 <mailto:phaley at mit.edu>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                     Without the oflag=sync and
>>>>>>>>>>>>>>>>>                                     only
>>>>>>>>>>>>>>>>>                                     a single test of each, the
>>>>>>>>>>>>>>>>>                                     FUSE
>>>>>>>>>>>>>>>>>                                     is going faster than NFS:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                     FUSE:
>>>>>>>>>>>>>>>>>                                     mseas-data2(dri_nascar)% dd
>>>>>>>>>>>>>>>>>                                     if=/dev/zero count=4096
>>>>>>>>>>>>>>>>>                                     bs=1048576 of=zeros.txt
>>>>>>>>>>>>>>>>>                                     conv=sync
>>>>>>>>>>>>>>>>>                                     4096+0 records in
>>>>>>>>>>>>>>>>>                                     4096+0 records out
>>>>>>>>>>>>>>>>>                                     4294967296 bytes (4.3 GB)
>>>>>>>>>>>>>>>>>                                     copied, 7.46961 s, 575 MB/s
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                     NFS
>>>>>>>>>>>>>>>>>                                     mseas-data2(HYCOM)% dd
>>>>>>>>>>>>>>>>>                                     if=/dev/zero count=4096
>>>>>>>>>>>>>>>>>                                     bs=1048576 of=zeros.txt
>>>>>>>>>>>>>>>>>                                     conv=sync
>>>>>>>>>>>>>>>>>                                     4096+0 records in
>>>>>>>>>>>>>>>>>                                     4096+0 records out
>>>>>>>>>>>>>>>>>                                     4294967296 bytes (4.3 GB)
>>>>>>>>>>>>>>>>>                                     copied, 11.4264 s, 376 MB/s
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                     On 05/10/2017 11:53 AM,
>>>>>>>>>>>>>>>>>                                     Pranith
>>>>>>>>>>>>>>>>>                                     Kumar Karampuri wrote:
>>>>>>>>>>>>>>>>>>                                     Could you let me know the
>>>>>>>>>>>>>>>>>>                                     speed without oflag=sync
>>>>>>>>>>>>>>>>>>                                     on
>>>>>>>>>>>>>>>>>>                                     both the mounts? No need
>>>>>>>>>>>>>>>>>>                                     to
>>>>>>>>>>>>>>>>>>                                     collect profiles.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                     On Wed, May 10, 2017 at
>>>>>>>>>>>>>>>>>>                                     9:17
>>>>>>>>>>>>>>>>>>                                     PM, Pat Haley
>>>>>>>>>>>>>>>>>>                                     <phaley at mit.edu
>>>>>>>>>>>>>>>>>>                                     <mailto:phaley at mit.edu>>
>>>>>>>>>>>>>>>>>>                                     wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                         Here is what I see
>>>>>>>>>>>>>>>>>>                                         now:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                         [root at mseas-data2 ~]#
>>>>>>>>>>>>>>>>>>                                         gluster volume info
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                         Volume Name:
>>>>>>>>>>>>>>>>>>                                         data-volume
>>>>>>>>>>>>>>>>>>                                         Type: Distribute
>>>>>>>>>>>>>>>>>>                                         Volume ID:
>>>>>>>>>>>>>>>>>>                                         c162161e-2a2d-4dac-b015-f31fd89ceb18
>>>>>>>>>>>>>>>>>>                                         Status: Started
>>>>>>>>>>>>>>>>>>                                         Number of Bricks: 2
>>>>>>>>>>>>>>>>>>                                         Transport-type: tcp
>>>>>>>>>>>>>>>>>>                                         Bricks:
>>>>>>>>>>>>>>>>>>                                         Brick1:
>>>>>>>>>>>>>>>>>>                                         mseas-data2:/mnt/brick1
>>>>>>>>>>>>>>>>>>                                         Brick2:
>>>>>>>>>>>>>>>>>>                                         mseas-data2:/mnt/brick2
>>>>>>>>>>>>>>>>>>                                         Options Reconfigured:
>>>>>>>>>>>>>>>>>>                                         diagnostics.count-fop-hits:
>>>>>>>>>>>>>>>>>>                                         on
>>>>>>>>>>>>>>>>>>                                         diagnostics.latency-measurement:
>>>>>>>>>>>>>>>>>>                                         on
>>>>>>>>>>>>>>>>>>                                         nfs.exports-auth-enable:
>>>>>>>>>>>>>>>>>>                                         on
>>>>>>>>>>>>>>>>>>                                         diagnostics.brick-sys-log-level:
>>>>>>>>>>>>>>>>>>                                         WARNING
>>>>>>>>>>>>>>>>>>                                         performance.readdir-ahead:
>>>>>>>>>>>>>>>>>>                                         on
>>>>>>>>>>>>>>>>>>                                         nfs.disable: on
>>>>>>>>>>>>>>>>>>                                         nfs.export-volumes:
>>>>>>>>>>>>>>>>>>                                         off
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                         On 05/10/2017 11:44
>>>>>>>>>>>>>>>>>>                                         AM,
>>>>>>>>>>>>>>>>>>                                         Pranith Kumar
>>>>>>>>>>>>>>>>>>                                         Karampuri
>>>>>>>>>>>>>>>>>>                                         wrote:
>>>>>>>>>>>>>>>>>>>                                         Is this the volume
>>>>>>>>>>>>>>>>>>>                                         info
>>>>>>>>>>>>>>>>>>>                                         you have?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>                                         >/[root at
>>>>>>>>>>>>>>>>>>>                                         >mseas-data2
>>>>>>>>>>>>>>>>>>>                                         <http://www.gluster.org/mailman/listinfo/gluster-users>
>>>>>>>>>>>>>>>>>>>                                         ~]# gluster volume
>>>>>>>>>>>>>>>>>>>                                         info
>>>>>>>>>>>>>>>>>>>                                         />//>/Volume Name:
>>>>>>>>>>>>>>>>>>>                                         data-volume />/Type:
>>>>>>>>>>>>>>>>>>>                                         Distribute />/Volume
>>>>>>>>>>>>>>>>>>>                                         ID:
>>>>>>>>>>>>>>>>>>>                                         c162161e-2a2d-4dac-b015-f31fd89ceb18
>>>>>>>>>>>>>>>>>>>                                         />/Status: Started
>>>>>>>>>>>>>>>>>>>                                         />/Number
>>>>>>>>>>>>>>>>>>>                                         of Bricks: 2
>>>>>>>>>>>>>>>>>>>                                         />/Transport-type:
>>>>>>>>>>>>>>>>>>>                                         tcp
>>>>>>>>>>>>>>>>>>>                                         />/Bricks: />/Brick1:
>>>>>>>>>>>>>>>>>>>                                         mseas-data2:/mnt/brick1
>>>>>>>>>>>>>>>>>>>                                         />/Brick2:
>>>>>>>>>>>>>>>>>>>                                         mseas-data2:/mnt/brick2
>>>>>>>>>>>>>>>>>>>                                         />/Options
>>>>>>>>>>>>>>>>>>>                                         Reconfigured:
>>>>>>>>>>>>>>>>>>>                                         />/performance.readdir-ahead:
>>>>>>>>>>>>>>>>>>>                                         on />/nfs.disable: on
>>>>>>>>>>>>>>>>>>>                                         />/nfs.export-volumes:
>>>>>>>>>>>>>>>>>>>                                         off
>>>>>>>>>>>>>>>>>>>                                         /
>>>>>>>>>>>>>>>>>>>                                         ​I copied this from
>>>>>>>>>>>>>>>>>>>                                         old
>>>>>>>>>>>>>>>>>>>                                         thread from 2016.
>>>>>>>>>>>>>>>>>>>                                         This
>>>>>>>>>>>>>>>>>>>                                         is
>>>>>>>>>>>>>>>>>>>                                         distribute volume.
>>>>>>>>>>>>>>>>>>>                                         Did
>>>>>>>>>>>>>>>>>>>                                         you change any of the
>>>>>>>>>>>>>>>>>>>                                         options in between?
>>>>>>>>>>>>>>>>>>                                         --
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                         -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>>>>>>>>>>                                         Pat Haley
>>>>>>>>>>>>>>>>>>                                         Email:phaley at mit.edu
>>>>>>>>>>>>>>>>>>                                         <mailto:phaley at mit.edu>
>>>>>>>>>>>>>>>>>>                                         Center for Ocean
>>>>>>>>>>>>>>>>>>                                         Engineering
>>>>>>>>>>>>>>>>>>                                         Phone:  (617) 253-6824
>>>>>>>>>>>>>>>>>>                                         Dept. of Mechanical
>>>>>>>>>>>>>>>>>>                                         Engineering
>>>>>>>>>>>>>>>>>>                                         Fax:    (617) 253-8125
>>>>>>>>>>>>>>>>>>                                         MIT, Room
>>>>>>>>>>>>>>>>>>                                         5-213http://web.mit.edu/phaley/www/
>>>>>>>>>>>>>>>>>>                                         77 Massachusetts
>>>>>>>>>>>>>>>>>>                                         Avenue
>>>>>>>>>>>>>>>>>>                                         Cambridge, MA
>>>>>>>>>>>>>>>>>>                                         02139-4301
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>                                     --
>>>>>>>>>>>>>>>>>>                                     Pranith
>>>>>>>>>>>>>>>>>                                     --
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                     -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>>>>>>>>>                                     Pat Haley
>>>>>>>>>>>>>>>>>                                     Email:phaley at mit.edu
>>>>>>>>>>>>>>>>>                                     <mailto:phaley at mit.edu>
>>>>>>>>>>>>>>>>>                                     Center for Ocean
>>>>>>>>>>>>>>>>>                                     Engineering
>>>>>>>>>>>>>>>>>                                     Phone:  (617) 253-6824
>>>>>>>>>>>>>>>>>                                     Dept. of Mechanical
>>>>>>>>>>>>>>>>>                                     Engineering
>>>>>>>>>>>>>>>>>                                     Fax:    (617) 253-8125
>>>>>>>>>>>>>>>>>                                     MIT, Room
>>>>>>>>>>>>>>>>>                                     5-213http://web.mit.edu/phaley/www/
>>>>>>>>>>>>>>>>>                                     77 Massachusetts Avenue
>>>>>>>>>>>>>>>>>                                     Cambridge, MA  02139-4301
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>                                 --
>>>>>>>>>>>>>>>>>                                 Pranith
>>>>>>>>>>>>>>>>                                 --
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                                 -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>>>>>>>>                                 Pat Haley
>>>>>>>>>>>>>>>>                                 Email:phaley at mit.edu
>>>>>>>>>>>>>>>>                                 <mailto:phaley at mit.edu>
>>>>>>>>>>>>>>>>                                 Center for Ocean Engineering
>>>>>>>>>>>>>>>>                                 Phone:
>>>>>>>>>>>>>>>>                                 (617) 253-6824
>>>>>>>>>>>>>>>>                                 Dept. of Mechanical Engineering
>>>>>>>>>>>>>>>>                                 Fax:
>>>>>>>>>>>>>>>>                                 (617) 253-8125
>>>>>>>>>>>>>>>>                                 MIT, Room
>>>>>>>>>>>>>>>>                                 5-213http://web.mit.edu/phaley/www/
>>>>>>>>>>>>>>>>                                 77 Massachusetts Avenue
>>>>>>>>>>>>>>>>                                 Cambridge, MA  02139-4301
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>                             --
>>>>>>>>>>>>>>>>                             Pranith
>>>>>>>>>>>>>>>                             --
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                             -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>>>>>>>                             Pat Haley
>>>>>>>>>>>>>>>                             Email:phaley at mit.edu
>>>>>>>>>>>>>>>                             <mailto:phaley at mit.edu>
>>>>>>>>>>>>>>>                             Center for Ocean Engineering
>>>>>>>>>>>>>>>                             Phone:
>>>>>>>>>>>>>>>                             (617) 253-6824
>>>>>>>>>>>>>>>                             Dept. of Mechanical Engineering
>>>>>>>>>>>>>>>                             Fax:
>>>>>>>>>>>>>>>                             (617) 253-8125
>>>>>>>>>>>>>>>                             MIT, Room
>>>>>>>>>>>>>>>                             5-213http://web.mit.edu/phaley/www/
>>>>>>>>>>>>>>>                             77 Massachusetts Avenue
>>>>>>>>>>>>>>>                             Cambridge, MA  02139-4301
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                         --
>>>>>>>>>>>>>>>                         Pranith
>>>>>>>>>>>>>>                         --
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                         -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>>>>>>                         Pat Haley
>>>>>>>>>>>>>>                         Email:phaley at mit.edu
>>>>>>>>>>>>>>                         <mailto:phaley at mit.edu>
>>>>>>>>>>>>>>                         Center for Ocean Engineering       Phone:
>>>>>>>>>>>>>>                         (617)
>>>>>>>>>>>>>>                         253-6824
>>>>>>>>>>>>>>                         Dept. of Mechanical Engineering    Fax:
>>>>>>>>>>>>>>                         (617)
>>>>>>>>>>>>>>                         253-8125
>>>>>>>>>>>>>>                         MIT, Room
>>>>>>>>>>>>>>                         5-213http://web.mit.edu/phaley/www/
>>>>>>>>>>>>>>                         77 Massachusetts Avenue
>>>>>>>>>>>>>>                         Cambridge, MA  02139-4301
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                     --
>>>>>>>>>>>>>>                     Pranith
>>>>>>>>>>>>>                     --
>>>>>>>>>>>>>
>>>>>>>>>>>>>                     -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>>>>>                     Pat Haley
>>>>>>>>>>>>>                     Email:phaley at mit.edu
>>>>>>>>>>>>>                     <mailto:phaley at mit.edu>
>>>>>>>>>>>>>                     Center for Ocean Engineering       Phone:
>>>>>>>>>>>>>                     (617)
>>>>>>>>>>>>>                     253-6824
>>>>>>>>>>>>>                     Dept. of Mechanical Engineering    Fax:
>>>>>>>>>>>>>                     (617)
>>>>>>>>>>>>>                     253-8125
>>>>>>>>>>>>>                     MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>>>>>>>>>>                     77 Massachusetts Avenue
>>>>>>>>>>>>>                     Cambridge, MA  02139-4301
>>>>>>>>>>>>>
>>>>>>>>>>>>>                 --
>>>>>>>>>>>>>                 Pranith
>>>>>>>>>>>>                 --
>>>>>>>>>>>>
>>>>>>>>>>>>                 -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>>>>                 Pat Haley
>>>>>>>>>>>>                 Email:phaley at mit.edu
>>>>>>>>>>>>                 <mailto:phaley at mit.edu>
>>>>>>>>>>>>                 Center for Ocean Engineering       Phone:  (617)
>>>>>>>>>>>>                 253-6824
>>>>>>>>>>>>                 Dept. of Mechanical Engineering    Fax:    (617)
>>>>>>>>>>>>                 253-8125
>>>>>>>>>>>>                 MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>>>>>>>>>                 77 Massachusetts Avenue
>>>>>>>>>>>>                 Cambridge, MA  02139-4301
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>             --
>>>>>>>>>>>>             Pranith
>>>>>>>>>>>             --
>>>>>>>>>>>
>>>>>>>>>>>             -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>>>             Pat HaleyEmail:phaley at mit.edu
>>>>>>>>>>>             <mailto:phaley at mit.edu>
>>>>>>>>>>>             Center for Ocean Engineering       Phone:  (617) 253-6824
>>>>>>>>>>>             Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>>>>>>>>>>             MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>>>>>>>>             77 Massachusetts Avenue
>>>>>>>>>>>             Cambridge, MA  02139-4301
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>         --
>>>>>>>>>>>         Pranith
>>>>>>>>>>         --
>>>>>>>>>>
>>>>>>>>>>         -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>>         Pat HaleyEmail:phaley at mit.edu
>>>>>>>>>>         <mailto:phaley at mit.edu>
>>>>>>>>>>         Center for Ocean Engineering       Phone:  (617) 253-6824
>>>>>>>>>>         Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>>>>>>>>>         MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>>>>>>>         77 Massachusetts Avenue
>>>>>>>>>>         Cambridge, MA  02139-4301
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Pranith
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>> Pat Haley                          Email:phaley at mit.edu
>>>>>>>>> Center for Ocean Engineering       Phone:  (617) 253-6824
>>>>>>>>> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>>>>>>>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>>>>>> 77 Massachusetts Avenue
>>>>>>>>> Cambridge, MA  02139-4301
>>>>>>>>>
>>>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>> Pat Haley                          Email:phaley at mit.edu
>>>>>>> Center for Ocean Engineering       Phone:  (617) 253-6824
>>>>>>> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>>>>>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>>>> 77 Massachusetts Avenue
>>>>>>> Cambridge, MA  02139-4301
>>>>>>>
>>>>>>>
>>>>> --
>>>>>
>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>> Pat Haley                          Email:phaley at mit.edu
>>>>> Center for Ocean Engineering       Phone:  (617) 253-6824
>>>>> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>>>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>> 77 Massachusetts Avenue
>>>>> Cambridge, MA  02139-4301
>>>>>
>>>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley                          Email:phaley at mit.edu
>>> Center for Ocean Engineering       Phone:  (617) 253-6824
>>> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA  02139-4301
>>>
>>>
>
> -- 
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley                          Email:phaley at mit.edu
> Center for Ocean Engineering       Phone:  (617) 253-6824
> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
> MIT, Room 5-213http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA  02139-4301
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users

-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170622/7bc75870/attachment.html>


More information about the Gluster-users mailing list