[Gluster-users] Slow write times to gluster disk
Pat Haley
phaley at mit.edu
Tue Jun 20 16:06:30 UTC 2017
Hi Ben,
Sorry this took so long, but we had a real-time forecasting exercise
last week and I could only get to this now.
Backend Hardware/OS:
* Much of the information on our back end system is included at the
top of
http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html
* The specific model of the hard disks is SeaGate ENTERPRISE CAPACITY
V.4 6TB (ST6000NM0024). The rated speed is 6Gb/s.
* Note: there is one physical server that hosts both the NFS and the
GlusterFS areas
Latest tests
I have had time to run the tests for one of the dd tests you requested
to the underlying XFS FS. The median rate was 170 MB/s. The dd results
and iostat record are in
http://mseas.mit.edu/download/phaley/GlusterUsers/TestXFS/
I'll add tests for the other brick and to the NFS area later.
Thanks
Pat
On 06/12/2017 06:06 PM, Ben Turner wrote:
> Ok you are correct, you have a pure distributed volume. IE no replication overhead. So normally for pure dist I use:
>
> throughput = slowest of disks / NIC * .6-.7
>
> In your case we have:
>
> 1200 * .6 = 720
>
> So you are seeing a little less throughput than I would expect in your configuration. What I like to do here is:
>
> -First tell me more about your back end storage, will it sustain 1200 MB / sec? What kind of HW? How many disks? What type and specs are the disks? What kind of RAID are you using?
>
> -Second can you refresh me on your workload? Are you doing reads / writes or both? If both what mix? Since we are using DD I assume you are working iwth large file sequential I/O, is this correct?
>
> -Run some DD tests on the back end XFS FS. I normally have /xfs-mount/gluster-brick, if you have something similar just mkdir on the XFS -> /xfs-mount/my-test-dir. Inside the test dir run:
>
> If you are focusing on a write workload run:
>
> # dd if=/dev/zero of=/xfs-mount/file bs=1024k count=10000 conv=fdatasync
>
> If you are focusing on a read workload run:
>
> # echo 3 > /proc/sys/vm/drop_caches
> # dd if=/gluster-mount/file of=/dev/null bs=1024k count=10000
>
> ** MAKE SURE TO DROP CACHE IN BETWEEN READS!! **
>
> Run this in a loop similar to how you did in:
>
> http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/dd_testvol_gluster.txt
>
> Run this on both servers one at a time and if you are running on a SAN then run again on both at the same time. While this is running gather iostat for me:
>
> # iostat -c -m -x 1 > iostat-$(hostname).txt
>
> Lets see how the back end performs on both servers while capturing iostat, then see how the same workload / data looks on gluster.
>
> -Last thing, when you run your kernel NFS tests are you using the same filesystem / storage you are using for the gluster bricks? I want to be sure we have an apples to apples comparison here.
>
> -b
>
>
>
> ----- Original Message -----
>> From: "Pat Haley" <phaley at mit.edu>
>> To: "Ben Turner" <bturner at redhat.com>
>> Sent: Monday, June 12, 2017 5:18:07 PM
>> Subject: Re: [Gluster-users] Slow write times to gluster disk
>>
>>
>> Hi Ben,
>>
>> Here is the output:
>>
>> [root at mseas-data2 ~]# gluster volume info
>>
>> Volume Name: data-volume
>> Type: Distribute
>> Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
>> Status: Started
>> Number of Bricks: 2
>> Transport-type: tcp
>> Bricks:
>> Brick1: mseas-data2:/mnt/brick1
>> Brick2: mseas-data2:/mnt/brick2
>> Options Reconfigured:
>> nfs.exports-auth-enable: on
>> diagnostics.brick-sys-log-level: WARNING
>> performance.readdir-ahead: on
>> nfs.disable: on
>> nfs.export-volumes: off
>>
>>
>> On 06/12/2017 05:01 PM, Ben Turner wrote:
>>> What is the output of gluster v info? That will tell us more about your
>>> config.
>>>
>>> -b
>>>
>>> ----- Original Message -----
>>>> From: "Pat Haley" <phaley at mit.edu>
>>>> To: "Ben Turner" <bturner at redhat.com>
>>>> Sent: Monday, June 12, 2017 4:54:00 PM
>>>> Subject: Re: [Gluster-users] Slow write times to gluster disk
>>>>
>>>>
>>>> Hi Ben,
>>>>
>>>> I guess I'm confused about what you mean by replication. If I look at
>>>> the underlying bricks I only ever have a single copy of any file. It
>>>> either resides on one brick or the other (directories exist on both
>>>> bricks but not files). We are not using gluster for redundancy (or at
>>>> least that wasn't our intent). Is that what you meant by replication
>>>> or is it something else?
>>>>
>>>> Thanks
>>>>
>>>> Pat
>>>>
>>>> On 06/12/2017 04:28 PM, Ben Turner wrote:
>>>>> ----- Original Message -----
>>>>>> From: "Pat Haley" <phaley at mit.edu>
>>>>>> To: "Ben Turner" <bturner at redhat.com>, "Pranith Kumar Karampuri"
>>>>>> <pkarampu at redhat.com>
>>>>>> Cc: "Ravishankar N" <ravishankar at redhat.com>, gluster-users at gluster.org,
>>>>>> "Steve Postma" <SPostma at ztechnet.com>
>>>>>> Sent: Monday, June 12, 2017 2:35:41 PM
>>>>>> Subject: Re: [Gluster-users] Slow write times to gluster disk
>>>>>>
>>>>>>
>>>>>> Hi Guys,
>>>>>>
>>>>>> I was wondering what our next steps should be to solve the slow write
>>>>>> times.
>>>>>>
>>>>>> Recently I was debugging a large code and writing a lot of output at
>>>>>> every time step. When I tried writing to our gluster disks, it was
>>>>>> taking over a day to do a single time step whereas if I had the same
>>>>>> program (same hardware, network) write to our nfs disk the time per
>>>>>> time-step was about 45 minutes. What we are shooting for here would be
>>>>>> to have similar times to either gluster of nfs.
>>>>> I can see in your test:
>>>>>
>>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/dd_testvol_gluster.txt
>>>>>
>>>>> You averaged ~600 MB / sec(expected for replica 2 with 10G, {~1200 MB /
>>>>> sec} / #replicas{2} = 600). Gluster does client side replication so with
>>>>> replica 2 you will only ever see 1/2 the speed of your slowest part of
>>>>> the
>>>>> stack(NW, disk, RAM, CPU). This is usually NW or disk and 600 is
>>>>> normally
>>>>> a best case. Now in your output I do see the instances where you went
>>>>> down to 200 MB / sec. I can only explain this in three ways:
>>>>>
>>>>> 1. You are not using conv=fdatasync and writes are actually going to
>>>>> page
>>>>> cache and then being flushed to disk. During the fsync the memory is not
>>>>> yet available and the disks are busy flushing dirty pages.
>>>>> 2. Your storage RAID group is shared across multiple LUNS(like in a SAN)
>>>>> and when write times are slow the RAID group is busy serviceing other
>>>>> LUNs.
>>>>> 3. Gluster bug / config issue / some other unknown unknown.
>>>>>
>>>>> So I see 2 issues here:
>>>>>
>>>>> 1. NFS does in 45 minutes what gluster can do in 24 hours.
>>>>> 2. Sometimes your throughput drops dramatically.
>>>>>
>>>>> WRT #1 - have a look at my estimates above. My formula for guestimating
>>>>> gluster perf is: throughput = NIC throughput or storage(whatever is
>>>>> slower) / # replicas * overhead(figure .7 or .8). Also the larger the
>>>>> record size the better for glusterfs mounts, I normally like to be at
>>>>> LEAST 64k up to 1024k:
>>>>>
>>>>> # dd if=/dev/zero of=/gluster-mount/file bs=1024k count=10000
>>>>> conv=fdatasync
>>>>>
>>>>> WRT #2 - Again, I question your testing and your storage config. Try
>>>>> using
>>>>> conv=fdatasync for your DDs, use a larger record size, and make sure that
>>>>> your back end storage is not causing your slowdowns. Also remember that
>>>>> with replica 2 you will take ~50% hit on writes because the client uses
>>>>> 50% of its bandwidth to write to one replica and 50% to the other.
>>>>>
>>>>> -b
>>>>>
>>>>>
>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Pat
>>>>>>
>>>>>>
>>>>>> On 06/02/2017 01:07 AM, Ben Turner wrote:
>>>>>>> Are you sure using conv=sync is what you want? I normally use
>>>>>>> conv=fdatasync, I'll look up the difference between the two and see if
>>>>>>> it
>>>>>>> affects your test.
>>>>>>>
>>>>>>>
>>>>>>> -b
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>>> From: "Pat Haley" <phaley at mit.edu>
>>>>>>>> To: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
>>>>>>>> Cc: "Ravishankar N" <ravishankar at redhat.com>,
>>>>>>>> gluster-users at gluster.org,
>>>>>>>> "Steve Postma" <SPostma at ztechnet.com>, "Ben
>>>>>>>> Turner" <bturner at redhat.com>
>>>>>>>> Sent: Tuesday, May 30, 2017 9:40:34 PM
>>>>>>>> Subject: Re: [Gluster-users] Slow write times to gluster disk
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Pranith,
>>>>>>>>
>>>>>>>> The "dd" command was:
>>>>>>>>
>>>>>>>> dd if=/dev/zero count=4096 bs=1048576 of=zeros.txt conv=sync
>>>>>>>>
>>>>>>>> There were 2 instances where dd reported 22 seconds. The output from
>>>>>>>> the
>>>>>>>> dd tests are in
>>>>>>>>
>>>>>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/dd_testvol_gluster.txt
>>>>>>>>
>>>>>>>> Pat
>>>>>>>>
>>>>>>>> On 05/30/2017 09:27 PM, Pranith Kumar Karampuri wrote:
>>>>>>>>> Pat,
>>>>>>>>> What is the command you used? As per the following output,
>>>>>>>>> it
>>>>>>>>> seems like at least one write operation took 16 seconds. Which is
>>>>>>>>> really bad.
>>>>>>>>> 96.39 1165.10 us 89.00 us*16487014.00 us*
>>>>>>>>> 393212
>>>>>>>>> WRITE
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, May 30, 2017 at 10:36 PM, Pat Haley <phaley at mit.edu
>>>>>>>>> <mailto:phaley at mit.edu>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Pranith,
>>>>>>>>>
>>>>>>>>> I ran the same 'dd' test both in the gluster test volume and
>>>>>>>>> in
>>>>>>>>> the .glusterfs directory of each brick. The median results
>>>>>>>>> (12
>>>>>>>>> dd
>>>>>>>>> trials in each test) are similar to before
>>>>>>>>>
>>>>>>>>> * gluster test volume: 586.5 MB/s
>>>>>>>>> * bricks (in .glusterfs): 1.4 GB/s
>>>>>>>>>
>>>>>>>>> The profile for the gluster test-volume is in
>>>>>>>>>
>>>>>>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/profile_testvol_gluster.txt
>>>>>>>>> <http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/profile_testvol_gluster.txt>
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Pat
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 05/30/2017 12:10 PM, Pranith Kumar Karampuri wrote:
>>>>>>>>>> Let's start with the same 'dd' test we were testing with to
>>>>>>>>>> see,
>>>>>>>>>> what the numbers are. Please provide profile numbers for the
>>>>>>>>>> same. From there on we will start tuning the volume to see
>>>>>>>>>> what
>>>>>>>>>> we can do.
>>>>>>>>>>
>>>>>>>>>> On Tue, May 30, 2017 at 9:16 PM, Pat Haley <phaley at mit.edu
>>>>>>>>>> <mailto:phaley at mit.edu>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Pranith,
>>>>>>>>>>
>>>>>>>>>> Thanks for the tip. We now have the gluster volume
>>>>>>>>>> mounted
>>>>>>>>>> under /home. What tests do you recommend we run?
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> Pat
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 05/17/2017 05:01 AM, Pranith Kumar Karampuri wrote:
>>>>>>>>>>> On Tue, May 16, 2017 at 9:20 PM, Pat Haley
>>>>>>>>>>> <phaley at mit.edu
>>>>>>>>>>> <mailto:phaley at mit.edu>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi Pranith,
>>>>>>>>>>>
>>>>>>>>>>> Sorry for the delay. I never saw received your
>>>>>>>>>>> reply
>>>>>>>>>>> (but I did receive Ben Turner's follow-up to your
>>>>>>>>>>> reply). So we tried to create a gluster volume
>>>>>>>>>>> under
>>>>>>>>>>> /home using different variations of
>>>>>>>>>>>
>>>>>>>>>>> gluster volume create test-volume
>>>>>>>>>>> mseas-data2:/home/gbrick_test_1
>>>>>>>>>>> mseas-data2:/home/gbrick_test_2 transport tcp
>>>>>>>>>>>
>>>>>>>>>>> However we keep getting errors of the form
>>>>>>>>>>>
>>>>>>>>>>> Wrong brick type: transport, use
>>>>>>>>>>> <HOSTNAME>:<export-dir-abs-path>
>>>>>>>>>>>
>>>>>>>>>>> Any thoughts on what we're doing wrong?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> You should give transport tcp at the beginning I think.
>>>>>>>>>>> Anyways, transport tcp is the default, so no need to
>>>>>>>>>>> specify
>>>>>>>>>>> so remove those two words from the CLI.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Also do you have a list of the test we should be
>>>>>>>>>>> running
>>>>>>>>>>> once we get this volume created? Given the
>>>>>>>>>>> time-zone
>>>>>>>>>>> difference it might help if we can run a small
>>>>>>>>>>> battery
>>>>>>>>>>> of tests and post the results rather than
>>>>>>>>>>> test-post-new
>>>>>>>>>>> test-post... .
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This is the first time I am doing performance analysis
>>>>>>>>>>> on
>>>>>>>>>>> users as far as I remember. In our team there are
>>>>>>>>>>> separate
>>>>>>>>>>> engineers who do these tests. Ben who replied earlier is
>>>>>>>>>>> one
>>>>>>>>>>> such engineer.
>>>>>>>>>>>
>>>>>>>>>>> Ben,
>>>>>>>>>>> Have any suggestions?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> Pat
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 05/11/2017 12:06 PM, Pranith Kumar Karampuri
>>>>>>>>>>> wrote:
>>>>>>>>>>>> On Thu, May 11, 2017 at 9:32 PM, Pat Haley
>>>>>>>>>>>> <phaley at mit.edu <mailto:phaley at mit.edu>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Pranith,
>>>>>>>>>>>>
>>>>>>>>>>>> The /home partition is mounted as ext4
>>>>>>>>>>>> /home ext4 defaults,usrquota,grpquota 1 2
>>>>>>>>>>>>
>>>>>>>>>>>> The brick partitions are mounted ax xfs
>>>>>>>>>>>> /mnt/brick1 xfs defaults 0 0
>>>>>>>>>>>> /mnt/brick2 xfs defaults 0 0
>>>>>>>>>>>>
>>>>>>>>>>>> Will this cause a problem with creating a
>>>>>>>>>>>> volume
>>>>>>>>>>>> under /home?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I don't think the bottleneck is disk. You can do
>>>>>>>>>>>> the
>>>>>>>>>>>> same tests you did on your new volume to confirm?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Pat
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 05/11/2017 11:32 AM, Pranith Kumar Karampuri
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> On Thu, May 11, 2017 at 8:57 PM, Pat Haley
>>>>>>>>>>>>> <phaley at mit.edu <mailto:phaley at mit.edu>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Pranith,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Unfortunately, we don't have similar
>>>>>>>>>>>>> hardware
>>>>>>>>>>>>> for a small scale test. All we have is
>>>>>>>>>>>>> our
>>>>>>>>>>>>> production hardware.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> You said something about /home partition which
>>>>>>>>>>>>> has
>>>>>>>>>>>>> lesser disks, we can create plain distribute
>>>>>>>>>>>>> volume inside one of those directories. After
>>>>>>>>>>>>> we
>>>>>>>>>>>>> are done, we can remove the setup. What do you
>>>>>>>>>>>>> say?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Pat
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 05/11/2017 07:05 AM, Pranith Kumar
>>>>>>>>>>>>> Karampuri wrote:
>>>>>>>>>>>>>> On Thu, May 11, 2017 at 2:48 AM, Pat
>>>>>>>>>>>>>> Haley
>>>>>>>>>>>>>> <phaley at mit.edu <mailto:phaley at mit.edu>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Pranith,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Since we are mounting the partitions
>>>>>>>>>>>>>> as
>>>>>>>>>>>>>> the bricks, I tried the dd test
>>>>>>>>>>>>>> writing
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> <brick-path>/.glusterfs/<file-to-be-removed-after-test>.
>>>>>>>>>>>>>> The results without oflag=sync were
>>>>>>>>>>>>>> 1.6
>>>>>>>>>>>>>> Gb/s (faster than gluster but not as
>>>>>>>>>>>>>> fast
>>>>>>>>>>>>>> as I was expecting given the 1.2 Gb/s
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> the no-gluster area w/ fewer disks).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Okay, then 1.6Gb/s is what we need to
>>>>>>>>>>>>>> target
>>>>>>>>>>>>>> for, considering your volume is just
>>>>>>>>>>>>>> distribute. Is there any way you can do
>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>> on similar hardware but at a small scale?
>>>>>>>>>>>>>> Just so we can run the workload to learn
>>>>>>>>>>>>>> more
>>>>>>>>>>>>>> about the bottlenecks in the system? We
>>>>>>>>>>>>>> can
>>>>>>>>>>>>>> probably try to get the speed to 1.2Gb/s
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>> your /home partition you were telling me
>>>>>>>>>>>>>> yesterday. Let me know if that is
>>>>>>>>>>>>>> something
>>>>>>>>>>>>>> you are okay to do.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Pat
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 05/10/2017 01:27 PM, Pranith Kumar
>>>>>>>>>>>>>> Karampuri wrote:
>>>>>>>>>>>>>>> On Wed, May 10, 2017 at 10:15 PM,
>>>>>>>>>>>>>>> Pat
>>>>>>>>>>>>>>> Haley <phaley at mit.edu
>>>>>>>>>>>>>>> <mailto:phaley at mit.edu>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Pranith,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Not entirely sure (this isn't my
>>>>>>>>>>>>>>> area of expertise). I'll run
>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>> answer by some other people who
>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>> more familiar with this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am also uncertain about how to
>>>>>>>>>>>>>>> interpret the results when we
>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>> add the dd tests writing to the
>>>>>>>>>>>>>>> /home area (no gluster, still on
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> same machine)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> * dd test without oflag=sync
>>>>>>>>>>>>>>> (rough average of multiple
>>>>>>>>>>>>>>> tests)
>>>>>>>>>>>>>>> o gluster w/ fuse mount :
>>>>>>>>>>>>>>> 570
>>>>>>>>>>>>>>> Mb/s
>>>>>>>>>>>>>>> o gluster w/ nfs mount:
>>>>>>>>>>>>>>> 390
>>>>>>>>>>>>>>> Mb/s
>>>>>>>>>>>>>>> o nfs (no gluster): 1.2
>>>>>>>>>>>>>>> Gb/s
>>>>>>>>>>>>>>> * dd test with oflag=sync
>>>>>>>>>>>>>>> (rough
>>>>>>>>>>>>>>> average of multiple tests)
>>>>>>>>>>>>>>> o gluster w/ fuse mount:
>>>>>>>>>>>>>>> 5
>>>>>>>>>>>>>>> Mb/s
>>>>>>>>>>>>>>> o gluster w/ nfs mount:
>>>>>>>>>>>>>>> 200
>>>>>>>>>>>>>>> Mb/s
>>>>>>>>>>>>>>> o nfs (no gluster): 20
>>>>>>>>>>>>>>> Mb/s
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Given that the non-gluster area
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> RAID-6 of 4 disks while each
>>>>>>>>>>>>>>> brick
>>>>>>>>>>>>>>> of the gluster area is a RAID-6
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> 32 disks, I would naively expect
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> writes to the gluster area to be
>>>>>>>>>>>>>>> roughly 8x faster than to the
>>>>>>>>>>>>>>> non-gluster.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think a better test is to try and
>>>>>>>>>>>>>>> write to a file using nfs without
>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>> gluster to a location that is not
>>>>>>>>>>>>>>> inside
>>>>>>>>>>>>>>> the brick but someother location
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> on same disk(s). If you are mounting
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> partition as the brick, then we can
>>>>>>>>>>>>>>> write to a file inside .glusterfs
>>>>>>>>>>>>>>> directory, something like
>>>>>>>>>>>>>>> <brick-path>/.glusterfs/<file-to-be-removed-after-test>.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I still think we have a speed
>>>>>>>>>>>>>>> issue,
>>>>>>>>>>>>>>> I can't tell if fuse vs nfs is
>>>>>>>>>>>>>>> part
>>>>>>>>>>>>>>> of the problem.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I got interested in the post because
>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>> read that fuse speed is lesser than
>>>>>>>>>>>>>>> nfs
>>>>>>>>>>>>>>> speed which is counter-intuitive to
>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>> understanding. So wanted
>>>>>>>>>>>>>>> clarifications.
>>>>>>>>>>>>>>> Now that I got my clarifications
>>>>>>>>>>>>>>> where
>>>>>>>>>>>>>>> fuse outperformed nfs without sync,
>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>> can resume testing as described
>>>>>>>>>>>>>>> above
>>>>>>>>>>>>>>> and try to find what it is. Based on
>>>>>>>>>>>>>>> your email-id I am guessing you are
>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>> Boston and I am from Bangalore so if
>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> are okay with doing this debugging
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> multiple days because of timezones,
>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>> will be happy to help. Please be a
>>>>>>>>>>>>>>> bit
>>>>>>>>>>>>>>> patient with me, I am under a
>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>> crunch but I am very curious with
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> problem you posted.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Was there anything useful in the
>>>>>>>>>>>>>>> profiles?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Unfortunately profiles didn't help
>>>>>>>>>>>>>>> me
>>>>>>>>>>>>>>> much, I think we are collecting the
>>>>>>>>>>>>>>> profiles from an active volume, so
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> has a lot of information that is not
>>>>>>>>>>>>>>> pertaining to dd so it is difficult
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> find the contributions of dd. So I
>>>>>>>>>>>>>>> went
>>>>>>>>>>>>>>> through your post again and found
>>>>>>>>>>>>>>> something I didn't pay much
>>>>>>>>>>>>>>> attention
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> earlier i.e. oflag=sync, so did my
>>>>>>>>>>>>>>> own
>>>>>>>>>>>>>>> tests on my setup with FUSE so sent
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> reply.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Pat
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 05/10/2017 12:15 PM, Pranith
>>>>>>>>>>>>>>> Kumar Karampuri wrote:
>>>>>>>>>>>>>>>> Okay good. At least this
>>>>>>>>>>>>>>>> validates
>>>>>>>>>>>>>>>> my doubts. Handling O_SYNC in
>>>>>>>>>>>>>>>> gluster NFS and fuse is a bit
>>>>>>>>>>>>>>>> different.
>>>>>>>>>>>>>>>> When application opens a file
>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> O_SYNC on fuse mount then each
>>>>>>>>>>>>>>>> write syscall has to be written
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> disk as part of the syscall
>>>>>>>>>>>>>>>> where
>>>>>>>>>>>>>>>> as in case of NFS, there is no
>>>>>>>>>>>>>>>> concept of open. NFS performs
>>>>>>>>>>>>>>>> write
>>>>>>>>>>>>>>>> though a handle saying it needs
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> be a synchronous write, so
>>>>>>>>>>>>>>>> write()
>>>>>>>>>>>>>>>> syscall is performed first then
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> performs fsync(). so an write
>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>> fd with O_SYNC becomes
>>>>>>>>>>>>>>>> write+fsync.
>>>>>>>>>>>>>>>> I am suspecting that when
>>>>>>>>>>>>>>>> multiple
>>>>>>>>>>>>>>>> threads do this write+fsync()
>>>>>>>>>>>>>>>> operation on the same file,
>>>>>>>>>>>>>>>> multiple writes are batched
>>>>>>>>>>>>>>>> together to be written do disk
>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>> the throughput on the disk is
>>>>>>>>>>>>>>>> increasing is my guess.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Does it answer your doubts?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, May 10, 2017 at 9:35
>>>>>>>>>>>>>>>> PM,
>>>>>>>>>>>>>>>> Pat Haley <phaley at mit.edu
>>>>>>>>>>>>>>>> <mailto:phaley at mit.edu>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Without the oflag=sync and
>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>> a single test of each, the
>>>>>>>>>>>>>>>> FUSE
>>>>>>>>>>>>>>>> is going faster than NFS:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> FUSE:
>>>>>>>>>>>>>>>> mseas-data2(dri_nascar)% dd
>>>>>>>>>>>>>>>> if=/dev/zero count=4096
>>>>>>>>>>>>>>>> bs=1048576 of=zeros.txt
>>>>>>>>>>>>>>>> conv=sync
>>>>>>>>>>>>>>>> 4096+0 records in
>>>>>>>>>>>>>>>> 4096+0 records out
>>>>>>>>>>>>>>>> 4294967296 bytes (4.3 GB)
>>>>>>>>>>>>>>>> copied, 7.46961 s, 575 MB/s
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> NFS
>>>>>>>>>>>>>>>> mseas-data2(HYCOM)% dd
>>>>>>>>>>>>>>>> if=/dev/zero count=4096
>>>>>>>>>>>>>>>> bs=1048576 of=zeros.txt
>>>>>>>>>>>>>>>> conv=sync
>>>>>>>>>>>>>>>> 4096+0 records in
>>>>>>>>>>>>>>>> 4096+0 records out
>>>>>>>>>>>>>>>> 4294967296 bytes (4.3 GB)
>>>>>>>>>>>>>>>> copied, 11.4264 s, 376 MB/s
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 05/10/2017 11:53 AM,
>>>>>>>>>>>>>>>> Pranith
>>>>>>>>>>>>>>>> Kumar Karampuri wrote:
>>>>>>>>>>>>>>>>> Could you let me know the
>>>>>>>>>>>>>>>>> speed without oflag=sync
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> both the mounts? No need
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> collect profiles.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, May 10, 2017 at
>>>>>>>>>>>>>>>>> 9:17
>>>>>>>>>>>>>>>>> PM, Pat Haley
>>>>>>>>>>>>>>>>> <phaley at mit.edu
>>>>>>>>>>>>>>>>> <mailto:phaley at mit.edu>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Here is what I see
>>>>>>>>>>>>>>>>> now:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [root at mseas-data2 ~]#
>>>>>>>>>>>>>>>>> gluster volume info
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Volume Name:
>>>>>>>>>>>>>>>>> data-volume
>>>>>>>>>>>>>>>>> Type: Distribute
>>>>>>>>>>>>>>>>> Volume ID:
>>>>>>>>>>>>>>>>> c162161e-2a2d-4dac-b015-f31fd89ceb18
>>>>>>>>>>>>>>>>> Status: Started
>>>>>>>>>>>>>>>>> Number of Bricks: 2
>>>>>>>>>>>>>>>>> Transport-type: tcp
>>>>>>>>>>>>>>>>> Bricks:
>>>>>>>>>>>>>>>>> Brick1:
>>>>>>>>>>>>>>>>> mseas-data2:/mnt/brick1
>>>>>>>>>>>>>>>>> Brick2:
>>>>>>>>>>>>>>>>> mseas-data2:/mnt/brick2
>>>>>>>>>>>>>>>>> Options Reconfigured:
>>>>>>>>>>>>>>>>> diagnostics.count-fop-hits:
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> diagnostics.latency-measurement:
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> nfs.exports-auth-enable:
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> diagnostics.brick-sys-log-level:
>>>>>>>>>>>>>>>>> WARNING
>>>>>>>>>>>>>>>>> performance.readdir-ahead:
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> nfs.disable: on
>>>>>>>>>>>>>>>>> nfs.export-volumes:
>>>>>>>>>>>>>>>>> off
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 05/10/2017 11:44
>>>>>>>>>>>>>>>>> AM,
>>>>>>>>>>>>>>>>> Pranith Kumar
>>>>>>>>>>>>>>>>> Karampuri
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> Is this the volume
>>>>>>>>>>>>>>>>>> info
>>>>>>>>>>>>>>>>>> you have?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> >/[root at
>>>>>>>>>>>>>>>>>> >mseas-data2
>>>>>>>>>>>>>>>>>> <http://www.gluster.org/mailman/listinfo/gluster-users>
>>>>>>>>>>>>>>>>>> ~]# gluster volume
>>>>>>>>>>>>>>>>>> info
>>>>>>>>>>>>>>>>>> />//>/Volume Name:
>>>>>>>>>>>>>>>>>> data-volume />/Type:
>>>>>>>>>>>>>>>>>> Distribute />/Volume
>>>>>>>>>>>>>>>>>> ID:
>>>>>>>>>>>>>>>>>> c162161e-2a2d-4dac-b015-f31fd89ceb18
>>>>>>>>>>>>>>>>>> />/Status: Started
>>>>>>>>>>>>>>>>>> />/Number
>>>>>>>>>>>>>>>>>> of Bricks: 2
>>>>>>>>>>>>>>>>>> />/Transport-type:
>>>>>>>>>>>>>>>>>> tcp
>>>>>>>>>>>>>>>>>> />/Bricks: />/Brick1:
>>>>>>>>>>>>>>>>>> mseas-data2:/mnt/brick1
>>>>>>>>>>>>>>>>>> />/Brick2:
>>>>>>>>>>>>>>>>>> mseas-data2:/mnt/brick2
>>>>>>>>>>>>>>>>>> />/Options
>>>>>>>>>>>>>>>>>> Reconfigured:
>>>>>>>>>>>>>>>>>> />/performance.readdir-ahead:
>>>>>>>>>>>>>>>>>> on />/nfs.disable: on
>>>>>>>>>>>>>>>>>> />/nfs.export-volumes:
>>>>>>>>>>>>>>>>>> off
>>>>>>>>>>>>>>>>>> /
>>>>>>>>>>>>>>>>>> I copied this from
>>>>>>>>>>>>>>>>>> old
>>>>>>>>>>>>>>>>>> thread from 2016.
>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> distribute volume.
>>>>>>>>>>>>>>>>>> Did
>>>>>>>>>>>>>>>>>> you change any of the
>>>>>>>>>>>>>>>>>> options in between?
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>>>>>>>>> Pat Haley
>>>>>>>>>>>>>>>>> Email:phaley at mit.edu
>>>>>>>>>>>>>>>>> <mailto:phaley at mit.edu>
>>>>>>>>>>>>>>>>> Center for Ocean
>>>>>>>>>>>>>>>>> Engineering
>>>>>>>>>>>>>>>>> Phone: (617) 253-6824
>>>>>>>>>>>>>>>>> Dept. of Mechanical
>>>>>>>>>>>>>>>>> Engineering
>>>>>>>>>>>>>>>>> Fax: (617) 253-8125
>>>>>>>>>>>>>>>>> MIT, Room
>>>>>>>>>>>>>>>>> 5-213http://web.mit.edu/phaley/www/
>>>>>>>>>>>>>>>>> 77 Massachusetts
>>>>>>>>>>>>>>>>> Avenue
>>>>>>>>>>>>>>>>> Cambridge, MA
>>>>>>>>>>>>>>>>> 02139-4301
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Pranith
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>>>>>>>> Pat Haley
>>>>>>>>>>>>>>>> Email:phaley at mit.edu
>>>>>>>>>>>>>>>> <mailto:phaley at mit.edu>
>>>>>>>>>>>>>>>> Center for Ocean
>>>>>>>>>>>>>>>> Engineering
>>>>>>>>>>>>>>>> Phone: (617) 253-6824
>>>>>>>>>>>>>>>> Dept. of Mechanical
>>>>>>>>>>>>>>>> Engineering
>>>>>>>>>>>>>>>> Fax: (617) 253-8125
>>>>>>>>>>>>>>>> MIT, Room
>>>>>>>>>>>>>>>> 5-213http://web.mit.edu/phaley/www/
>>>>>>>>>>>>>>>> 77 Massachusetts Avenue
>>>>>>>>>>>>>>>> Cambridge, MA 02139-4301
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Pranith
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>>>>>>> Pat Haley
>>>>>>>>>>>>>>> Email:phaley at mit.edu
>>>>>>>>>>>>>>> <mailto:phaley at mit.edu>
>>>>>>>>>>>>>>> Center for Ocean Engineering
>>>>>>>>>>>>>>> Phone:
>>>>>>>>>>>>>>> (617) 253-6824
>>>>>>>>>>>>>>> Dept. of Mechanical Engineering
>>>>>>>>>>>>>>> Fax:
>>>>>>>>>>>>>>> (617) 253-8125
>>>>>>>>>>>>>>> MIT, Room
>>>>>>>>>>>>>>> 5-213http://web.mit.edu/phaley/www/
>>>>>>>>>>>>>>> 77 Massachusetts Avenue
>>>>>>>>>>>>>>> Cambridge, MA 02139-4301
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Pranith
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>>>>>> Pat Haley
>>>>>>>>>>>>>> Email:phaley at mit.edu
>>>>>>>>>>>>>> <mailto:phaley at mit.edu>
>>>>>>>>>>>>>> Center for Ocean Engineering
>>>>>>>>>>>>>> Phone:
>>>>>>>>>>>>>> (617) 253-6824
>>>>>>>>>>>>>> Dept. of Mechanical Engineering
>>>>>>>>>>>>>> Fax:
>>>>>>>>>>>>>> (617) 253-8125
>>>>>>>>>>>>>> MIT, Room
>>>>>>>>>>>>>> 5-213http://web.mit.edu/phaley/www/
>>>>>>>>>>>>>> 77 Massachusetts Avenue
>>>>>>>>>>>>>> Cambridge, MA 02139-4301
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Pranith
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>>>>> Pat Haley
>>>>>>>>>>>>> Email:phaley at mit.edu
>>>>>>>>>>>>> <mailto:phaley at mit.edu>
>>>>>>>>>>>>> Center for Ocean Engineering Phone:
>>>>>>>>>>>>> (617)
>>>>>>>>>>>>> 253-6824
>>>>>>>>>>>>> Dept. of Mechanical Engineering Fax:
>>>>>>>>>>>>> (617)
>>>>>>>>>>>>> 253-8125
>>>>>>>>>>>>> MIT, Room
>>>>>>>>>>>>> 5-213http://web.mit.edu/phaley/www/
>>>>>>>>>>>>> 77 Massachusetts Avenue
>>>>>>>>>>>>> Cambridge, MA 02139-4301
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Pranith
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>>>> Pat Haley
>>>>>>>>>>>> Email:phaley at mit.edu
>>>>>>>>>>>> <mailto:phaley at mit.edu>
>>>>>>>>>>>> Center for Ocean Engineering Phone:
>>>>>>>>>>>> (617)
>>>>>>>>>>>> 253-6824
>>>>>>>>>>>> Dept. of Mechanical Engineering Fax:
>>>>>>>>>>>> (617)
>>>>>>>>>>>> 253-8125
>>>>>>>>>>>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>>>>>>>>> 77 Massachusetts Avenue
>>>>>>>>>>>> Cambridge, MA 02139-4301
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Pranith
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>>> Pat Haley
>>>>>>>>>>> Email:phaley at mit.edu
>>>>>>>>>>> <mailto:phaley at mit.edu>
>>>>>>>>>>> Center for Ocean Engineering Phone: (617)
>>>>>>>>>>> 253-6824
>>>>>>>>>>> Dept. of Mechanical Engineering Fax: (617)
>>>>>>>>>>> 253-8125
>>>>>>>>>>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>>>>>>>> 77 Massachusetts Avenue
>>>>>>>>>>> Cambridge, MA 02139-4301
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Pranith
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>>> Pat Haley Email:phaley at mit.edu
>>>>>>>>>> <mailto:phaley at mit.edu>
>>>>>>>>>> Center for Ocean Engineering Phone: (617) 253-6824
>>>>>>>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>>>>>>>>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>>>>>>> 77 Massachusetts Avenue
>>>>>>>>>> Cambridge, MA 02139-4301
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Pranith
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>>> Pat Haley Email:phaley at mit.edu
>>>>>>>>> <mailto:phaley at mit.edu>
>>>>>>>>> Center for Ocean Engineering Phone: (617) 253-6824
>>>>>>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>>>>>>>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>>>>>> 77 Massachusetts Avenue
>>>>>>>>> Cambridge, MA 02139-4301
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Pranith
>>>>>>>> --
>>>>>>>>
>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>> Pat Haley Email: phaley at mit.edu
>>>>>>>> Center for Ocean Engineering Phone: (617) 253-6824
>>>>>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>>>>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>>>>>>> 77 Massachusetts Avenue
>>>>>>>> Cambridge, MA 02139-4301
>>>>>>>>
>>>>>>>>
>>>>>> --
>>>>>>
>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>> Pat Haley Email: phaley at mit.edu
>>>>>> Center for Ocean Engineering Phone: (617) 253-6824
>>>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>>>>> 77 Massachusetts Avenue
>>>>>> Cambridge, MA 02139-4301
>>>>>>
>>>>>>
>>>> --
>>>>
>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>> Pat Haley Email: phaley at mit.edu
>>>> Center for Ocean Engineering Phone: (617) 253-6824
>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>>> 77 Massachusetts Avenue
>>>> Cambridge, MA 02139-4301
>>>>
>>>>
>> --
>>
>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> Pat Haley Email: phaley at mit.edu
>> Center for Ocean Engineering Phone: (617) 253-6824
>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>> 77 Massachusetts Avenue
>> Cambridge, MA 02139-4301
>>
>>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: phaley at mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170620/6ddc35ef/attachment.html>
More information about the Gluster-users
mailing list