[Gluster-users] Slow write times to gluster disk
Pranith Kumar Karampuri
pkarampu at redhat.com
Tue Jun 27 04:47:40 UTC 2017
On Mon, Jun 26, 2017 at 7:40 PM, Pat Haley <phaley at mit.edu> wrote:
>
> Hi All,
>
> Decided to try another tests of gluster mounted via FUSE vs gluster
> mounted via NFS, this time using the software we run in production (i.e.
> our ocean model writing a netCDF file).
>
> gluster mounted via NFS the run took 2.3 hr
>
> gluster mounted via FUSE: the run took 44.2 hr
>
> The only problem with using gluster mounted via NFS is that it does not
> respect the group write permissions which we need.
>
> We have an exercise coming up in the a couple of weeks. It seems to me
> that in order to improve our write times before then, it would be good to
> solve the group write permissions for gluster mounted via NFS now. We can
> then revisit gluster mounted via FUSE afterwards.
>
> What information would you need to help us force gluster mounted via NFS
> to respect the group write permissions?
>
+Niels, +Jiffin
I added 2 more guys who work on NFS to check why this problem happens in
your environment. Let's see what information they may need to find the
problem and solve this issue.
>
> Thanks
>
> Pat
>
>
>
>
> On 06/24/2017 01:43 AM, Pranith Kumar Karampuri wrote:
>
>
>
> On Fri, Jun 23, 2017 at 9:10 AM, Pranith Kumar Karampuri <
> pkarampu at redhat.com> wrote:
>
>>
>>
>> On Fri, Jun 23, 2017 at 2:23 AM, Pat Haley <phaley at mit.edu> wrote:
>>
>>>
>>> Hi,
>>>
>>> Today we experimented with some of the FUSE options that we found in the
>>> list.
>>>
>>> Changing these options had no effect:
>>>
>>> gluster volume set test-volume performance.cache-max-file-size 2MB
>>> gluster volume set test-volume performance.cache-refresh-timeout 4
>>> gluster volume set test-volume performance.cache-size 256MB
>>> gluster volume set test-volume performance.write-behind-window-size 4MB
>>> gluster volume set test-volume performance.write-behind-window-size 8MB
>>>
>>>
>> This is a good coincidence, I am meeting with write-behind
>> maintainer(+Raghavendra G) today for the same doubt. I think we will have
>> something by EOD IST. I will update you.
>>
>
> Sorry, forgot to update you. It seems like there is a bug in Write-behind
> and Facebook guys sent a patch http://review.gluster.org/16079 to fix the
> same. But even with that I am not seeing any improvement. May be I am doing
> something wrong. Will update you if I find anything more.
>
>> Changing the following option from its default value made the speed slower
>>>
>>> gluster volume set test-volume performance.write-behind off (on by default)
>>>
>>> Changing the following options initially appeared to give a 10% increase
>>> in speed, but this vanished in subsequent tests (we think the apparent
>>> increase may have been to a lighter workload on the computer from other
>>> users)
>>>
>>> gluster volume set test-volume performance.stat-prefetch on
>>> gluster volume set test-volume client.event-threads 4
>>> gluster volume set test-volume server.event-threads 4
>>>
>>> Can anything be gleaned from these observations? Are there other things
>>> we can try?
>>>
>>> Thanks
>>>
>>> Pat
>>>
>>>
>>>
>>> On 06/20/2017 12:06 PM, Pat Haley wrote:
>>>
>>>
>>> Hi Ben,
>>>
>>> Sorry this took so long, but we had a real-time forecasting exercise
>>> last week and I could only get to this now.
>>>
>>> Backend Hardware/OS:
>>>
>>> - Much of the information on our back end system is included at the
>>> top of http://lists.gluster.org/pipermail/gluster-users/2017-April/
>>> 030529.html
>>> - The specific model of the hard disks is SeaGate ENTERPRISE
>>> CAPACITY V.4 6TB (ST6000NM0024). The rated speed is 6Gb/s.
>>> - Note: there is one physical server that hosts both the NFS and the
>>> GlusterFS areas
>>>
>>> Latest tests
>>>
>>> I have had time to run the tests for one of the dd tests you requested
>>> to the underlying XFS FS. The median rate was 170 MB/s. The dd results
>>> and iostat record are in
>>>
>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestXFS/
>>>
>>> I'll add tests for the other brick and to the NFS area later.
>>>
>>> Thanks
>>>
>>> Pat
>>>
>>>
>>> On 06/12/2017 06:06 PM, Ben Turner wrote:
>>>
>>> Ok you are correct, you have a pure distributed volume. IE no replication overhead. So normally for pure dist I use:
>>>
>>> throughput = slowest of disks / NIC * .6-.7
>>>
>>> In your case we have:
>>>
>>> 1200 * .6 = 720
>>>
>>> So you are seeing a little less throughput than I would expect in your configuration. What I like to do here is:
>>>
>>> -First tell me more about your back end storage, will it sustain 1200 MB / sec? What kind of HW? How many disks? What type and specs are the disks? What kind of RAID are you using?
>>>
>>> -Second can you refresh me on your workload? Are you doing reads / writes or both? If both what mix? Since we are using DD I assume you are working iwth large file sequential I/O, is this correct?
>>>
>>> -Run some DD tests on the back end XFS FS. I normally have /xfs-mount/gluster-brick, if you have something similar just mkdir on the XFS -> /xfs-mount/my-test-dir. Inside the test dir run:
>>>
>>> If you are focusing on a write workload run:
>>>
>>> # dd if=/dev/zero of=/xfs-mount/file bs=1024k count=10000 conv=fdatasync
>>>
>>> If you are focusing on a read workload run:
>>>
>>> # echo 3 > /proc/sys/vm/drop_caches
>>> # dd if=/gluster-mount/file of=/dev/null bs=1024k count=10000
>>>
>>> ** MAKE SURE TO DROP CACHE IN BETWEEN READS!! **
>>>
>>> Run this in a loop similar to how you did in:
>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/dd_testvol_gluster.txt
>>>
>>> Run this on both servers one at a time and if you are running on a SAN then run again on both at the same time. While this is running gather iostat for me:
>>>
>>> # iostat -c -m -x 1 > iostat-$(hostname).txt
>>>
>>> Lets see how the back end performs on both servers while capturing iostat, then see how the same workload / data looks on gluster.
>>>
>>> -Last thing, when you run your kernel NFS tests are you using the same filesystem / storage you are using for the gluster bricks? I want to be sure we have an apples to apples comparison here.
>>>
>>> -b
>>>
>>>
>>>
>>> ----- Original Message -----
>>>
>>> From: "Pat Haley" <phaley at mit.edu> <phaley at mit.edu>
>>> To: "Ben Turner" <bturner at redhat.com> <bturner at redhat.com>
>>> Sent: Monday, June 12, 2017 5:18:07 PM
>>> Subject: Re: [Gluster-users] Slow write times to gluster disk
>>>
>>>
>>> Hi Ben,
>>>
>>> Here is the output:
>>>
>>> [root at mseas-data2 ~]# gluster volume info
>>>
>>> Volume Name: data-volume
>>> Type: Distribute
>>> Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
>>> Status: Started
>>> Number of Bricks: 2
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: mseas-data2:/mnt/brick1
>>> Brick2: mseas-data2:/mnt/brick2
>>> Options Reconfigured:
>>> nfs.exports-auth-enable: on
>>> diagnostics.brick-sys-log-level: WARNING
>>> performance.readdir-ahead: on
>>> nfs.disable: on
>>> nfs.export-volumes: off
>>>
>>>
>>> On 06/12/2017 05:01 PM, Ben Turner wrote:
>>>
>>> What is the output of gluster v info? That will tell us more about your
>>> config.
>>>
>>> -b
>>>
>>> ----- Original Message -----
>>>
>>> From: "Pat Haley" <phaley at mit.edu> <phaley at mit.edu>
>>> To: "Ben Turner" <bturner at redhat.com> <bturner at redhat.com>
>>> Sent: Monday, June 12, 2017 4:54:00 PM
>>> Subject: Re: [Gluster-users] Slow write times to gluster disk
>>>
>>>
>>> Hi Ben,
>>>
>>> I guess I'm confused about what you mean by replication. If I look at
>>> the underlying bricks I only ever have a single copy of any file. It
>>> either resides on one brick or the other (directories exist on both
>>> bricks but not files). We are not using gluster for redundancy (or at
>>> least that wasn't our intent). Is that what you meant by replication
>>> or is it something else?
>>>
>>> Thanks
>>>
>>> Pat
>>>
>>> On 06/12/2017 04:28 PM, Ben Turner wrote:
>>>
>>> ----- Original Message -----
>>>
>>> From: "Pat Haley" <phaley at mit.edu> <phaley at mit.edu>
>>> To: "Ben Turner" <bturner at redhat.com> <bturner at redhat.com>, "Pranith Kumar Karampuri"<pkarampu at redhat.com> <pkarampu at redhat.com>
>>> Cc: "Ravishankar N" <ravishankar at redhat.com> <ravishankar at redhat.com>, gluster-users at gluster.org,
>>> "Steve Postma" <SPostma at ztechnet.com> <SPostma at ztechnet.com>
>>> Sent: Monday, June 12, 2017 2:35:41 PM
>>> Subject: Re: [Gluster-users] Slow write times to gluster disk
>>>
>>>
>>> Hi Guys,
>>>
>>> I was wondering what our next steps should be to solve the slow write
>>> times.
>>>
>>> Recently I was debugging a large code and writing a lot of output at
>>> every time step. When I tried writing to our gluster disks, it was
>>> taking over a day to do a single time step whereas if I had the same
>>> program (same hardware, network) write to our nfs disk the time per
>>> time-step was about 45 minutes. What we are shooting for here would be
>>> to have similar times to either gluster of nfs.
>>>
>>> I can see in your test:
>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/dd_testvol_gluster.txt
>>>
>>> You averaged ~600 MB / sec(expected for replica 2 with 10G, {~1200 MB /
>>> sec} / #replicas{2} = 600). Gluster does client side replication so with
>>> replica 2 you will only ever see 1/2 the speed of your slowest part of
>>> the
>>> stack(NW, disk, RAM, CPU). This is usually NW or disk and 600 is
>>> normally
>>> a best case. Now in your output I do see the instances where you went
>>> down to 200 MB / sec. I can only explain this in three ways:
>>>
>>> 1. You are not using conv=fdatasync and writes are actually going to
>>> page
>>> cache and then being flushed to disk. During the fsync the memory is not
>>> yet available and the disks are busy flushing dirty pages.
>>> 2. Your storage RAID group is shared across multiple LUNS(like in a SAN)
>>> and when write times are slow the RAID group is busy serviceing other
>>> LUNs.
>>> 3. Gluster bug / config issue / some other unknown unknown.
>>>
>>> So I see 2 issues here:
>>>
>>> 1. NFS does in 45 minutes what gluster can do in 24 hours.
>>> 2. Sometimes your throughput drops dramatically.
>>>
>>> WRT #1 - have a look at my estimates above. My formula for guestimating
>>> gluster perf is: throughput = NIC throughput or storage(whatever is
>>> slower) / # replicas * overhead(figure .7 or .8). Also the larger the
>>> record size the better for glusterfs mounts, I normally like to be at
>>> LEAST 64k up to 1024k:
>>>
>>> # dd if=/dev/zero of=/gluster-mount/file bs=1024k count=10000
>>> conv=fdatasync
>>>
>>> WRT #2 - Again, I question your testing and your storage config. Try
>>> using
>>> conv=fdatasync for your DDs, use a larger record size, and make sure that
>>> your back end storage is not causing your slowdowns. Also remember that
>>> with replica 2 you will take ~50% hit on writes because the client uses
>>> 50% of its bandwidth to write to one replica and 50% to the other.
>>>
>>> -b
>>>
>>>
>>>
>>>
>>> Thanks
>>>
>>> Pat
>>>
>>>
>>> On 06/02/2017 01:07 AM, Ben Turner wrote:
>>>
>>> Are you sure using conv=sync is what you want? I normally use
>>> conv=fdatasync, I'll look up the difference between the two and see if
>>> it
>>> affects your test.
>>>
>>>
>>> -b
>>>
>>> ----- Original Message -----
>>>
>>> From: "Pat Haley" <phaley at mit.edu> <phaley at mit.edu>
>>> To: "Pranith Kumar Karampuri" <pkarampu at redhat.com> <pkarampu at redhat.com>
>>> Cc: "Ravishankar N" <ravishankar at redhat.com> <ravishankar at redhat.com>,gluster-users at gluster.org,
>>> "Steve Postma" <SPostma at ztechnet.com> <SPostma at ztechnet.com>, "Ben
>>> Turner" <bturner at redhat.com> <bturner at redhat.com>
>>> Sent: Tuesday, May 30, 2017 9:40:34 PM
>>> Subject: Re: [Gluster-users] Slow write times to gluster disk
>>>
>>>
>>> Hi Pranith,
>>>
>>> The "dd" command was:
>>>
>>> dd if=/dev/zero count=4096 bs=1048576 of=zeros.txt conv=sync
>>>
>>> There were 2 instances where dd reported 22 seconds. The output from
>>> the
>>> dd tests are in
>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/dd_testvol_gluster.txt
>>>
>>> Pat
>>>
>>> On 05/30/2017 09:27 PM, Pranith Kumar Karampuri wrote:
>>>
>>> Pat,
>>> What is the command you used? As per the following output,
>>> it
>>> seems like at least one write operation took 16 seconds. Which is
>>> really bad.
>>> 96.39 1165.10 us 89.00 us*16487014.00 us*
>>> 393212
>>> WRITE
>>>
>>>
>>> On Tue, May 30, 2017 at 10:36 PM, Pat Haley <phaley at mit.edu<mailto:phaley at mit.edu> <phaley at mit.edu>> wrote:
>>>
>>>
>>> Hi Pranith,
>>>
>>> I ran the same 'dd' test both in the gluster test volume and
>>> in
>>> the .glusterfs directory of each brick. The median results
>>> (12
>>> dd
>>> trials in each test) are similar to before
>>>
>>> * gluster test volume: 586.5 MB/s
>>> * bricks (in .glusterfs): 1.4 GB/s
>>>
>>> The profile for the gluster test-volume is in
>>>
>>> http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/profile_testvol_gluster.txt
>>> <http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/profile_testvol_gluster.txt> <http://mseas.mit.edu/download/phaley/GlusterUsers/TestVol/profile_testvol_gluster.txt>
>>>
>>> Thanks
>>>
>>> Pat
>>>
>>>
>>>
>>>
>>> On 05/30/2017 12:10 PM, Pranith Kumar Karampuri wrote:
>>>
>>> Let's start with the same 'dd' test we were testing with to
>>> see,
>>> what the numbers are. Please provide profile numbers for the
>>> same. From there on we will start tuning the volume to see
>>> what
>>> we can do.
>>>
>>> On Tue, May 30, 2017 at 9:16 PM, Pat Haley <phaley at mit.edu
>>> <mailto:phaley at mit.edu> <phaley at mit.edu>> wrote:
>>>
>>>
>>> Hi Pranith,
>>>
>>> Thanks for the tip. We now have the gluster volume
>>> mounted
>>> under /home. What tests do you recommend we run?
>>>
>>> Thanks
>>>
>>> Pat
>>>
>>>
>>>
>>> On 05/17/2017 05:01 AM, Pranith Kumar Karampuri wrote:
>>>
>>> On Tue, May 16, 2017 at 9:20 PM, Pat Haley
>>> <phaley at mit.edu
>>> <mailto:phaley at mit.edu> <phaley at mit.edu>> wrote:
>>>
>>>
>>> Hi Pranith,
>>>
>>> Sorry for the delay. I never saw received your
>>> reply
>>> (but I did receive Ben Turner's follow-up to your
>>> reply). So we tried to create a gluster volume
>>> under
>>> /home using different variations of
>>>
>>> gluster volume create test-volume
>>> mseas-data2:/home/gbrick_test_1
>>> mseas-data2:/home/gbrick_test_2 transport tcp
>>>
>>> However we keep getting errors of the form
>>>
>>> Wrong brick type: transport, use
>>> <HOSTNAME>:<export-dir-abs-path>
>>>
>>> Any thoughts on what we're doing wrong?
>>>
>>>
>>> You should give transport tcp at the beginning I think.
>>> Anyways, transport tcp is the default, so no need to
>>> specify
>>> so remove those two words from the CLI.
>>>
>>>
>>> Also do you have a list of the test we should be
>>> running
>>> once we get this volume created? Given the
>>> time-zone
>>> difference it might help if we can run a small
>>> battery
>>> of tests and post the results rather than
>>> test-post-new
>>> test-post... .
>>>
>>>
>>> This is the first time I am doing performance analysis
>>> on
>>> users as far as I remember. In our team there are
>>> separate
>>> engineers who do these tests. Ben who replied earlier is
>>> one
>>> such engineer.
>>>
>>> Ben,
>>> Have any suggestions?
>>>
>>>
>>> Thanks
>>>
>>> Pat
>>>
>>>
>>>
>>> On 05/11/2017 12:06 PM, Pranith Kumar Karampuri
>>> wrote:
>>>
>>> On Thu, May 11, 2017 at 9:32 PM, Pat Haley
>>> <phaley at mit.edu <mailto:phaley at mit.edu> <phaley at mit.edu>> wrote:
>>>
>>>
>>> Hi Pranith,
>>>
>>> The /home partition is mounted as ext4
>>> /home ext4 defaults,usrquota,grpquota 1 2
>>>
>>> The brick partitions are mounted ax xfs
>>> /mnt/brick1 xfs defaults 0 0
>>> /mnt/brick2 xfs defaults 0 0
>>>
>>> Will this cause a problem with creating a
>>> volume
>>> under /home?
>>>
>>>
>>> I don't think the bottleneck is disk. You can do
>>> the
>>> same tests you did on your new volume to confirm?
>>>
>>>
>>> Pat
>>>
>>>
>>>
>>> On 05/11/2017 11:32 AM, Pranith Kumar Karampuri
>>> wrote:
>>>
>>> On Thu, May 11, 2017 at 8:57 PM, Pat Haley
>>> <phaley at mit.edu <mailto:phaley at mit.edu> <phaley at mit.edu>>
>>> wrote:
>>>
>>>
>>> Hi Pranith,
>>>
>>> Unfortunately, we don't have similar
>>> hardware
>>> for a small scale test. All we have is
>>> our
>>> production hardware.
>>>
>>>
>>> You said something about /home partition which
>>> has
>>> lesser disks, we can create plain distribute
>>> volume inside one of those directories. After
>>> we
>>> are done, we can remove the setup. What do you
>>> say?
>>>
>>>
>>> Pat
>>>
>>>
>>>
>>>
>>> On 05/11/2017 07:05 AM, Pranith Kumar
>>> Karampuri wrote:
>>>
>>> On Thu, May 11, 2017 at 2:48 AM, Pat
>>> Haley
>>> <phaley at mit.edu <mailto:phaley at mit.edu> <phaley at mit.edu>>
>>> wrote:
>>>
>>>
>>> Hi Pranith,
>>>
>>> Since we are mounting the partitions
>>> as
>>> the bricks, I tried the dd test
>>> writing
>>> to
>>> <brick-path>/.glusterfs/<file-to-be-removed-after-test>.
>>> The results without oflag=sync were
>>> 1.6
>>> Gb/s (faster than gluster but not as
>>> fast
>>> as I was expecting given the 1.2 Gb/s
>>> to
>>> the no-gluster area w/ fewer disks).
>>>
>>>
>>> Okay, then 1.6Gb/s is what we need to
>>> target
>>> for, considering your volume is just
>>> distribute. Is there any way you can do
>>> tests
>>> on similar hardware but at a small scale?
>>> Just so we can run the workload to learn
>>> more
>>> about the bottlenecks in the system? We
>>> can
>>> probably try to get the speed to 1.2Gb/s
>>> on
>>> your /home partition you were telling me
>>> yesterday. Let me know if that is
>>> something
>>> you are okay to do.
>>>
>>>
>>> Pat
>>>
>>>
>>>
>>> On 05/10/2017 01:27 PM, Pranith Kumar
>>> Karampuri wrote:
>>>
>>> On Wed, May 10, 2017 at 10:15 PM,
>>> Pat
>>> Haley <phaley at mit.edu
>>> <mailto:phaley at mit.edu> <phaley at mit.edu>> wrote:
>>>
>>>
>>> Hi Pranith,
>>>
>>> Not entirely sure (this isn't my
>>> area of expertise). I'll run
>>> your
>>> answer by some other people who
>>> are
>>> more familiar with this.
>>>
>>> I am also uncertain about how to
>>> interpret the results when we
>>> also
>>> add the dd tests writing to the
>>> /home area (no gluster, still on
>>> the
>>> same machine)
>>>
>>> * dd test without oflag=sync
>>> (rough average of multiple
>>> tests)
>>> o gluster w/ fuse mount :
>>> 570
>>> Mb/s
>>> o gluster w/ nfs mount:
>>> 390
>>> Mb/s
>>> o nfs (no gluster): 1.2
>>> Gb/s
>>> * dd test with oflag=sync
>>> (rough
>>> average of multiple tests)
>>> o gluster w/ fuse mount:
>>> 5
>>> Mb/s
>>> o gluster w/ nfs mount:
>>> 200
>>> Mb/s
>>> o nfs (no gluster): 20
>>> Mb/s
>>>
>>> Given that the non-gluster area
>>> is
>>> a
>>> RAID-6 of 4 disks while each
>>> brick
>>> of the gluster area is a RAID-6
>>> of
>>> 32 disks, I would naively expect
>>> the
>>> writes to the gluster area to be
>>> roughly 8x faster than to the
>>> non-gluster.
>>>
>>>
>>> I think a better test is to try and
>>> write to a file using nfs without
>>> any
>>> gluster to a location that is not
>>> inside
>>> the brick but someother location
>>> that
>>> is
>>> on same disk(s). If you are mounting
>>> the
>>> partition as the brick, then we can
>>> write to a file inside .glusterfs
>>> directory, something like
>>> <brick-path>/.glusterfs/<file-to-be-removed-after-test>.
>>>
>>>
>>>
>>> I still think we have a speed
>>> issue,
>>> I can't tell if fuse vs nfs is
>>> part
>>> of the problem.
>>>
>>>
>>> I got interested in the post because
>>> I
>>> read that fuse speed is lesser than
>>> nfs
>>> speed which is counter-intuitive to
>>> my
>>> understanding. So wanted
>>> clarifications.
>>> Now that I got my clarifications
>>> where
>>> fuse outperformed nfs without sync,
>>> we
>>> can resume testing as described
>>> above
>>> and try to find what it is. Based on
>>> your email-id I am guessing you are
>>> from
>>> Boston and I am from Bangalore so if
>>> you
>>> are okay with doing this debugging
>>> for
>>> multiple days because of timezones,
>>> I
>>> will be happy to help. Please be a
>>> bit
>>> patient with me, I am under a
>>> release
>>> crunch but I am very curious with
>>> the
>>> problem you posted.
>>>
>>> Was there anything useful in the
>>> profiles?
>>>
>>>
>>> Unfortunately profiles didn't help
>>> me
>>> much, I think we are collecting the
>>> profiles from an active volume, so
>>> it
>>> has a lot of information that is not
>>> pertaining to dd so it is difficult
>>> to
>>> find the contributions of dd. So I
>>> went
>>> through your post again and found
>>> something I didn't pay much
>>> attention
>>> to
>>> earlier i.e. oflag=sync, so did my
>>> own
>>> tests on my setup with FUSE so sent
>>> that
>>> reply.
>>>
>>>
>>> Pat
>>>
>>>
>>>
>>> On 05/10/2017 12:15 PM, Pranith
>>> Kumar Karampuri wrote:
>>>
>>> Okay good. At least this
>>> validates
>>> my doubts. Handling O_SYNC in
>>> gluster NFS and fuse is a bit
>>> different.
>>> When application opens a file
>>> with
>>> O_SYNC on fuse mount then each
>>> write syscall has to be written
>>> to
>>> disk as part of the syscall
>>> where
>>> as in case of NFS, there is no
>>> concept of open. NFS performs
>>> write
>>> though a handle saying it needs
>>> to
>>> be a synchronous write, so
>>> write()
>>> syscall is performed first then
>>> it
>>> performs fsync(). so an write
>>> on
>>> an
>>> fd with O_SYNC becomes
>>> write+fsync.
>>> I am suspecting that when
>>> multiple
>>> threads do this write+fsync()
>>> operation on the same file,
>>> multiple writes are batched
>>> together to be written do disk
>>> so
>>> the throughput on the disk is
>>> increasing is my guess.
>>>
>>> Does it answer your doubts?
>>>
>>> On Wed, May 10, 2017 at 9:35
>>> PM,
>>> Pat Haley <phaley at mit.edu
>>> <mailto:phaley at mit.edu> <phaley at mit.edu>> wrote:
>>>
>>>
>>> Without the oflag=sync and
>>> only
>>> a single test of each, the
>>> FUSE
>>> is going faster than NFS:
>>>
>>> FUSE:
>>> mseas-data2(dri_nascar)% dd
>>> if=/dev/zero count=4096
>>> bs=1048576 of=zeros.txt
>>> conv=sync
>>> 4096+0 records in
>>> 4096+0 records out
>>> 4294967296 bytes (4.3 GB)
>>> copied, 7.46961 s, 575 MB/s
>>>
>>>
>>> NFS
>>> mseas-data2(HYCOM)% dd
>>> if=/dev/zero count=4096
>>> bs=1048576 of=zeros.txt
>>> conv=sync
>>> 4096+0 records in
>>> 4096+0 records out
>>> 4294967296 bytes (4.3 GB)
>>> copied, 11.4264 s, 376 MB/s
>>>
>>>
>>>
>>> On 05/10/2017 11:53 AM,
>>> Pranith
>>> Kumar Karampuri wrote:
>>>
>>> Could you let me know the
>>> speed without oflag=sync
>>> on
>>> both the mounts? No need
>>> to
>>> collect profiles.
>>>
>>> On Wed, May 10, 2017 at
>>> 9:17
>>> PM, Pat Haley
>>> <phaley at mit.edu
>>> <mailto:phaley at mit.edu> <phaley at mit.edu>>
>>> wrote:
>>>
>>>
>>> Here is what I see
>>> now:
>>>
>>> [root at mseas-data2 ~]#
>>> gluster volume info
>>>
>>> Volume Name:
>>> data-volume
>>> Type: Distribute
>>> Volume ID:
>>> c162161e-2a2d-4dac-b015-f31fd89ceb18
>>> Status: Started
>>> Number of Bricks: 2
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1:
>>> mseas-data2:/mnt/brick1
>>> Brick2:
>>> mseas-data2:/mnt/brick2
>>> Options Reconfigured:
>>> diagnostics.count-fop-hits:
>>> on
>>> diagnostics.latency-measurement:
>>> on
>>> nfs.exports-auth-enable:
>>> on
>>> diagnostics.brick-sys-log-level:
>>> WARNING
>>> performance.readdir-ahead:
>>> on
>>> nfs.disable: on
>>> nfs.export-volumes:
>>> off
>>>
>>>
>>>
>>> On 05/10/2017 11:44
>>> AM,
>>> Pranith Kumar
>>> Karampuri
>>> wrote:
>>>
>>> Is this the volume
>>> info
>>> you have?
>>>
>>> >/[root at
>>> >mseas-data2
>>> <http://www.gluster.org/mailman/listinfo/gluster-users> <http://www.gluster.org/mailman/listinfo/gluster-users>
>>> ~]# gluster volume
>>> info
>>> />//>/Volume Name:
>>> data-volume />/Type:
>>> Distribute />/Volume
>>> ID:
>>> c162161e-2a2d-4dac-b015-f31fd89ceb18
>>> />/Status: Started
>>> />/Number
>>> of Bricks: 2
>>> />/Transport-type:
>>> tcp
>>> />/Bricks: />/Brick1:
>>> mseas-data2:/mnt/brick1
>>> />/Brick2:
>>> mseas-data2:/mnt/brick2
>>> />/Options
>>> Reconfigured:
>>> />/performance.readdir-ahead:
>>> on />/nfs.disable: on
>>> />/nfs.export-volumes:
>>> off
>>> /
>>> I copied this from
>>> old
>>> thread from 2016.
>>> This
>>> is
>>> distribute volume.
>>> Did
>>> you change any of the
>>> options in between?
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley
>>> Email:phaley at mit.edu
>>> <mailto:phaley at mit.edu> <phaley at mit.edu>
>>> Center for Ocean
>>> Engineering
>>> Phone: (617) 253-6824
>>> Dept. of Mechanical
>>> Engineering
>>> Fax: (617) 253-8125
>>> MIT, Room
>>> 5-213http://web.mit.edu/phaley/www/
>>> 77 Massachusetts
>>> Avenue
>>> Cambridge, MA
>>> 02139-4301
>>>
>>> --
>>> Pranith
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley
>>> Email:phaley at mit.edu
>>> <mailto:phaley at mit.edu> <phaley at mit.edu>
>>> Center for Ocean
>>> Engineering
>>> Phone: (617) 253-6824
>>> Dept. of Mechanical
>>> Engineering
>>> Fax: (617) 253-8125
>>> MIT, Room
>>> 5-213http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>> --
>>> Pranith
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley
>>> Email:phaley at mit.edu
>>> <mailto:phaley at mit.edu> <phaley at mit.edu>
>>> Center for Ocean Engineering
>>> Phone:
>>> (617) 253-6824
>>> Dept. of Mechanical Engineering
>>> Fax:
>>> (617) 253-8125
>>> MIT, Room
>>> 5-213http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>> --
>>> Pranith
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley
>>> Email:phaley at mit.edu
>>> <mailto:phaley at mit.edu> <phaley at mit.edu>
>>> Center for Ocean Engineering
>>> Phone:
>>> (617) 253-6824
>>> Dept. of Mechanical Engineering
>>> Fax:
>>> (617) 253-8125
>>> MIT, Room
>>> 5-213http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>> --
>>> Pranith
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley
>>> Email:phaley at mit.edu
>>> <mailto:phaley at mit.edu> <phaley at mit.edu>
>>> Center for Ocean Engineering Phone:
>>> (617)
>>> 253-6824
>>> Dept. of Mechanical Engineering Fax:
>>> (617)
>>> 253-8125
>>> MIT, Room
>>> 5-213http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>> --
>>> Pranith
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley
>>> Email:phaley at mit.edu
>>> <mailto:phaley at mit.edu> <phaley at mit.edu>
>>> Center for Ocean Engineering Phone:
>>> (617)
>>> 253-6824
>>> Dept. of Mechanical Engineering Fax:
>>> (617)
>>> 253-8125
>>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>> --
>>> Pranith
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley
>>> Email:phaley at mit.edu
>>> <mailto:phaley at mit.edu> <phaley at mit.edu>
>>> Center for Ocean Engineering Phone: (617)
>>> 253-6824
>>> Dept. of Mechanical Engineering Fax: (617)
>>> 253-8125
>>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>>
>>>
>>>
>>> --
>>> Pranith
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley Email:phaley at mit.edu
>>> <mailto:phaley at mit.edu> <phaley at mit.edu>
>>> Center for Ocean Engineering Phone: (617) 253-6824
>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>>
>>>
>>>
>>> --
>>> Pranith
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley Email:phaley at mit.edu
>>> <mailto:phaley at mit.edu> <phaley at mit.edu>
>>> Center for Ocean Engineering Phone: (617) 253-6824
>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>>
>>>
>>>
>>> --
>>> Pranith
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley Email: phaley at mit.edu
>>> Center for Ocean Engineering Phone: (617) 253-6824
>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>>
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley Email: phaley at mit.edu
>>> Center for Ocean Engineering Phone: (617) 253-6824
>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>>
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley Email: phaley at mit.edu
>>> Center for Ocean Engineering Phone: (617) 253-6824
>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>>
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley Email: phaley at mit.edu
>>> Center for Ocean Engineering Phone: (617) 253-6824
>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>>
>>>
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley Email: phaley at mit.edu
>>> Center for Ocean Engineering Phone: (617) 253-6824
>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing listGluster-users at gluster.orghttp://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley Email: phaley at mit.edu
>>> Center for Ocean Engineering Phone: (617) 253-6824
>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>>
>>
>>
>> --
>> Pranith
>>
>
>
>
> --
> Pranith
>
>
> --
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley Email: phaley at mit.edu
> Center for Ocean Engineering Phone: (617) 253-6824
> Dept. of Mechanical Engineering Fax: (617) 253-8125
> MIT, Room 5-213 http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA 02139-4301
>
>
--
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170627/ecf5e496/attachment.html>
More information about the Gluster-users
mailing list