[Gluster-users] Slow write times to gluster disk
Pat Haley
phaley at mit.edu
Sat May 6 00:11:26 UTC 2017
Hi,
We redid the dd tests (this time using conv=sync oflag=sync to avoid
caching questions). The profile results are in
http://mseas.mit.edu/download/phaley/GlusterUsers/profile_gluster_fuse_test
On 05/05/2017 12:47 PM, Ravishankar N wrote:
> On 05/05/2017 08:42 PM, Pat Haley wrote:
>>
>> Hi Pranith,
>>
>> I presume you are asking for some version of the profile data that
>> just shows the dd test (or a repeat of the dd test). If yes, how do
>> I extract just that data?
> Yes, that is what he is asking for. Just clear the existing profile
> info using `gluster volume profile volname clear` and run the dd test
> once. Then when you run profile info again, it should just give you
> the stats for the dd test.
>>
>> Thanks
>>
>> Pat
>>
>>
>>
>> On 05/05/2017 10:58 AM, Pranith Kumar Karampuri wrote:
>>> hi Pat,
>>> Let us concentrate on the performance numbers part for now. We
>>> will look at the permissions one after this?
>>>
>>> As per the profile info, only 2.6% of the work-load is writes. There
>>> are too many Lookups.
>>>
>>> Would it be possible to get the data for just the dd test you were
>>> doing earlier?
>>>
>>>
>>> On Fri, May 5, 2017 at 8:14 PM, Pat Haley <phaley at mit.edu
>>> <mailto:phaley at mit.edu>> wrote:
>>>
>>>
>>> Hi Pranith & Ravi,
>>>
>>> A couple of quick questions
>>>
>>> We have profile turned on. Are there specific queries we should
>>> make that would help debug our configuration? (The default
>>> profile info was previously sent in
>>> http://lists.gluster.org/pipermail/gluster-users/2017-May/030840.html
>>> <http://lists.gluster.org/pipermail/gluster-users/2017-May/030840.html>
>>> but I'm not sure if that is what you were looking for.)
>>>
>>> We also started to do a test on serving gluster over NFS. We
>>> rediscovered an issue we previously reported (
>>> http://lists.gluster.org/pipermail/gluster-users/2016-September/028289.html
>>> <http://lists.gluster.org/pipermail/gluster-users/2016-September/028289.html>
>>> ) in that the NFS mounted version was ignoring the group write
>>> permissions. What specific information would be useful in
>>> debugging this?
>>>
>>> Thanks
>>>
>>> Pat
>>>
>>>
>>>
>>> On 04/14/2017 03:01 AM, Ravishankar N wrote:
>>>> On 04/14/2017 12:20 PM, Pranith Kumar Karampuri wrote:
>>>>>
>>>>>
>>>>> On Sat, Apr 8, 2017 at 10:28 AM, Ravishankar N
>>>>> <ravishankar at redhat.com <mailto:ravishankar at redhat.com>> wrote:
>>>>>
>>>>> Hi Pat,
>>>>>
>>>>> I'm assuming you are using gluster native (fuse mount). If
>>>>> it helps, you could try mounting it via gluster NFS (gnfs)
>>>>> and then see if there is an improvement in speed. Fuse
>>>>> mounts are slower than gnfs mounts but you get the benefit
>>>>> of avoiding a single point of failure. Unlike fuse mounts,
>>>>> if the gluster node containing the gnfs server goes down,
>>>>> all mounts done using that node will fail). For fuse
>>>>> mounts, you could try tweaking the write-behind xlator
>>>>> settings to see if it helps. See the
>>>>> performance.write-behind and
>>>>> performance.write-behind-window-size options in `gluster
>>>>> volume set help`. Of course, even for gnfs mounts, you can
>>>>> achieve fail-over by using CTDB.
>>>>>
>>>>>
>>>>> Ravi,
>>>>> Do you have any data that suggests fuse mounts are
>>>>> slower than gNFS servers?
>>>> I have heard anecdotal evidence time and again on the ML and
>>>> IRC, which is why I wanted to compare it with NFS numbers on
>>>> his setup.
>>>>>
>>>>> Pat,
>>>>> I see that I am late to the thread, but do you happen to
>>>>> have "profile info" of the workload?
>>>>>
>>>>> You can follow
>>>>> https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/
>>>>> <https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/>
>>>>> to get the information.
>>>> Yeah, Let's see if profile info shows up anything interesting.
>>>> -Ravi
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Ravi
>>>>>
>>>>>
>>>>> On 04/08/2017 12:07 AM, Pat Haley wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We noticed a dramatic slowness when writing to a gluster
>>>>>> disk when compared to writing to an NFS disk.
>>>>>> Specifically when using dd (data duplicator) to write a
>>>>>> 4.3 GB file of zeros:
>>>>>>
>>>>>> * on NFS disk (/home): 9.5 Gb/s
>>>>>> * on gluster disk (/gdata): 508 Mb/s
>>>>>>
>>>>>> The gluser disk is 2 bricks joined together, no
>>>>>> replication or anything else. The hardware is (literally)
>>>>>> the same:
>>>>>>
>>>>>> * one server with 70 hard disks and a hardware RAID card.
>>>>>> * 4 disks in a RAID-6 group (the NFS disk)
>>>>>> * 32 disks in a RAID-6 group (the max allowed by the
>>>>>> card, /mnt/brick1)
>>>>>> * 32 disks in another RAID-6 group (/mnt/brick2)
>>>>>> * 2 hot spare
>>>>>>
>>>>>> Some additional information and more tests results (after
>>>>>> changing the log level):
>>>>>>
>>>>>> glusterfs 3.7.11 built on Apr 27 2016 14:09:22
>>>>>> CentOS release 6.8 (Final)
>>>>>> RAID bus controller: LSI Logic / Symbios Logic MegaRAID
>>>>>> SAS-3 3108 [Invader] (rev 02)
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Create the file to /gdata (gluster)*
>>>>>> [root at mseas-data2 gdata]# dd if=/dev/zero of=/gdata/zero1
>>>>>> bs=1M count=1000
>>>>>> 1000+0 records in
>>>>>> 1000+0 records out
>>>>>> 1048576000 bytes (1.0 GB) copied, 1.91876 s, *546 MB/s*
>>>>>>
>>>>>> *Create the file to /home (ext4)*
>>>>>> [root at mseas-data2 gdata]# dd if=/dev/zero of=/home/zero1
>>>>>> bs=1M count=1000
>>>>>> 1000+0 records in
>>>>>> 1000+0 records out
>>>>>> 1048576000 bytes (1.0 GB) copied, 0.686021 s, *1.5 GB/s -
>>>>>> *3 times as fast*
>>>>>>
>>>>>>
>>>>>> Copy from /gdata to /gdata (gluster to gluster)
>>>>>> *[root at mseas-data2 gdata]# dd if=/gdata/zero1 of=/gdata/zero2
>>>>>> 2048000+0 records in
>>>>>> 2048000+0 records out
>>>>>> 1048576000 bytes (1.0 GB) copied, 101.052 s, *10.4 MB/s*
>>>>>> - realllyyy slooowww
>>>>>>
>>>>>>
>>>>>> *Copy from /gdata to /gdata* *2nd time *(gluster to
>>>>>> gluster)**
>>>>>> [root at mseas-data2 gdata]# dd if=/gdata/zero1 of=/gdata/zero2
>>>>>> 2048000+0 records in
>>>>>> 2048000+0 records out
>>>>>> 1048576000 bytes (1.0 GB) copied, 92.4904 s, *11.3 MB/s*
>>>>>> - realllyyy slooowww again
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Copy from /home to /home (ext4 to ext4)*
>>>>>> [root at mseas-data2 gdata]# dd if=/home/zero1 of=/home/zero2
>>>>>> 2048000+0 records in
>>>>>> 2048000+0 records out
>>>>>> 1048576000 bytes (1.0 GB) copied, 3.53263 s, *297 MB/s
>>>>>> *30 times as fast
>>>>>>
>>>>>>
>>>>>> *Copy from /home to /home (ext4 to ext4)*
>>>>>> [root at mseas-data2 gdata]# dd if=/home/zero1 of=/home/zero3
>>>>>> 2048000+0 records in
>>>>>> 2048000+0 records out
>>>>>> 1048576000 bytes (1.0 GB) copied, 4.1737 s, *251 MB/s* -
>>>>>> 30 times as fast
>>>>>>
>>>>>>
>>>>>> As a test, can we copy data directly to the xfs
>>>>>> mountpoint (/mnt/brick1) and bypass gluster?
>>>>>>
>>>>>>
>>>>>> Any help you could give us would be appreciated.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> --
>>>>>>
>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>> Pat Haley Email:phaley at mit.edu <mailto:phaley at mit.edu>
>>>>>> Center for Ocean Engineering Phone: (617) 253-6824
>>>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>>>>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>>> 77 Massachusetts Avenue
>>>>>> Cambridge, MA 02139-4301
>>>>>>
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list Gluster-users at gluster.org
>>>>> <mailto:Gluster-users at gluster.org>
>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>>>
>>>>> --
>>>>> Pranith
>>>>
>>> --
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley Email:phaley at mit.edu <mailto:phaley at mit.edu>
>>> Center for Ocean Engineering Phone: (617) 253-6824
>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA 02139-4301
>>>
>>> --
>>> Pranith
>> --
>>
>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> Pat Haley Email:phaley at mit.edu
>> Center for Ocean Engineering Phone: (617) 253-6824
>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>> 77 Massachusetts Avenue
>> Cambridge, MA 02139-4301
>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: phaley at mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170505/955eb833/attachment-0001.html>
More information about the Gluster-users
mailing list