[Gluster-users] Slow write times to gluster disk

Pat Haley phaley at mit.edu
Wed May 10 14:32:48 UTC 2017


Hi,

We finally managed to do the dd tests for an NFS-mounted gluster file 
system.  The profile results during that test are in

http://mseas.mit.edu/download/phaley/GlusterUsers/profile_gluster_nfs_test

The summary of the dd tests are

  * writing to gluster disk mounted with fuse:  5 Mb/s
  * writing to gluster disk mounted with nfs:  200 Mb/s

Pat


On 05/05/2017 08:11 PM, Pat Haley wrote:
>
> Hi,
>
> We redid the dd tests (this time using conv=sync oflag=sync to avoid 
> caching questions).  The profile results are in
>
> http://mseas.mit.edu/download/phaley/GlusterUsers/profile_gluster_fuse_test
>
>
> On 05/05/2017 12:47 PM, Ravishankar N wrote:
>> On 05/05/2017 08:42 PM, Pat Haley wrote:
>>>
>>> Hi Pranith,
>>>
>>> I presume you are asking for some version of the profile data that 
>>> just shows the dd test (or a repeat of the dd test).  If yes, how do 
>>> I extract just that data?
>> Yes, that is what he is asking for. Just clear the existing profile 
>> info using `gluster volume profile volname clear` and run the dd test 
>> once. Then when you run profile info again, it should just give you 
>> the stats for the dd test.
>>>
>>> Thanks
>>>
>>> Pat
>>>
>>>
>>>
>>> On 05/05/2017 10:58 AM, Pranith Kumar Karampuri wrote:
>>>> hi Pat,
>>>>       Let us concentrate on the performance numbers part for now. 
>>>> We will look at the permissions one after this?
>>>>
>>>> As per the profile info, only 2.6% of the work-load is writes. 
>>>> There are too many Lookups.
>>>>
>>>> Would it be possible to get the data for just the dd test you were 
>>>> doing earlier?
>>>>
>>>>
>>>> On Fri, May 5, 2017 at 8:14 PM, Pat Haley <phaley at mit.edu 
>>>> <mailto:phaley at mit.edu>> wrote:
>>>>
>>>>
>>>>     Hi Pranith & Ravi,
>>>>
>>>>     A couple of quick questions
>>>>
>>>>     We have profile turned on. Are there specific queries we should
>>>>     make that would help debug our configuration?  (The default
>>>>     profile info was previously sent in
>>>>     http://lists.gluster.org/pipermail/gluster-users/2017-May/030840.html
>>>>     <http://lists.gluster.org/pipermail/gluster-users/2017-May/030840.html>
>>>>     but I'm not sure if that is what you were looking for.)
>>>>
>>>>     We also started to do a test on serving gluster over NFS.  We
>>>>     rediscovered an issue we previously reported (
>>>>     http://lists.gluster.org/pipermail/gluster-users/2016-September/028289.html
>>>>     <http://lists.gluster.org/pipermail/gluster-users/2016-September/028289.html>
>>>>     ) in that the NFS mounted version was ignoring the group write
>>>>     permissions.  What specific information would be useful in
>>>>     debugging this?
>>>>
>>>>     Thanks
>>>>
>>>>     Pat
>>>>
>>>>
>>>>
>>>>     On 04/14/2017 03:01 AM, Ravishankar N wrote:
>>>>>     On 04/14/2017 12:20 PM, Pranith Kumar Karampuri wrote:
>>>>>>
>>>>>>
>>>>>>     On Sat, Apr 8, 2017 at 10:28 AM, Ravishankar N
>>>>>>     <ravishankar at redhat.com <mailto:ravishankar at redhat.com>> wrote:
>>>>>>
>>>>>>         Hi Pat,
>>>>>>
>>>>>>         I'm assuming you are using gluster native (fuse mount).
>>>>>>         If it helps, you could try mounting it via gluster NFS
>>>>>>         (gnfs) and then see if there is an improvement in speed.
>>>>>>         Fuse mounts are slower than gnfs mounts but you get the
>>>>>>         benefit of avoiding a single point of failure. Unlike
>>>>>>         fuse mounts, if the gluster node containing the gnfs
>>>>>>         server goes down, all mounts done using that node will
>>>>>>         fail). For fuse mounts, you could try tweaking the
>>>>>>         write-behind xlator settings to see if it helps. See the
>>>>>>         performance.write-behind and
>>>>>>         performance.write-behind-window-size options in `gluster
>>>>>>         volume set help`. Of course, even for gnfs mounts, you
>>>>>>         can achieve fail-over by using CTDB.
>>>>>>
>>>>>>
>>>>>>     Ravi,
>>>>>>           Do you have any data that suggests fuse mounts are
>>>>>>     slower than gNFS servers?
>>>>>     I have heard anecdotal evidence time and again on the ML and
>>>>>     IRC, which is why I wanted to compare it with NFS numbers on
>>>>>     his setup.
>>>>>>
>>>>>>     Pat,
>>>>>>           I see that I am late to the thread, but do you happen
>>>>>>     to have "profile info" of the workload?
>>>>>>
>>>>>>     You can follow
>>>>>>     https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/
>>>>>>     <https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/>
>>>>>>     to get the information.
>>>>>     Yeah, Let's see if profile info shows up anything interesting.
>>>>>     -Ravi
>>>>>>
>>>>>>
>>>>>>         Thanks,
>>>>>>         Ravi
>>>>>>
>>>>>>
>>>>>>         On 04/08/2017 12:07 AM, Pat Haley wrote:
>>>>>>>
>>>>>>>         Hi,
>>>>>>>
>>>>>>>         We noticed a dramatic slowness when writing to a gluster
>>>>>>>         disk when compared to writing to an NFS disk.
>>>>>>>         Specifically when using dd (data duplicator) to write a
>>>>>>>         4.3 GB file of zeros:
>>>>>>>
>>>>>>>           * on NFS disk (/home): 9.5 Gb/s
>>>>>>>           * on gluster disk (/gdata): 508 Mb/s
>>>>>>>
>>>>>>>         The gluser disk is 2 bricks joined together, no
>>>>>>>         replication or anything else. The hardware is
>>>>>>>         (literally) the same:
>>>>>>>
>>>>>>>           * one server with 70 hard disks  and a hardware RAID card.
>>>>>>>           * 4 disks in a RAID-6 group (the NFS disk)
>>>>>>>           * 32 disks in a RAID-6 group (the max allowed by the
>>>>>>>             card, /mnt/brick1)
>>>>>>>           * 32 disks in another RAID-6 group (/mnt/brick2)
>>>>>>>           * 2 hot spare
>>>>>>>
>>>>>>>         Some additional information and more tests results
>>>>>>>         (after changing the log level):
>>>>>>>
>>>>>>>         glusterfs 3.7.11 built on Apr 27 2016 14:09:22
>>>>>>>         CentOS release 6.8 (Final)
>>>>>>>         RAID bus controller: LSI Logic / Symbios Logic MegaRAID
>>>>>>>         SAS-3 3108 [Invader] (rev 02)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>         *Create the file to /gdata (gluster)*
>>>>>>>         [root at mseas-data2 gdata]# dd if=/dev/zero
>>>>>>>         of=/gdata/zero1 bs=1M count=1000
>>>>>>>         1000+0 records in
>>>>>>>         1000+0 records out
>>>>>>>         1048576000 bytes (1.0 GB) copied, 1.91876 s, *546 MB/s*
>>>>>>>
>>>>>>>         *Create the file to /home (ext4)*
>>>>>>>         [root at mseas-data2 gdata]# dd if=/dev/zero of=/home/zero1
>>>>>>>         bs=1M count=1000
>>>>>>>         1000+0 records in
>>>>>>>         1000+0 records out
>>>>>>>         1048576000 bytes (1.0 GB) copied, 0.686021 s, *1.5 GB/s
>>>>>>>         - *3 times as fast*
>>>>>>>
>>>>>>>
>>>>>>>         Copy from /gdata to /gdata (gluster to gluster)
>>>>>>>         *[root at mseas-data2 gdata]# dd if=/gdata/zero1
>>>>>>>         of=/gdata/zero2
>>>>>>>         2048000+0 records in
>>>>>>>         2048000+0 records out
>>>>>>>         1048576000 bytes (1.0 GB) copied, 101.052 s, *10.4 MB/s*
>>>>>>>         - realllyyy slooowww
>>>>>>>
>>>>>>>
>>>>>>>         *Copy from /gdata to /gdata* *2nd time *(gluster to
>>>>>>>         gluster)**
>>>>>>>         [root at mseas-data2 gdata]# dd if=/gdata/zero1 of=/gdata/zero2
>>>>>>>         2048000+0 records in
>>>>>>>         2048000+0 records out
>>>>>>>         1048576000 bytes (1.0 GB) copied, 92.4904 s, *11.3 MB/s*
>>>>>>>         - realllyyy slooowww again
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>         *Copy from /home to /home (ext4 to ext4)*
>>>>>>>         [root at mseas-data2 gdata]# dd if=/home/zero1 of=/home/zero2
>>>>>>>         2048000+0 records in
>>>>>>>         2048000+0 records out
>>>>>>>         1048576000 bytes (1.0 GB) copied, 3.53263 s, *297 MB/s
>>>>>>>         *30 times as fast
>>>>>>>
>>>>>>>
>>>>>>>         *Copy from /home to /home (ext4 to ext4)*
>>>>>>>         [root at mseas-data2 gdata]# dd if=/home/zero1 of=/home/zero3
>>>>>>>         2048000+0 records in
>>>>>>>         2048000+0 records out
>>>>>>>         1048576000 bytes (1.0 GB) copied, 4.1737 s, *251 MB/s* -
>>>>>>>         30 times as fast
>>>>>>>
>>>>>>>
>>>>>>>         As a test, can we copy data directly to the xfs
>>>>>>>         mountpoint (/mnt/brick1) and bypass gluster?
>>>>>>>
>>>>>>>
>>>>>>>         Any help you could give us would be appreciated.
>>>>>>>
>>>>>>>         Thanks
>>>>>>>
>>>>>>>         -- 
>>>>>>>
>>>>>>>         -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>         Pat Haley                          Email:phaley at mit.edu <mailto:phaley at mit.edu>
>>>>>>>         Center for Ocean Engineering       Phone:  (617) 253-6824
>>>>>>>         Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>>>>>>         MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>>>>         77 Massachusetts Avenue
>>>>>>>         Cambridge, MA  02139-4301
>>>>>>>
>>>>>>>         _______________________________________________
>>>>>>>         Gluster-users mailing list
>>>>>>>         Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>>>>>         http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>         <http://lists.gluster.org/mailman/listinfo/gluster-users>
>>>>>>
>>>>>>         _______________________________________________
>>>>>>         Gluster-users mailing list Gluster-users at gluster.org
>>>>>>         <mailto:Gluster-users at gluster.org>
>>>>>>         http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>         <http://lists.gluster.org/mailman/listinfo/gluster-users> 
>>>>>>
>>>>>>     -- 
>>>>>>     Pranith
>>>>>
>>>>     -- 
>>>>
>>>>     -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>     Pat Haley                          Email:phaley at mit.edu <mailto:phaley at mit.edu>
>>>>     Center for Ocean Engineering       Phone:  (617) 253-6824
>>>>     Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>>>     MIT, Room 5-213http://web.mit.edu/phaley/www/
>>>>     77 Massachusetts Avenue
>>>>     Cambridge, MA  02139-4301
>>>>
>>>> -- 
>>>> Pranith
>>> -- 
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley                          Email:phaley at mit.edu
>>> Center for Ocean Engineering       Phone:  (617) 253-6824
>>> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA  02139-4301
>>
> -- 
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley                          Email:phaley at mit.edu
> Center for Ocean Engineering       Phone:  (617) 253-6824
> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
> MIT, Room 5-213http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA  02139-4301
-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170510/633585c5/attachment.html>


More information about the Gluster-users mailing list