[Gluster-users] Very slow roaming profiles on top of glusterfs

Mon Sep 14 15:38:42 UTC 2015

See below....

On Mon, Sep 14, 2015 at 11:06 AM, Ben Turner <bturner at redhat.com> wrote:
> ----- Original Message -----
>> From: "Diego Remolina" <dijuremo at gmail.com>
>> To: "Alex Crow" <acrow at integrafin.co.uk>
>> Cc: gluster-users at gluster.org
>> Sent: Monday, September 14, 2015 9:26:17 AM
>> Subject: Re: [Gluster-users] Very slow roaming profiles on top of glusterfs
>>
>> Hi Alex,
>>
>> Thanks for the reply, I was aware of the performance issues with small
>> files, but never expected an order of magnitude slower. I understand
>> some improvements were made to 3.7.x to help with low small file
>> performance, however I did not see any big changes after upgrading
>> from 3.6.x to 3.7.x.
>>
>> http://www.gluster.org/community/documentation/index.php/Features/Feature_Smallfile_Perf
>>
>> And the ssd metadata support feature seems to have not had any changes
>> since September 2014:
>>
>> https://forge.gluster.org/gluster-meta-data-on-ssd
>>
>> Am I just totally out of luck with gluster for now?
>
> Are you using glusterFS mounts or SMB mounts?  As for SMB mounts we are working VERY hard to improve metadata / smallfile performance but as it sits right now we are limited by the number of lookup / stat calls that are issued.  When we can reduce the number of lookups and prefetch the xattrs that SMB / windows needs(I am working on the stat prefetch but don't have a testable solution yet) I expect to see a vast perf improvement but I don't have an ETA for you.
>

The clients are all Windows machines so they mount from the server
using SMB mounts. We do have the gluster vfs plugin for samba enabled
on all shares. Using the VFS plugin has around a 10% improvment on
large file transfers vs accessing the fuse mounted data directory.

Server automounts it's gluster storage via fuse on /export

[root at ysmha01 ~]# mount | grep export
10.0.1.6:/export on /export type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072)

Samba shares, including home directories have the following configuration:

   vfs objects = glusterfs
   glusterfs:loglevel = 7
   glusterfs:logfile = /var/log/samba/glusterfs-homes.log
   glusterfs:volume = export

Thanks,

Diego

> On the glusterFS side I see ~300% improvement in smallfile create performance between 3.6 and 3.7.  Try setting:
>
> gluster volume set testvol server.event-threads 4
> gluster volume set testvol client.event-threads 4
> gluster volume set testvol cluster.lookup-optimize on
>
> Unfortunately WRT to metadata operations a fix went in that has negatively affected performance:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1250241
>
> I used to see about 25k metatdata operations per second, now I am only seeking 6k.  It looks like there is a patch but I don't know if the fix will get us back to the 25k OPs per second, maybe Pranith can comment on expectations for:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1250803
>
> To summarize:
>
> SMB - no ETA for improvement
> GlusterFS smallfile create - 300% increase in my env between 3.6 and 3.7
> GlusterFS metadata - BZ is in POST(patch is submitted) but I am not sure on the ETA of the fix and if the fix will get back to what I was seeing in 3.6
>
> Hope this helps.
>
> -b
>
>>
>> Diego
>>
>> On Mon, Sep 14, 2015 at 8:37 AM, Alex Crow <acrow at integrafin.co.uk> wrote:
>> > Hi Diego,
>> >
>> > I think it's the overhead of fstat() calls. Gluster keeps its metadata on
>> > the bricks themselves, and this has to be looked up for every file access.
>> > For big files this is not an issue as it only happens once, but when
>> > accessing lots of small files this overhead rapidly builds up, the smaller
>> > the file the worse the issue. Profiles do have hundreds of very small
>> > files!
>> >
>> > I was looking to use GlusterFS for generic file sharing as well, but I
>> > noticed the same issue while testing backups from a GlusterFS volume. On
>> > one
>> > vol (scanned 4-bit greyscale images and small PDFs) backups were taking
>> > over
>> > 16 hours whereas with a traditional FS they were completing in just over 1
>> > hour.
>> >
>> > It may be worth trying out one of the distributed filesystems that use a
>> > separate in-memory metadata server. I've tried LizardFS and MooseFS and
>> > they
>> > are both much faster than GlusterFS for small files, although large-file
>> > sequential performance is not as good (but still plenty for a Samba
>> > server).
>> >
>> > Alex
>> >
>> >
>> > On 14/09/15 13:21, Diego Remolina wrote:
>> >>
>> >> Bump...
>> >>
>> >> Anybody has any clues as to how I can try and identify the cause of
>> >> the slowness?
>> >>
>> >> Diego
>> >>
>> >> On Wed, Sep 9, 2015 at 7:42 PM, Diego Remolina <dijuremo at gmail.com> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> I am running two glusterfs servers as replicas. I have a 3rd server
>> >>> which provides quorum. Since gluster was introduced, we have had an
>> >>> issue where windows roaming profiles are extremely slow. The initial
>> >>> setup was done on 3.6.x and since 3.7.x has small file performance
>> >>> improvements, I upgraded to 3.7.3, but that has not helped.
>> >>>
>> >>> It seems that for some reason gluster is very slow when dealing with
>> >>> lots of small files. I am not sure how to really troubleshoot this via
>> >>> samba, but I have come up with other tests that produce rather
>> >>> disconcerting results as shown below.
>> >>>
>> >>> If I run directly on the brick:
>> >>> [root at ysmha01 /]# time ( find
>> >>> /bricks/hdds/brick/home/jgibbs/.winprofile.V2 -type f > /dev/null )
>> >>> real 0m3.683s
>> >>> user 0m0.042s
>> >>> sys 0m0.154s
>> >>>
>> >>> Now running on the gluster volume mounted via fuse:
>> >>> [root at ysmha01 /]# mount | grep export
>> >>> 10.0.1.6:/export on /export type fuse.glusterfs
>> >>> (rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072)
>> >>>
>> >>> [root at ysmha01 /]# time ( find /export/home/jgibbs/.winprofile.V2 -type
>> >>> f > /dev/null )
>> >>> real 0m57.812s
>> >>> user 0m0.118s
>> >>> sys 0m0.374s
>> >>>
>> >>> In general, the time to run the command on this particular user can be
>> >>> up to 2 minutes. If I run the command on the brick first, then it
>> >>> seems the time to run on the mounted gluster volume is lower like in
>> >>> the example above. I assume some caching in preserved.
>> >>>
>> >>> This particular user has 13,216 files in his roaming profile, which
>> >>> adds up to about 452MB of data.
>> >>>
>> >>> The server performance over samba for copying big files (both read and
>> >>> write) is great, I can almost max out the gigabit connections on the
>> >>> desktops.
>> >>>
>> >>> Reading from samba share on the server and writing to local drive:
>> >>> 111MB/s (Copying a 650MB iso file)
>> >>> Reading from local drive and writing to server samba share: 94MB/s
>> >>> (Copying a 3.2GB ISO file)
>> >>>
>> >>> The servers are connected to the network with 10Gbit adapters and also
>> >>> use separate adapters; one 10 Gbit adapter is used for services, and
>> >>> other for the backend storage communication.
>> >>>
>> >>> The servers have hardware raid controllers and the samba shares are on
>> >>> top of an Areca ARC-1882 controller, with a volume made out of 12 2TB
>> >>> drives in raid 6.
>> >>>
>> >>> If you can provide any steps to better troubleshoot this problem and
>> >>> fix the issue, I will really appreciate it.
>> >>>
>> >>> Diego
>> >>>
>> >>> Further details about the machines below:
>> >>>
>> >>> [root at ysmha01 /]# cat /etc/redhat-release
>> >>> CentOS Linux release 7.1.1503 (Core)
>> >>>
>> >>> [root at ysmha01 /]# gluster volume info export
>> >>> Volume Name: export
>> >>> Type: Replicate
>> >>> Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff
>> >>> Status: Started
>> >>> Number of Bricks: 1 x 2 = 2
>> >>> Transport-type: tcp
>> >>> Bricks:
>> >>> Brick1: 10.0.1.7:/bricks/hdds/brick
>> >>> Brick2: 10.0.1.6:/bricks/hdds/brick
>> >>> Options Reconfigured:
>> >>> performance.io-cache: on
>> >>> performance.io-thread-count: 64
>> >>> nfs.disable: on
>> >>> cluster.server-quorum-type: server
>> >>> performance.cache-size: 1024MB
>> >>> server.allow-insecure: on
>> >>> cluster.server-quorum-ratio: 51%
>> >>>
>> >>> Each server has dual Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz with
>> >>> 32GB of memory.
>> >>
>> >> _______________________________________________
>> >> Gluster-users mailing list
>> >> Gluster-users at gluster.org
>> >> http://www.gluster.org/mailman/listinfo/gluster-users
>> >
>> >
>> > --
>> > This message is intended only for the addressee and may contain
>> > confidential information. Unless you are that person, you may not
>> > disclose its contents or use it in any way and are requested to delete
>> > the message along with any attachments and notify us immediately.
>> > "Transact" is operated by Integrated Financial Arrangements plc. 29
>> > Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608
>> > 5300. (Registered office: as above; Registered in England and Wales
>> > under number: 3727592). Authorised and regulated by the Financial
>> > Conduct Authority (entered on the Financial Services Register; no. 190856).
>> >
>> > .
>> >
>> > _______________________________________________
>> > Gluster-users mailing list
>> > Gluster-users at gluster.org
>> > http://www.gluster.org/mailman/listinfo/gluster-users
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>