[Gluster-users] Extremely slow file listing in folders with many files
Artem Russakovskii
archon810 at gmail.com
Thu Apr 30 17:54:14 UTC 2020
getfattr -d -m. -e hex .
# file: .
trusted.afr.SNIP_data1-client-0=0x000000000000000000000000
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0x44b2db00267a47508b2a8a921f20e0f5
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
trusted.glusterfs.dht.mds=0x00000000
Sincerely,
Artem
--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | @ArtemR <http://twitter.com/ArtemR>
On Thu, Apr 30, 2020 at 9:05 AM Felix Kölzow <felix.koelzow at gmx.de> wrote:
> Dear Artem,
>
> sry for the noise, since you already provide the xfs_info.
>
> Could you provide the output of
>
>
> getfattr -d -m. -e hex /DirectoryPathOfInterest_onTheBrick/
>
>
> Felix
>
> On 30/04/2020 18:01, Felix Kölzow wrote:
>
> Dear Artem,
>
> can you also provide some information w.r.t your xfs filesystem, i.e.
> xfs_info of your block device?
>
>
> Regards,
>
> Felix
> On 30/04/2020 17:27, Artem Russakovskii wrote:
>
> Hi Strahil, in the original email I included both the times for the first
> and subsequent reads on the fuse mounted gluster volume as well as the xfs
> filesystem the gluster data resides on (this is the brick, right?).
>
> On Thu, Apr 30, 2020, 7:44 AM Strahil Nikolov <hunter86_bg at yahoo.com>
> wrote:
>
>> On April 30, 2020 4:24:23 AM GMT+03:00, Artem Russakovskii <
>> archon810 at gmail.com> wrote:
>> >Hi all,
>> >
>> >We have 500GB and 10TB 4x1 replicate xfs-based gluster volumes, and the
>> >10TB one especially is extremely slow to do certain things with (and
>> >has
>> >been since gluster 3.x when we started). We're currently on 5.13.
>> >
>> >The number of files isn't even what I'd consider that great - under
>> >100k
>> >per dir.
>> >
>> >Here are some numbers to look at:
>> >
>> >On gluster volume in a dir of 45k files:
>> >The first time
>> >
>> >time find | wc -l
>> >45423
>> >real 8m44.819s
>> >user 0m0.459s
>> >sys 0m0.998s
>> >
>> >And again
>> >
>> >time find | wc -l
>> >45423
>> >real 0m34.677s
>> >user 0m0.291s
>> >sys 0m0.754s
>> >
>> >
>> >If I run the same operation on the xfs block device itself:
>> >The first time
>> >
>> >time find | wc -l
>> >45423
>> >real 0m13.514s
>> >user 0m0.144s
>> >sys 0m0.501s
>> >
>> >And again
>> >
>> >time find | wc -l
>> >45423
>> >real 0m0.197s
>> >user 0m0.088s
>> >sys 0m0.106s
>> >
>> >
>> >I'd expect a performance difference here but just as it was several
>> >years
>> >ago when we started with gluster, it's still huge, and simple file
>> >listings
>> >are incredibly slow.
>> >
>> >At the time, the team was looking to do some optimizations, but I'm not
>> >sure this has happened.
>> >
>> >What can we do to try to improve performance?
>> >
>> >Thank you.
>> >
>> >
>> >
>> >Some setup values follow.
>> >
>> >xfs_info /mnt/SNIP_block1
>> >meta-data=/dev/sdc isize=512 agcount=103,
>> >agsize=26214400
>> >blks
>> > = sectsz=512 attr=2, projid32bit=1
>> > = crc=1 finobt=1, sparse=0, rmapbt=0
>> > = reflink=0
>> >data = bsize=4096 blocks=2684354560,
>> >imaxpct=25
>> > = sunit=0 swidth=0 blks
>> >naming =version 2 bsize=4096 ascii-ci=0, ftype=1
>> >log =internal log bsize=4096 blocks=51200, version=2
>> > = sectsz=512 sunit=0 blks, lazy-count=1
>> >realtime =none extsz=4096 blocks=0, rtextents=0
>> >
>> >Volume Name: SNIP_data1
>> >Type: Replicate
>> >Volume ID: SNIP
>> >Status: Started
>> >Snapshot Count: 0
>> >Number of Bricks: 1 x 4 = 4
>> >Transport-type: tcp
>> >Bricks:
>> >Brick1: nexus2:/mnt/SNIP_block1/SNIP_data1
>> >Brick2: forge:/mnt/SNIP_block1/SNIP_data1
>> >Brick3: hive:/mnt/SNIP_block1/SNIP_data1
>> >Brick4: citadel:/mnt/SNIP_block1/SNIP_data1
>> >Options Reconfigured:
>> >cluster.quorum-count: 1
>> >cluster.quorum-type: fixed
>> >network.ping-timeout: 5
>> >network.remote-dio: enable
>> >performance.rda-cache-limit: 256MB
>> >performance.readdir-ahead: on
>> >performance.parallel-readdir: on
>> >network.inode-lru-limit: 500000
>> >performance.md-cache-timeout: 600
>> >performance.cache-invalidation: on
>> >performance.stat-prefetch: on
>> >features.cache-invalidation-timeout: 600
>> >features.cache-invalidation: on
>> >cluster.readdir-optimize: on
>> >performance.io-thread-count: 32
>> >server.event-threads: 4
>> >client.event-threads: 4
>> >performance.read-ahead: off
>> >cluster.lookup-optimize: on
>> >performance.cache-size: 1GB
>> >cluster.self-heal-daemon: enable
>> >transport.address-family: inet
>> >nfs.disable: on
>> >performance.client-io-threads: on
>> >cluster.granular-entry-heal: enable
>> >cluster.data-self-heal-algorithm: full
>> >
>> >Sincerely,
>> >Artem
>> >
>> >--
>> >Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>> ><http://www.apkmirror.com/>, Illogical Robot LLC
>> >beerpla.net | @ArtemR <http://twitter.com/ArtemR>
>>
>> Hi Artem,
>>
>> Have you checked the same on brick level ? How big is the difference ?
>>
>> Best Regards,
>> Strahil Nikolov
>>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing listGluster-users at gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing listGluster-users at gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200430/e5344898/attachment.html>
More information about the Gluster-users
mailing list