[Gluster-users] Extremely slow file listing in folders with many files

[Strahil Nikolov]

On April 30, 2020 4:24:23 AM GMT+03:00, Artem Russakovskii <archon810 at gmail.com> wrote:
>Hi all,
>
>We have 500GB and 10TB 4x1 replicate xfs-based gluster volumes, and the
>10TB one especially is extremely slow to do certain things with (and
>has
>been since gluster 3.x when we started). We're currently on 5.13.
>
>The number of files isn't even what I'd consider that great - under
>100k
>per dir.
>
>Here are some numbers to look at:
>
>On gluster volume in a dir of 45k files:
>The first time
>
>time find | wc -l
>45423
>real    8m44.819s
>user    0m0.459s
>sys     0m0.998s
>
>And again
>
>time find | wc -l
>45423
>real    0m34.677s
>user    0m0.291s
>sys     0m0.754s
>
>
>If I run the same operation on the xfs block device itself:
>The first time
>
>time find | wc -l
>45423
>real    0m13.514s
>user    0m0.144s
>sys     0m0.501s
>
>And again
>
>time find | wc -l
>45423
>real    0m0.197s
>user    0m0.088s
>sys     0m0.106s
>
>
>I'd expect a performance difference here but just as it was several
>years
>ago when we started with gluster, it's still huge, and simple file
>listings
>are incredibly slow.
>
>At the time, the team was looking to do some optimizations, but I'm not
>sure this has happened.
>
>What can we do to try to improve performance?
>
>Thank you.
>
>
>
>Some setup values follow.
>
>xfs_info /mnt/SNIP_block1
>meta-data=/dev/sdc               isize=512    agcount=103,
>agsize=26214400
>blks
>         =                       sectsz=512   attr=2, projid32bit=1
>      =                       crc=1        finobt=1, sparse=0, rmapbt=0
>         =                       reflink=0
>data     =                       bsize=4096   blocks=2684354560,
>imaxpct=25
>         =                       sunit=0      swidth=0 blks
>naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
>log      =internal log           bsize=4096   blocks=51200, version=2
>        =                       sectsz=512   sunit=0 blks, lazy-count=1
>realtime =none                   extsz=4096   blocks=0, rtextents=0
>
>Volume Name: SNIP_data1
>Type: Replicate
>Volume ID: SNIP
>Status: Started
>Snapshot Count: 0
>Number of Bricks: 1 x 4 = 4
>Transport-type: tcp
>Bricks:
>Brick1: nexus2:/mnt/SNIP_block1/SNIP_data1
>Brick2: forge:/mnt/SNIP_block1/SNIP_data1
>Brick3: hive:/mnt/SNIP_block1/SNIP_data1
>Brick4: citadel:/mnt/SNIP_block1/SNIP_data1
>Options Reconfigured:
>cluster.quorum-count: 1
>cluster.quorum-type: fixed
>network.ping-timeout: 5
>network.remote-dio: enable
>performance.rda-cache-limit: 256MB
>performance.readdir-ahead: on
>performance.parallel-readdir: on
>network.inode-lru-limit: 500000
>performance.md-cache-timeout: 600
>performance.cache-invalidation: on
>performance.stat-prefetch: on
>features.cache-invalidation-timeout: 600
>features.cache-invalidation: on
>cluster.readdir-optimize: on
>performance.io-thread-count: 32
>server.event-threads: 4
>client.event-threads: 4
>performance.read-ahead: off
>cluster.lookup-optimize: on
>performance.cache-size: 1GB
>cluster.self-heal-daemon: enable
>transport.address-family: inet
>nfs.disable: on
>performance.client-io-threads: on
>cluster.granular-entry-heal: enable
>cluster.data-self-heal-algorithm: full
>
>Sincerely,
>Artem
>
>--
>Founder, Android Police <http://www.androidpolice.com>, APK Mirror
><http://www.apkmirror.com/>, Illogical Robot LLC
>beerpla.net | @ArtemR <http://twitter.com/ArtemR>

Hi Artem,

Have you checked the same on brick level ? How big is the difference ?

Best Regards,
Strahil Nikolov