<div dir="ltr">Hi all,<div><br></div><div>We have 500GB and 10TB 4x1 replicate xfs-based gluster volumes, and the 10TB one especially is extremely slow to do certain things with (and has been since gluster 3.x when we started). We're currently on 5.13.</div><div><br></div><div>The number of files isn't even what I'd consider that great - under 100k per dir.</div><div><br></div><div>Here are some numbers to look at:</div><div><br></div><div>On gluster volume in a dir of 45k files:</div><div>The first time</div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div>time find | wc -l</div><div>45423</div><div>real   8m44.819s</div><div>user   0m0.459s</div><div>sys   0m0.998s</div></blockquote><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div>And again </div></div></div></div></div></div></div></div></div></div></div></div></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><div class="gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div>time find | wc -l</div></div></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div>45423</div></div></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div>real   0m34.677s</div></div></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div>user   0m0.291s</div></div></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div>sys   0m0.754s</div></div></div></div></div></div></div></div></div></div></div></div></div></div></blockquote><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><br></div><div>If I run the same operation on the xfs block device itself:</div></div><div>The first time</div></div></div></div></div></div></div></div></div></div></div></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><div class="gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div>time find | wc -l</div></div></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div>45423</div></div></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div>real   0m13.514s</div></div></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div>user   0m0.144s</div></div></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div>sys   0m0.501s</div></div></div></div></div></div></div></div></div></div></div></div></div></div></blockquote><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div>And again</div></div></div></div></div></div></div></div></div></div></div></div></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><div class="gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div>time find | wc -l</div></div></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div>45423</div></div></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div>real   0m0.197s</div></div></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div>user   0m0.088s</div></div></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div>sys   0m0.106s</div></div></div></div></div></div></div></div></div></div></div></div></div></div></blockquote><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div><div>I'd expect a performance difference here but just as it was several years ago when we started with gluster, it's still huge, and simple file listings are incredibly slow.</div><div><br></div><div>At the time, the team was looking to do some optimizations, but I'm not sure this has happened.</div><div><br></div><div>What can we do to try to improve performance?</div><div><br></div><div>Thank you.</div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">Some setup values follow.<br></div><div dir="ltr"><br></div><div dir="ltr"><font face="monospace">xfs_info /mnt/SNIP_block1<br>meta-data=/dev/sdc        isize=512   agcount=103, agsize=26214400 blks<br>     =            sectsz=512  attr=2, projid32bit=1<br>     =            crc=1     finobt=1, sparse=0, rmapbt=0<br>     =            reflink=0<br>data   =            bsize=4096  blocks=2684354560, imaxpct=25<br>     =            sunit=0    swidth=0 blks<br>naming  =version 2        bsize=4096  ascii-ci=0, ftype=1<br>log    =internal log      bsize=4096  blocks=51200, version=2<br>     =            sectsz=512  sunit=0 blks, lazy-count=1<br>realtime =none          extsz=4096  blocks=0, rtextents=0</font><br></div><div dir="ltr"><br></div><div dir="ltr"><font face="monospace">Volume Name: SNIP_data1<br>Type: Replicate<br>Volume ID: SNIP<br>Status: Started<br>Snapshot Count: 0<br>Number of Bricks: 1 x 4 = 4<br>Transport-type: tcp<br>Bricks:<br>Brick1: nexus2:/mnt/SNIP_block1/SNIP_data1<br>Brick2: forge:/mnt/SNIP_block1/SNIP_data1<br>Brick3: hive:/mnt/SNIP_block1/SNIP_data1<br>Brick4: citadel:/mnt/SNIP_block1/SNIP_data1<br>Options Reconfigured:<br>cluster.quorum-count: 1<br>cluster.quorum-type: fixed<br>network.ping-timeout: 5<br>network.remote-dio: enable<br>performance.rda-cache-limit: 256MB<br>performance.readdir-ahead: on<br>performance.parallel-readdir: on<br>network.inode-lru-limit: 500000<br>performance.md-cache-timeout: 600<br>performance.cache-invalidation: on<br>performance.stat-prefetch: on<br>features.cache-invalidation-timeout: 600<br>features.cache-invalidation: on<br>cluster.readdir-optimize: on<br>performance.io-thread-count: 32<br>server.event-threads: 4<br>client.event-threads: 4<br>performance.read-ahead: off<br>cluster.lookup-optimize: on<br>performance.cache-size: 1GB<br>cluster.self-heal-daemon: enable<br>transport.address-family: inet<br>nfs.disable: on<br>performance.client-io-threads: on<br>cluster.granular-entry-heal: enable<br>cluster.data-self-heal-algorithm: full</font><br></div><div dir="ltr"><br>Sincerely,<br>Artem<br><br>--<br>Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8px" target="_blank">APK Mirror</a><span style="font-size:12.8px">, Illogical Robot LLC</span></div><div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br></div></div></div></div></div></div></div></div></div>