<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Howdy all,<div class=""><br class=""></div><div class="">We're experiencing terrible small file performance when copying or moving files on gluster clients.</div><div class=""><br class=""></div><div class="">In the example below, Gluster is taking 6mins~ to copy 128MB / 21,000 files sideways on a client, doing the same thing on NFS (which I know is a totally different solution etc. etc.) takes approximately 10-15 seconds(!).</div><div class=""><br class=""></div><div class="">Any advice for tuning the volume or XFS settings would be greatly appreciated.</div><div class=""><br class=""></div><div class="">Hopefully I've included enough relevant information below.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class=""><font face="IBMPlexMono" class="">## Gluster Client</font></div><div class=""><font face="IBMPlexMono" class=""><br class=""></font></div><div class=""><font face="IBMPlexMono" class="">root@gluster-client:/mnt/gluster_perf_test/ # du -sh .</font></div><div class=""><font face="IBMPlexMono" class="">127M .</font></div><div class=""><font face="IBMPlexMono" class="">root@gluster-client:/mnt/gluster_perf_test/ # find . -type f | wc -l</font></div><div class=""><font face="IBMPlexMono" class="">21791</font></div><div class=""><font face="IBMPlexMono" class="">root@gluster-client:/mnt/gluster_perf_test/ # du 9584toto9584.txt</font></div><div class=""><font face="IBMPlexMono" class="">4 9584toto9584.txt</font></div><div class=""><font face="IBMPlexMono" class=""><br class=""></font></div><div class=""><font face="IBMPlexMono" class=""><br class=""></font></div><div class=""><font face="IBMPlexMono" class="">root@gluster-client:/mnt/gluster_perf_test/ # time cp -a private private_perf_test</font></div><div class=""><font face="IBMPlexMono" class=""><br class=""></font></div><div class=""><font face="IBMPlexMono" class="">real 5m51.862s</font></div><div class=""><font face="IBMPlexMono" class="">user 0m0.862s</font></div><div class=""><font face="IBMPlexMono" class="">sys 0m8.334s</font></div><div class=""><font face="IBMPlexMono" class=""><br class=""></font></div><div class=""><font face="IBMPlexMono" class="">root@gluster-client:/mnt/gluster_perf_test/ # time rm -rf private_perf_test/</font></div><div class=""><font face="IBMPlexMono" class=""><br class=""></font></div><div class=""><font face="IBMPlexMono" class="">real 0m49.702s</font></div><div class=""><font face="IBMPlexMono" class="">user 0m0.087s</font></div><div class=""><font face="IBMPlexMono" class="">sys 0m0.958s</font></div></div><div class=""><font face="IBMPlexMono" class=""><br class=""></font></div><div class=""><font face="IBMPlexMono" class=""><br class=""></font></div><div class=""><div class=""><font face="IBMPlexMono" class="">## Hosts</font></div><div class=""><font face="IBMPlexMono" class=""><br class=""></font></div><div class=""><font face="IBMPlexMono" class="">- 16x Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz per Gluster host / client</font></div><div class=""><font face="IBMPlexMono" class="">- Storage: iSCSI provisioned (via 10Gbit DAC/Fibre), SSD disk, 50K R/RW 4k IOP/s, 400MB/s per Gluster host</font></div><div class=""><font face="IBMPlexMono" class="">- Volumes are replicated across two hosts and one arbiter only host</font></div><div class=""><font face="IBMPlexMono" class="">- Networking is 10Gbit DAC/Fibre between Gluster hosts and clients</font></div><div class=""><font face="IBMPlexMono" class="">- 18GB DDR4 ECC memory</font></div><div class=""><font face="IBMPlexMono" class=""><br class=""></font></div><div class=""><font face="IBMPlexMono" class="">## Volume Info</font></div><div class=""><font face="IBMPlexMono" class=""><br class=""></font></div><div class=""><font face="IBMPlexMono" class="">root@gluster-host-01:~ # gluster pool list</font></div><div class=""><font face="IBMPlexMono" class="">UUID Hostname State</font></div><div class=""><font face="IBMPlexMono" class="">ad02970b-e2aa-4ca8-998c-bd10d5970faa gluster-host-02.fqdn Connected</font></div><div class=""><font face="IBMPlexMono" class="">ea116a94-c19e-48db-b108-0be3ae622e2e gluster-host-03.fqdn Connected</font></div><div class=""><font face="IBMPlexMono" class="">2e855c25-e7ac-4ff6-be85-e8bcc6f45ee4 localhost Connected</font></div><div class=""><font face="IBMPlexMono" class=""><br class=""></font></div><div class=""><font face="IBMPlexMono" class="">root@gluster-host-01:~ # gluster volume info uat_storage</font></div><div class=""><font face="IBMPlexMono" class=""><br class=""></font></div><div class=""><font face="IBMPlexMono" class="">Volume Name: uat_storage</font></div><div class=""><font face="IBMPlexMono" class="">Type: Replicate</font></div><div class=""><font face="IBMPlexMono" class="">Volume ID: 7918f1c5-5031-47b8-b054-56f6f0c569a2</font></div><div class=""><font face="IBMPlexMono" class="">Status: Started</font></div><div class=""><font face="IBMPlexMono" class="">Snapshot Count: 0</font></div><div class=""><font face="IBMPlexMono" class="">Number of Bricks: 1 x (2 + 1) = 3</font></div><div class=""><font face="IBMPlexMono" class="">Transport-type: tcp</font></div><div class=""><font face="IBMPlexMono" class="">Bricks:</font></div><div class=""><font face="IBMPlexMono" class="">Brick1: gluster-host-01.fqdn:/mnt/gluster-storage/uat_storage</font></div><div class=""><font face="IBMPlexMono" class="">Brick2: gluster-host-02.fqdn:/mnt/gluster-storage/uat_storage</font></div><div class=""><font face="IBMPlexMono" class="">Brick3: gluster-host-03.fqdn:/mnt/gluster-storage/uat_storage (arbiter)</font></div><div class=""><font face="IBMPlexMono" class="">Options Reconfigured:</font></div><div class=""><font face="IBMPlexMono" class="">performance.rda-cache-limit: 256MB</font></div><div class=""><font face="IBMPlexMono" class="">network.inode-lru-limit: 50000</font></div><div class=""><font face="IBMPlexMono" class="">server.outstanding-rpc-limit: 256</font></div><div class=""><font face="IBMPlexMono" class="">performance.client-io-threads: true</font></div><div class=""><font face="IBMPlexMono" class="">nfs.disable: on</font></div><div class=""><font face="IBMPlexMono" class="">transport.address-family: inet</font></div><div class=""><font face="IBMPlexMono" class="">client.event-threads: 8</font></div><div class=""><font face="IBMPlexMono" class="">cluster.eager-lock: true</font></div><div class=""><font face="IBMPlexMono" class="">cluster.favorite-child-policy: size</font></div><div class=""><font face="IBMPlexMono" class="">cluster.lookup-optimize: true</font></div><div class=""><font face="IBMPlexMono" class="">cluster.readdir-optimize: true</font></div><div class=""><font face="IBMPlexMono" class="">cluster.use-compound-fops: true</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.brick-log-level: ERROR</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.client-log-level: ERROR</font></div><div class=""><font face="IBMPlexMono" class="">features.cache-invalidation-timeout: 600</font></div><div class=""><font face="IBMPlexMono" class="">features.cache-invalidation: true</font></div><div class=""><font face="IBMPlexMono" class="">network.ping-timeout: 15</font></div><div class=""><font face="IBMPlexMono" class="">performance.cache-invalidation: true</font></div><div class=""><font face="IBMPlexMono" class="">performance.cache-max-file-size: 6MB</font></div><div class=""><font face="IBMPlexMono" class="">performance.cache-refresh-timeout: 60</font></div><div class=""><font face="IBMPlexMono" class="">performance.cache-size: 1024MB</font></div><div class=""><font face="IBMPlexMono" class=""><a href="http://performance.io" class="">performance.io</a>-thread-count: 16</font></div><div class=""><font face="IBMPlexMono" class="">performance.md-cache-timeout: 600</font></div><div class=""><font face="IBMPlexMono" class="">performance.stat-prefetch: true</font></div><div class=""><font face="IBMPlexMono" class="">performance.write-behind-window-size: 256MB</font></div><div class=""><font face="IBMPlexMono" class="">server.event-threads: 8</font></div><div class=""><font face="IBMPlexMono" class="">transport.listen-backlog: 2048</font></div><div class=""><font face="IBMPlexMono" class=""><br class=""></font></div><div class=""><font face="IBMPlexMono" class="">root@gluster-host-01:~ # xfs_info /dev/mapper/gluster-storage-unlocked</font></div><div class=""><font face="IBMPlexMono" class="">meta-data=/dev/mapper/gluster-storage-unlocked isize=512 agcount=4, agsize=196607360 blks</font></div><div class=""><font face="IBMPlexMono" class=""> = sectsz=512 attr=2, projid32bit=1</font></div><div class=""><font face="IBMPlexMono" class=""> = crc=1 finobt=0 spinodes=0</font></div><div class=""><font face="IBMPlexMono" class="">data = bsize=4096 blocks=786429440, imaxpct=5</font></div><div class=""><font face="IBMPlexMono" class=""> = sunit=0 swidth=0 blks</font></div><div class=""><font face="IBMPlexMono" class="">naming =version 2 bsize=8192 ascii-ci=0 ftype=1</font></div><div class=""><font face="IBMPlexMono" class="">log =internal bsize=4096 blocks=383998, version=2</font></div><div class=""><font face="IBMPlexMono" class=""> = sectsz=512 sunit=0 blks, lazy-count=1</font></div><div class=""><font face="IBMPlexMono" class="">realtime =none extsz=4096 blocks=0, rtextents=0</font></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><span style="font-size: 12px;" class="">--</span></div></div><div class=""><div class=""><div dir="auto" style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">Sam McLeod (protoporpoise on IRC)</div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><a href="https://smcleod.net" class="">https://smcleod.net</a><br class="">https://twitter.com/s_mcleod<br class=""><br class="">Words are my own opinions and do not necessarily represent those of my employer or partners.</div></div></div></div>
</div>
<br class=""></div></body></html>