<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Tom,<div class=""><br class=""></div><div class="">Thanks for your reply.</div><div class=""><br class=""></div><div class="">1. Yes XFS is on a LUKs LV (see below).</div><div class="">2. Yes, I prefer FIO but each Gluster host gets between 50-100K 4K random IOP/s both write and read to disk.</div><div class="">3. Yes, we actually use 2x 10Gbit DACs in LACP, but we get full 10Gbit speeds (and very low latency thanks to the DACs).</div><div class="">4. I'd love to see that, it'd be much appreciated thanks.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class=""><font face="IBMPlexMono" class=""># lsblk</font></div><div class=""><font face="IBMPlexMono" class="">NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT</font></div><div class=""><font face="IBMPlexMono" class="">xvdc 202:32 0 1.5T 0 disk</font></div><div class=""><font face="IBMPlexMono" class="">└─xvdc1 202:33 0 1.5T 0 part</font></div><div class=""><font face="IBMPlexMono" class=""> └─gluster-storage 253:1 0 3T 0 lvm</font></div><div class=""><font face="IBMPlexMono" class=""> └─gluster-storage-unlocked 253:3 0 3T 0 crypt /mnt/gluster-storage</font></div><div class=""><font face="IBMPlexMono" class="">xvda 202:0 0 18G 0 disk</font></div><div class=""><font face="IBMPlexMono" class="">├─xvda2 202:2 0 17.5G 0 part</font></div><div class=""><font face="IBMPlexMono" class="">│ ├─centos-var 253:2 0 9.5G 0 lvm /var</font></div><div class=""><font face="IBMPlexMono" class="">│ └─centos-root 253:0 0 8G 0 lvm /</font></div><div class=""><font face="IBMPlexMono" class="">└─xvda1 202:1 0 500M 0 part /boot</font></div><div class=""><font face="IBMPlexMono" class="">sr0 11:0 1 1024M 0 rom</font></div><div class=""><font face="IBMPlexMono" class="">xvdb 202:16 0 1.5T 0 disk</font></div><div class=""><font face="IBMPlexMono" class="">└─xvdb1 202:17 0 1.5T 0 part</font></div><div class=""><font face="IBMPlexMono" class=""> └─gluster-storage 253:1 0 3T 0 lvm</font></div><div class=""><font face="IBMPlexMono" class=""> └─gluster-storage-unlocked 253:3 0 3T 0 crypt /mnt/gluster-storage</font></div><div class="">
<br class="">--<br class="">Sam McLeod<br class="">Please respond via email when possible.<br class=""><a href="https://smcleod.net" class="">https://smcleod.net</a><br class="">https://twitter.com/s_mcleod
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On 19 Mar 2018, at 10:37 am, TomK <<a href="mailto:tomkcpr@mdevsys.com" class="">tomkcpr@mdevsys.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">On 3/18/2018 6:13 PM, Sam McLeod wrote:</span><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Even your NFS transfers are 12.5 or so MB per second or less.</span><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">1) Did you use fdisk and LVM under that XFS filesystem?</span><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">2) Did you benchmark the XFS with something like bonnie++? (There's probably newer benchmark suites now.)</span><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">3) Did you benchmark your Network transfer speeds? Perhaps your NIC negotiated a lower speed.</span><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">3) I've done XFS tuning for another purpose but got good results. If it helps, I can send you the doc.</span><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Cheers,</span><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Tom</span><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Howdy all,<br class="">We're experiencing terrible small file performance when copying or moving files on gluster clients.<br class="">In the example below, Gluster is taking 6mins~ to copy 128MB / 21,000 files sideways on a client, doing the same thing on NFS (which I know is a totally different solution etc. etc.) takes approximately 10-15 seconds(!).<br class="">Any advice for tuning the volume or XFS settings would be greatly appreciated.<br class="">Hopefully I've included enough relevant information below.<br class="">## Gluster Client<br class="">root@gluster-client:/mnt/gluster_perf_test/ # du -sh .<br class="">127M .<br class="">root@gluster-client:/mnt/gluster_perf_test/ # find . -type f | wc -l<br class="">21791<br class="">root@gluster-client:/mnt/gluster_perf_test/ # du 9584toto9584.txt<br class="">4 9584toto9584.txt<br class="">root@gluster-client:/mnt/gluster_perf_test/ # time cp -a private private_perf_test<br class="">real 5m51.862s<br class="">user 0m0.862s<br class="">sys 0m8.334s<br class="">root@gluster-client:/mnt/gluster_perf_test/ # time rm -rf private_perf_test/<br class="">real 0m49.702s<br class="">user 0m0.087s<br class="">sys 0m0.958s<br class="">## Hosts<br class="">- 16x Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz per Gluster host / client<br class="">- Storage: iSCSI provisioned (via 10Gbit DAC/Fibre), SSD disk, 50K R/RW 4k IOP/s, 400MB/s per Gluster host<br class="">- Volumes are replicated across two hosts and one arbiter only host<br class="">- Networking is 10Gbit DAC/Fibre between Gluster hosts and clients<br class="">- 18GB DDR4 ECC memory<br class="">## Volume Info<br class="">root@gluster-host-01:~ # gluster pool list<br class="">UUID Hostname State<br class="">ad02970b-e2aa-4ca8-998c-bd10d5970faa gluster-host-02.fqdn Connected<br class="">ea116a94-c19e-48db-b108-0be3ae622e2e gluster-host-03.fqdn Connected<br class="">2e855c25-e7ac-4ff6-be85-e8bcc6f45ee4 localhost Connected<br class="">root@gluster-host-01:~ # gluster volume info uat_storage<br class="">Volume Name: uat_storage<br class="">Type: Replicate<br class="">Volume ID: 7918f1c5-5031-47b8-b054-56f6f0c569a2<br class="">Status: Started<br class="">Snapshot Count: 0<br class="">Number of Bricks: 1 x (2 + 1) = 3<br class="">Transport-type: tcp<br class="">Bricks:<br class="">Brick1: gluster-host-01.fqdn:/mnt/gluster-storage/uat_storage<br class="">Brick2: gluster-host-02.fqdn:/mnt/gluster-storage/uat_storage<br class="">Brick3: gluster-host-03.fqdn:/mnt/gluster-storage/uat_storage (arbiter)<br class="">Options Reconfigured:<br class="">performance.rda-cache-limit: 256MB<br class="">network.inode-lru-limit: 50000<br class="">server.outstanding-rpc-limit: 256<br class="">performance.client-io-threads: true<br class="">nfs.disable: on<br class="">transport.address-family: inet<br class="">client.event-threads: 8<br class="">cluster.eager-lock: true<br class="">cluster.favorite-child-policy: size<br class="">cluster.lookup-optimize: true<br class="">cluster.readdir-optimize: true<br class="">cluster.use-compound-fops: true<br class="">diagnostics.brick-log-level: ERROR<br class="">diagnostics.client-log-level: ERROR<br class="">features.cache-invalidation-timeout: 600<br class="">features.cache-invalidation: true<br class="">network.ping-timeout: 15<br class="">performance.cache-invalidation: true<br class="">performance.cache-max-file-size: 6MB<br class="">performance.cache-refresh-timeout: 60<br class="">performance.cache-size: 1024MB<br class=""><a href="http://performance.io/" class="">performance.io</a><span class="Apple-converted-space"> </span><<a href="http://performance.io/" class="">http://performance.io</a>>-thread-count: 16<br class="">performance.md-cache-timeout: 600<br class="">performance.stat-prefetch: true<br class="">performance.write-behind-window-size: 256MB<br class="">server.event-threads: 8<br class="">transport.listen-backlog: 2048<br class="">root@gluster-host-01:~ # xfs_info /dev/mapper/gluster-storage-unlocked<br class="">meta-data=/dev/mapper/gluster-storage-unlocked isize=512 agcount=4, agsize=196607360 blks<br class=""> = sectsz=512 attr=2, projid32bit=1<br class=""> = crc=1 finobt=0 spinodes=0<br class="">data = bsize=4096 blocks=786429440, imaxpct=5<br class=""> = sunit=0 swidth=0 blks<br class="">naming =version 2 bsize=8192 ascii-ci=0 ftype=1<br class="">log =internal bsize=4096 blocks=383998, version=2<br class=""> = sectsz=512 sunit=0 blks, lazy-count=1<br class="">realtime =none extsz=4096 blocks=0, rtextents=0<br class="">--<br class="">Sam McLeod (protoporpoise on IRC)<br class=""><a href="https://smcleod.net" class="">https://smcleod.net</a><br class="">https://twitter.com/s_mcleod<br class="">Words are my own opinions and do not necessarily represent those of my employer or partners.<br class="">_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class=""><a href="http://lists.gluster.org/mailman/listinfo/gluster-users" class="">http://lists.gluster.org/mailman/listinfo/gluster-users</a><br class=""></blockquote><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">--<span class="Apple-converted-space"> </span></span><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Cheers,</span><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Tom K.</span><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">-------------------------------------------------------------------------------------</span><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Living on earth is expensive, but it includes a free trip around the sun.</span></div></blockquote></div><br class=""></div></body></html>