<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Howdy all,<div class=""><br class=""></div><div class="">Sorry in Australia so most of your replies came in over night for me.</div><div class=""><br class=""></div><div class="">Note: At the end of this reply is a listing of <i class="">all</i> our volume settings (gluster get volname all).</div><div class="">Note 2: I really wish Gluster used Discourse for this kind of community troubleshooting an analysis, using a mailing list is really painful.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class=""></div><blockquote type="cite" class=""><div class="">On 19 Mar 2018, at 4:38 pm, TomK <<a href="mailto:tomkcpr@mdevsys.com" class="">tomkcpr@mdevsys.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">On 3/19/2018 1:07 AM, TomK wrote:<br class="">A few numbers you could try:<br class=""><br class="">performance.cache-refresh-timeout<span class="Apple-tab-span" style="white-space: pre;"> </span>Default: 1s<br class=""></div></blockquote><div class=""><br class=""></div><div class="">I've actually set this right up to 60 (seconds), I guess it's possible that's causing an issue but I thought that was more for forced eviction on idle files.</div><div class=""><br class=""></div><blockquote type="cite" class=""><div class="">cluster.stripe-block-size<span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span>Default: 128KB<br class=""></div></blockquote><div class=""><br class=""></div><div class="">Hmm yes I wonder if it might be worth looking at the stripe-block-size, I forgot about this as it sounds like it's for striped volumes (now deprecated) only.</div><div class="">The issue with this is that I don't want to tune the volume <i class="">just</i> for small files and hurt the performance of lager I/O operations.</div><br class=""><blockquote type="cite" class=""><div class=""><br class="">Looks like others are having this sort of performance problem:<br class=""><br class=""><a href="http://lists.gluster.org/pipermail/gluster-users/2015-April/021487.html" class="">http://lists.gluster.org/pipermail/gluster-users/2015-April/021487.html</a><br class=""><br class="">Some recommended values by one poster that might help out (https://forum.proxmox.com/threads/horribly-slow-gluster-performance.26319/) Going to try in my LAB and let you know:<br class=""><br class=""><br class="">> GlusterFS 3.7 parameters:<br class=""></div></blockquote><div class=""><br class=""></div><div class="">GlusterFS 3.7 is <i class=""><u class="">really</u></i> old so I'd be careful looking at settings / tuning for it.</div><br class=""><blockquote type="cite" class=""><div class="">nfs.trusted-sync: on<br class=""></div></blockquote><div class=""><br class=""></div><div class="">Not using NFS.</div><br class=""><blockquote type="cite" class=""><div class="">performance.cache-size: 1GB<br class=""></div></blockquote><div class=""><br class=""></div><div class="">Already set to 1024MB, but that's only for <i class="">reads</i> not writes.</div><br class=""><blockquote type="cite" class=""><div class=""><a href="http://performance.io" class="">performance.io</a>-thread-count: 16<br class=""></div></blockquote><div class=""><br class=""></div><div class="">That's my current setting.</div><br class=""><blockquote type="cite" class=""><div class="">performance.write-behind-window-size: 8MB<br class=""></div></blockquote><div class=""><br class=""></div><div class="">Currently allowing even more cache up at 256MB.</div><br class=""><blockquote type="cite" class=""><div class="">performance.readdir-ahead: on<br class=""></div></blockquote><div class=""><br class=""></div><div class="">That's my current setting (the default now I believe).</div><br class=""><blockquote type="cite" class=""><div class="">client.event-threads: 8<br class=""></div></blockquote><div class=""><br class=""></div>That's my current setting (the default now I believe).</div><div class=""><br class=""><blockquote type="cite" class=""><div class="">server.event-threads: 8<br class=""></div></blockquote><div class=""><br class=""></div>That's my current setting (the default now I believe).</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class="">cluster.quorum-type: auto<br class=""></div></blockquote><div class=""><br class=""></div><div class="">Not sure how that's going to impact small I/O performance.</div><div class="">I currently have this set to none, but do use an arbiter node.</div><br class=""><blockquote type="cite" class=""><div class="">cluster.server-quorum-type: server<br class=""></div></blockquote><div class=""><br class=""></div><div class="">Not sure how that's going to impact small I/O performance.</div><div class=""><div class="">I currently have this set to off, but do use an arbiter node.</div></div><br class=""><blockquote type="cite" class=""><div class="">cluster.server-quorum-ratio: 51%<br class=""></div></blockquote><div class=""><br class=""></div><div class=""><div class="">Not sure how that's going to impact small I/O performance.</div><div class="">I currently have this set to 0, but do use an arbiter node.</div></div><br class=""><blockquote type="cite" class=""><div class=""><br class="">> Kernel parameters:<br class="">net.ipv4.tcp_slow_start_after_idle = 0<br class=""></div></blockquote><div class=""><br class=""></div><div class="">That's my current setting.</div><br class=""><blockquote type="cite" class=""><div class="">net.ipv4.tcp_fin_timeout = 15<br class=""></div></blockquote><div class=""><br class=""></div><div class="">I've set this right down to 5.</div><br class=""><blockquote type="cite" class=""><div class="">net.core.somaxconn = 65535<br class=""></div></blockquote><div class=""><br class=""></div><div class="">That's my current setting.</div><br class=""><blockquote type="cite" class=""><div class="">vm.swappiness = 1<br class=""></div></blockquote><div class=""><br class=""></div><div class="">That's my current setting, we don't have swap - other than ZRAM enabled on any hosts.</div><br class=""><blockquote type="cite" class=""><div class="">vm.dirty_ratio = 5<br class=""></div></blockquote><div class=""><br class=""></div><div class="">N/A as swap disabled (ZRAM only)</div><br class=""><blockquote type="cite" class=""><div class="">vm.dirty_background_ratio = 2<br class=""></div></blockquote><div class=""><br class=""></div>N/A as swap disabled (ZRAM only)</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class="">vm.min_free_kbytes = 524288 <span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span><span class="Apple-tab-span" style="white-space: pre;"> </span># this is on 128GB RAM<br class=""></div></blockquote><div class=""><br class=""></div><div class="">I have this set to vm.min_free_kbytes = 67584, I'd be worried that setting this high would cause OOM as per the official kernel docs:</div><div class=""><br class=""></div><div class=""><pre class=""><i class="">min_free_kbytes:
This is used to force the Linux VM to keep a minimum number
of kilobytes free. The VM uses this number to compute a
watermark[WMARK_MIN] value for each lowmem zone in the system.
Each lowmem zone gets a number of reserved free pages based
proportionally on its size.
Some minimal amount of memory is needed to satisfy PF_MEMALLOC
allocations; if you set this to lower than 1024KB, your system will
become subtly broken, and prone to deadlock under high loads.
Setting this too high will OOM your machine instantly.</i>
</pre></div></div><div class=""><br class=""></div><div class="">
<div><br class=""><blockquote type="cite" class=""><div class="">On 20 Mar 2018, at 1:52 am, Rik Theys <<a href="mailto:Rik.Theys@esat.kuleuven.be" class="">Rik.Theys@esat.kuleuven.be</a>> wrote:</div><div class=""><br class=""><span style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">That's not really a fare comparison as you don't specify a blocksize.</span><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">What does</span><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">dd if=/dev/zero of=./some-file.bin bs=1M count=1000 oflag=direct</span><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">give?</span><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Rik</span><br style="font-family: IBMPlexMono; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""></div></blockquote></div><br class=""></div><div class=""><div class="">DD is not going to give anyone particularly useful benchmarks, especially with small file sizes, in fact it's more likely to mislead you than be useful.</div><div class="">See my short post on fio here: <a href="https://smcleod.net/tech/2016/04/29/benchmarking-io.html" class="">https://smcleod.net/tech/2016/04/29/benchmarking-io.html</a> , I believe it's one of the most useful tools for I/O benchmarking.</div><div class=""><br class=""></div><div class="">Just for a laugh I compared dd writes for 4k (small) writes between the client (gluster mounted on the cli) and a gluster host (to a directory on the same storage as the bricks).</div><div class="">The client came out faster, likely the direct I/O flag was not working as perhaps intended.</div><div class=""><br class=""></div><div class="">Client:</div><div class=""><br class=""></div><div class=""><div class=""><font face="IBMPlexMono" class=""># dd if=/dev/zero of=./some-file.bin bs=4K count=4096 oflag=direct</font></div><div class=""><font face="IBMPlexMono" class="">4096+0 records in</font></div><div class=""><font face="IBMPlexMono" class="">4096+0 records out</font></div><div class=""><font face="IBMPlexMono" class="">16777216 bytes (17 MB) copied, 2.27839 s, 7.4 MB/s</font></div></div><div class=""><br class=""></div><div class="">Server:</div><div class=""><br class=""></div><div class=""><div class=""><font face="IBMPlexMono" class="">dd if=/dev/zero of=./some-file.bin bs=4K count=4096 oflag=direct</font></div><div class=""><font face="IBMPlexMono" class="">4096+0 records in</font></div><div class=""><font face="IBMPlexMono" class="">4096+0 records out</font></div><div class=""><font face="IBMPlexMono" class="">16777216 bytes (17 MB) copied, 3.94093 s, 4.3 MB/s</font></div></div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class=""><blockquote type="cite" class="">Note: At the end of this reply is a listing of <i class="">all</i> our volume settings (gluster get volname all).</blockquote></div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Here is an output of all gluster volume settings as they currently stand:</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class=""><font face="IBMPlexMono" class=""> # gluster volume get uat_storage all</font></div><div class=""><font face="IBMPlexMono" class="">Option Value</font></div><div class=""><font face="IBMPlexMono" class="">------ -----</font></div><div class=""><font face="IBMPlexMono" class="">cluster.lookup-unhashed on</font></div><div class=""><font face="IBMPlexMono" class="">cluster.lookup-optimize true</font></div><div class=""><font face="IBMPlexMono" class="">cluster.min-free-disk 10%</font></div><div class=""><font face="IBMPlexMono" class="">cluster.min-free-inodes 5%</font></div><div class=""><font face="IBMPlexMono" class="">cluster.rebalance-stats off</font></div><div class=""><font face="IBMPlexMono" class="">cluster.subvols-per-directory (null)</font></div><div class=""><font face="IBMPlexMono" class="">cluster.readdir-optimize true</font></div><div class=""><font face="IBMPlexMono" class="">cluster.rsync-hash-regex (null)</font></div><div class=""><font face="IBMPlexMono" class="">cluster.extra-hash-regex (null)</font></div><div class=""><font face="IBMPlexMono" class="">cluster.dht-xattr-name trusted.glusterfs.dht</font></div><div class=""><font face="IBMPlexMono" class="">cluster.randomize-hash-range-by-gfid off</font></div><div class=""><font face="IBMPlexMono" class="">cluster.rebal-throttle normal</font></div><div class=""><font face="IBMPlexMono" class="">cluster.lock-migration off</font></div><div class=""><font face="IBMPlexMono" class="">cluster.local-volume-name (null)</font></div><div class=""><font face="IBMPlexMono" class="">cluster.weighted-rebalance on</font></div><div class=""><font face="IBMPlexMono" class="">cluster.switch-pattern (null)</font></div><div class=""><font face="IBMPlexMono" class="">cluster.entry-change-log on</font></div><div class=""><font face="IBMPlexMono" class="">cluster.read-subvolume (null)</font></div><div class=""><font face="IBMPlexMono" class="">cluster.read-subvolume-index -1</font></div><div class=""><font face="IBMPlexMono" class="">cluster.read-hash-mode 1</font></div><div class=""><font face="IBMPlexMono" class="">cluster.background-self-heal-count 8</font></div><div class=""><font face="IBMPlexMono" class="">cluster.metadata-self-heal on</font></div><div class=""><font face="IBMPlexMono" class="">cluster.data-self-heal on</font></div><div class=""><font face="IBMPlexMono" class="">cluster.entry-self-heal on</font></div><div class=""><font face="IBMPlexMono" class="">cluster.self-heal-daemon on</font></div><div class=""><font face="IBMPlexMono" class="">cluster.heal-timeout 600</font></div><div class=""><font face="IBMPlexMono" class="">cluster.self-heal-window-size 1</font></div><div class=""><font face="IBMPlexMono" class="">cluster.data-change-log on</font></div><div class=""><font face="IBMPlexMono" class="">cluster.metadata-change-log on</font></div><div class=""><font face="IBMPlexMono" class="">cluster.data-self-heal-algorithm (null)</font></div><div class=""><font face="IBMPlexMono" class="">cluster.eager-lock true</font></div><div class=""><font face="IBMPlexMono" class="">disperse.eager-lock on</font></div><div class=""><font face="IBMPlexMono" class="">disperse.other-eager-lock on</font></div><div class=""><font face="IBMPlexMono" class="">cluster.quorum-type none</font></div><div class=""><font face="IBMPlexMono" class="">cluster.quorum-count (null)</font></div><div class=""><font face="IBMPlexMono" class="">cluster.choose-local true</font></div><div class=""><font face="IBMPlexMono" class="">cluster.self-heal-readdir-size 1KB</font></div><div class=""><font face="IBMPlexMono" class="">cluster.post-op-delay-secs 1</font></div><div class=""><font face="IBMPlexMono" class="">cluster.ensure-durability on</font></div><div class=""><font face="IBMPlexMono" class="">cluster.consistent-metadata no</font></div><div class=""><font face="IBMPlexMono" class="">cluster.heal-wait-queue-length 128</font></div><div class=""><font face="IBMPlexMono" class="">cluster.favorite-child-policy size</font></div><div class=""><font face="IBMPlexMono" class="">cluster.full-lock yes</font></div><div class=""><font face="IBMPlexMono" class="">cluster.stripe-block-size 128KB</font></div><div class=""><font face="IBMPlexMono" class="">cluster.stripe-coalesce true</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.latency-measurement off</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.dump-fd-stats off</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.count-fop-hits off</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.brick-log-level ERROR</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.client-log-level ERROR</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.brick-sys-log-level CRITICAL</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.client-sys-log-level CRITICAL</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.brick-logger (null)</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.client-logger (null)</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.brick-log-format (null)</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.client-log-format (null)</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.brick-log-buf-size 5</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.client-log-buf-size 5</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.brick-log-flush-timeout 120</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.client-log-flush-timeout 120</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.stats-dump-interval 0</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.fop-sample-interval 0</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.stats-dump-format json</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.fop-sample-buf-size 65535</font></div><div class=""><font face="IBMPlexMono" class="">diagnostics.stats-dnscache-ttl-sec 86400</font></div><div class=""><font face="IBMPlexMono" class="">performance.cache-max-file-size 6MB</font></div><div class=""><font face="IBMPlexMono" class="">performance.cache-min-file-size 0</font></div><div class=""><font face="IBMPlexMono" class="">performance.cache-refresh-timeout 60</font></div><div class=""><font face="IBMPlexMono" class="">performance.cache-priority</font></div><div class=""><font face="IBMPlexMono" class="">performance.cache-size 1024MB</font></div><div class=""><font face="IBMPlexMono" class=""><a href="http://performance.io" class="">performance.io</a>-thread-count 16</font></div><div class=""><font face="IBMPlexMono" class="">performance.high-prio-threads 16</font></div><div class=""><font face="IBMPlexMono" class="">performance.normal-prio-threads 16</font></div><div class=""><font face="IBMPlexMono" class="">performance.low-prio-threads 16</font></div><div class=""><font face="IBMPlexMono" class="">performance.least-prio-threads 1</font></div><div class=""><font face="IBMPlexMono" class="">performance.enable-least-priority on</font></div><div class=""><font face="IBMPlexMono" class="">performance.cache-size 1024MB</font></div><div class=""><font face="IBMPlexMono" class="">performance.flush-behind on</font></div><div class=""><font face="IBMPlexMono" class="">performance.nfs.flush-behind on</font></div><div class=""><font face="IBMPlexMono" class="">performance.write-behind-window-size 256MB</font></div><div class=""><font face="IBMPlexMono" class="">performance.resync-failed-syncs-after-fsyncoff</font></div><div class=""><font face="IBMPlexMono" class="">performance.nfs.write-behind-window-size1MB</font></div><div class=""><font face="IBMPlexMono" class="">performance.strict-o-direct off</font></div><div class=""><font face="IBMPlexMono" class="">performance.nfs.strict-o-direct off</font></div><div class=""><font face="IBMPlexMono" class="">performance.strict-write-ordering off</font></div><div class=""><font face="IBMPlexMono" class="">performance.nfs.strict-write-ordering off</font></div><div class=""><font face="IBMPlexMono" class="">performance.write-behind-trickling-writeson</font></div><div class=""><font face="IBMPlexMono" class="">performance.nfs.write-behind-trickling-writeson</font></div><div class=""><font face="IBMPlexMono" class="">performance.lazy-open yes</font></div><div class=""><font face="IBMPlexMono" class="">performance.read-after-open no</font></div><div class=""><font face="IBMPlexMono" class="">performance.read-ahead-page-count 4</font></div><div class=""><font face="IBMPlexMono" class="">performance.md-cache-timeout 600</font></div><div class=""><font face="IBMPlexMono" class="">performance.cache-swift-metadata true</font></div><div class=""><font face="IBMPlexMono" class="">performance.cache-samba-metadata false</font></div><div class=""><font face="IBMPlexMono" class="">performance.cache-capability-xattrs true</font></div><div class=""><font face="IBMPlexMono" class="">performance.cache-ima-xattrs true</font></div><div class=""><font face="IBMPlexMono" class="">features.encryption off</font></div><div class=""><font face="IBMPlexMono" class="">encryption.master-key (null)</font></div><div class=""><font face="IBMPlexMono" class="">encryption.data-key-size 256</font></div><div class=""><font face="IBMPlexMono" class="">encryption.block-size 4096</font></div><div class=""><font face="IBMPlexMono" class="">network.frame-timeout 1800</font></div><div class=""><font face="IBMPlexMono" class="">network.ping-timeout 15</font></div><div class=""><font face="IBMPlexMono" class="">network.tcp-window-size (null)</font></div><div class=""><font face="IBMPlexMono" class="">features.lock-heal off</font></div><div class=""><font face="IBMPlexMono" class="">features.grace-timeout 10</font></div><div class=""><font face="IBMPlexMono" class="">network.remote-dio disable</font></div><div class=""><font face="IBMPlexMono" class="">client.event-threads 8</font></div><div class=""><font face="IBMPlexMono" class="">client.tcp-user-timeout 0</font></div><div class=""><font face="IBMPlexMono" class="">client.keepalive-time 20</font></div><div class=""><font face="IBMPlexMono" class="">client.keepalive-interval 2</font></div><div class=""><font face="IBMPlexMono" class="">client.keepalive-count 9</font></div><div class=""><font face="IBMPlexMono" class="">network.tcp-window-size (null)</font></div><div class=""><font face="IBMPlexMono" class="">network.inode-lru-limit 50000</font></div><div class=""><font face="IBMPlexMono" class="">auth.allow *</font></div><div class=""><font face="IBMPlexMono" class="">auth.reject (null)</font></div><div class=""><font face="IBMPlexMono" class="">transport.keepalive 1</font></div><div class=""><font face="IBMPlexMono" class="">server.allow-insecure (null)</font></div><div class=""><font face="IBMPlexMono" class="">server.root-squash off</font></div><div class=""><font face="IBMPlexMono" class="">server.anonuid 65534</font></div><div class=""><font face="IBMPlexMono" class="">server.anongid 65534</font></div><div class=""><font face="IBMPlexMono" class="">server.statedump-path /var/run/gluster</font></div><div class=""><font face="IBMPlexMono" class="">server.outstanding-rpc-limit 256</font></div><div class=""><font face="IBMPlexMono" class="">features.lock-heal off</font></div><div class=""><font face="IBMPlexMono" class="">features.grace-timeout 10</font></div><div class=""><font face="IBMPlexMono" class="">server.ssl (null)</font></div><div class=""><font face="IBMPlexMono" class="">auth.ssl-allow *</font></div><div class=""><font face="IBMPlexMono" class="">server.manage-gids off</font></div><div class=""><font face="IBMPlexMono" class="">server.dynamic-auth on</font></div><div class=""><font face="IBMPlexMono" class="">client.send-gids on</font></div><div class=""><font face="IBMPlexMono" class="">server.gid-timeout 300</font></div><div class=""><font face="IBMPlexMono" class="">server.own-thread (null)</font></div><div class=""><font face="IBMPlexMono" class="">server.event-threads 8</font></div><div class=""><font face="IBMPlexMono" class="">server.tcp-user-timeout 0</font></div><div class=""><font face="IBMPlexMono" class="">server.keepalive-time 20</font></div><div class=""><font face="IBMPlexMono" class="">server.keepalive-interval 2</font></div><div class=""><font face="IBMPlexMono" class="">server.keepalive-count 9</font></div><div class=""><font face="IBMPlexMono" class="">transport.listen-backlog 2048</font></div><div class=""><font face="IBMPlexMono" class="">ssl.own-cert (null)</font></div><div class=""><font face="IBMPlexMono" class="">ssl.private-key (null)</font></div><div class=""><font face="IBMPlexMono" class=""><a href="http://ssl.ca" class="">ssl.ca</a>-list (null)</font></div><div class=""><font face="IBMPlexMono" class="">ssl.crl-path (null)</font></div><div class=""><font face="IBMPlexMono" class="">ssl.certificate-depth (null)</font></div><div class=""><font face="IBMPlexMono" class="">ssl.cipher-list (null)</font></div><div class=""><font face="IBMPlexMono" class="">ssl.dh-param (null)</font></div><div class=""><font face="IBMPlexMono" class="">ssl.ec-curve (null)</font></div><div class=""><font face="IBMPlexMono" class="">transport.address-family inet</font></div><div class=""><font face="IBMPlexMono" class="">performance.write-behind on</font></div><div class=""><font face="IBMPlexMono" class="">performance.read-ahead on</font></div><div class=""><font face="IBMPlexMono" class="">performance.readdir-ahead on</font></div><div class=""><font face="IBMPlexMono" class=""><a href="http://performance.io" class="">performance.io</a>-cache on</font></div><div class=""><font face="IBMPlexMono" class="">performance.quick-read on</font></div><div class=""><font face="IBMPlexMono" class="">performance.open-behind on</font></div><div class=""><font face="IBMPlexMono" class=""><a href="http://performance.nl" class="">performance.nl</a>-cache off</font></div><div class=""><font face="IBMPlexMono" class="">performance.stat-prefetch true</font></div><div class=""><font face="IBMPlexMono" class="">performance.client-io-threads true</font></div><div class=""><font face="IBMPlexMono" class="">performance.nfs.write-behind on</font></div><div class=""><font face="IBMPlexMono" class="">performance.nfs.read-ahead off</font></div><div class=""><font face="IBMPlexMono" class=""><a href="http://performance.nfs.io" class="">performance.nfs.io</a>-cache off</font></div><div class=""><font face="IBMPlexMono" class="">performance.nfs.quick-read off</font></div><div class=""><font face="IBMPlexMono" class="">performance.nfs.stat-prefetch off</font></div><div class=""><font face="IBMPlexMono" class=""><a href="http://performance.nfs.io" class="">performance.nfs.io</a>-threads off</font></div><div class=""><font face="IBMPlexMono" class="">performance.force-readdirp true</font></div><div class=""><font face="IBMPlexMono" class="">performance.cache-invalidation true</font></div><div class=""><font face="IBMPlexMono" class="">features.uss off</font></div><div class=""><font face="IBMPlexMono" class="">features.snapshot-directory .snaps</font></div><div class=""><font face="IBMPlexMono" class="">features.show-snapshot-directory off</font></div><div class=""><font face="IBMPlexMono" class="">network.compression off</font></div><div class=""><font face="IBMPlexMono" class="">network.compression.window-size -15</font></div><div class=""><font face="IBMPlexMono" class="">network.compression.mem-level 8</font></div><div class=""><font face="IBMPlexMono" class="">network.compression.min-size 0</font></div><div class=""><font face="IBMPlexMono" class="">network.compression.compression-level -1</font></div><div class=""><font face="IBMPlexMono" class="">network.compression.debug false</font></div><div class=""><font face="IBMPlexMono" class="">features.limit-usage (null)</font></div><div class=""><font face="IBMPlexMono" class="">features.default-soft-limit 80%</font></div><div class=""><font face="IBMPlexMono" class="">features.soft-timeout 60</font></div><div class=""><font face="IBMPlexMono" class="">features.hard-timeout 5</font></div><div class=""><font face="IBMPlexMono" class="">features.alert-time 86400</font></div><div class=""><font face="IBMPlexMono" class="">features.quota-deem-statfs off</font></div><div class=""><font face="IBMPlexMono" class="">geo-replication.indexing off</font></div><div class=""><font face="IBMPlexMono" class="">geo-replication.indexing off</font></div><div class=""><font face="IBMPlexMono" class="">geo-replication.ignore-pid-check off</font></div><div class=""><font face="IBMPlexMono" class="">geo-replication.ignore-pid-check off</font></div><div class=""><font face="IBMPlexMono" class="">features.quota off</font></div><div class=""><font face="IBMPlexMono" class="">features.inode-quota off</font></div><div class=""><font face="IBMPlexMono" class="">features.bitrot disable</font></div><div class=""><font face="IBMPlexMono" class="">debug.trace off</font></div><div class=""><font face="IBMPlexMono" class="">debug.log-history no</font></div><div class=""><font face="IBMPlexMono" class="">debug.log-file no</font></div><div class=""><font face="IBMPlexMono" class="">debug.exclude-ops (null)</font></div><div class=""><font face="IBMPlexMono" class="">debug.include-ops (null)</font></div><div class=""><font face="IBMPlexMono" class="">debug.error-gen off</font></div><div class=""><font face="IBMPlexMono" class="">debug.error-failure (null)</font></div><div class=""><font face="IBMPlexMono" class="">debug.error-number (null)</font></div><div class=""><font face="IBMPlexMono" class="">debug.random-failure off</font></div><div class=""><font face="IBMPlexMono" class="">debug.error-fops (null)</font></div><div class=""><font face="IBMPlexMono" class="">nfs.disable on</font></div><div class=""><font face="IBMPlexMono" class="">features.read-only off</font></div><div class=""><font face="IBMPlexMono" class="">features.worm off</font></div><div class=""><font face="IBMPlexMono" class="">features.worm-file-level off</font></div><div class=""><font face="IBMPlexMono" class="">features.worm-files-deletable on</font></div><div class=""><font face="IBMPlexMono" class="">features.default-retention-period 120</font></div><div class=""><font face="IBMPlexMono" class="">features.retention-mode relax</font></div><div class=""><font face="IBMPlexMono" class="">features.auto-commit-period 180</font></div><div class=""><font face="IBMPlexMono" class="">storage.linux-aio off</font></div><div class=""><font face="IBMPlexMono" class="">storage.batch-fsync-mode reverse-fsync</font></div><div class=""><font face="IBMPlexMono" class="">storage.batch-fsync-delay-usec 0</font></div><div class=""><font face="IBMPlexMono" class="">storage.owner-uid -1</font></div><div class=""><font face="IBMPlexMono" class="">storage.owner-gid -1</font></div><div class=""><font face="IBMPlexMono" class="">storage.node-uuid-pathinfo off</font></div><div class=""><font face="IBMPlexMono" class="">storage.health-check-interval 30</font></div><div class=""><font face="IBMPlexMono" class="">storage.build-pgfid off</font></div><div class=""><font face="IBMPlexMono" class="">storage.gfid2path on</font></div><div class=""><font face="IBMPlexMono" class="">storage.gfid2path-separator :</font></div><div class=""><font face="IBMPlexMono" class="">storage.reserve 1</font></div><div class=""><font face="IBMPlexMono" class="">storage.bd-aio off</font></div><div class=""><font face="IBMPlexMono" class="">config.gfproxyd off</font></div><div class=""><font face="IBMPlexMono" class="">cluster.server-quorum-type off</font></div><div class=""><font face="IBMPlexMono" class="">cluster.server-quorum-ratio 0</font></div><div class=""><font face="IBMPlexMono" class="">changelog.changelog off</font></div><div class=""><font face="IBMPlexMono" class="">changelog.changelog-dir (null)</font></div><div class=""><font face="IBMPlexMono" class="">changelog.encoding ascii</font></div><div class=""><font face="IBMPlexMono" class="">changelog.rollover-time 15</font></div><div class=""><font face="IBMPlexMono" class="">changelog.fsync-interval 5</font></div><div class=""><font face="IBMPlexMono" class="">changelog.changelog-barrier-timeout 120</font></div><div class=""><font face="IBMPlexMono" class="">changelog.capture-del-path off</font></div><div class=""><font face="IBMPlexMono" class="">features.barrier disable</font></div><div class=""><font face="IBMPlexMono" class="">features.barrier-timeout 120</font></div><div class=""><font face="IBMPlexMono" class="">features.trash off</font></div><div class=""><font face="IBMPlexMono" class="">features.trash-dir .trashcan</font></div><div class=""><font face="IBMPlexMono" class="">features.trash-eliminate-path (null)</font></div><div class=""><font face="IBMPlexMono" class="">features.trash-max-filesize 5MB</font></div><div class=""><font face="IBMPlexMono" class="">features.trash-internal-op off</font></div><div class=""><font face="IBMPlexMono" class="">cluster.enable-shared-storage disable</font></div><div class=""><font face="IBMPlexMono" class="">cluster.write-freq-threshold 0</font></div><div class=""><font face="IBMPlexMono" class="">cluster.read-freq-threshold 0</font></div><div class=""><font face="IBMPlexMono" class="">cluster.tier-pause off</font></div><div class=""><font face="IBMPlexMono" class="">cluster.tier-promote-frequency 120</font></div><div class=""><font face="IBMPlexMono" class="">cluster.tier-demote-frequency 3600</font></div><div class=""><font face="IBMPlexMono" class="">cluster.watermark-hi 90</font></div><div class=""><font face="IBMPlexMono" class="">cluster.watermark-low 75</font></div><div class=""><font face="IBMPlexMono" class="">cluster.tier-mode cache</font></div><div class=""><font face="IBMPlexMono" class="">cluster.tier-max-promote-file-size 0</font></div><div class=""><font face="IBMPlexMono" class="">cluster.tier-max-mb 4000</font></div><div class=""><font face="IBMPlexMono" class="">cluster.tier-max-files 10000</font></div><div class=""><font face="IBMPlexMono" class="">cluster.tier-query-limit 100</font></div><div class=""><font face="IBMPlexMono" class="">cluster.tier-compact on</font></div><div class=""><font face="IBMPlexMono" class="">cluster.tier-hot-compact-frequency 604800</font></div><div class=""><font face="IBMPlexMono" class="">cluster.tier-cold-compact-frequency 604800</font></div><div class=""><font face="IBMPlexMono" class="">features.ctr-enabled off</font></div><div class=""><font face="IBMPlexMono" class="">features.record-counters off</font></div><div class=""><font face="IBMPlexMono" class="">features.ctr-record-metadata-heat off</font></div><div class=""><font face="IBMPlexMono" class="">features.ctr_link_consistency off</font></div><div class=""><font face="IBMPlexMono" class="">features.ctr_lookupheal_link_timeout 300</font></div><div class=""><font face="IBMPlexMono" class="">features.ctr_lookupheal_inode_timeout 300</font></div><div class=""><font face="IBMPlexMono" class="">features.ctr-sql-db-cachesize 12500</font></div><div class=""><font face="IBMPlexMono" class="">features.ctr-sql-db-wal-autocheckpoint 25000</font></div><div class=""><font face="IBMPlexMono" class="">features.selinux on</font></div><div class=""><font face="IBMPlexMono" class="">locks.trace off</font></div><div class=""><font face="IBMPlexMono" class="">locks.mandatory-locking off</font></div><div class=""><font face="IBMPlexMono" class="">cluster.disperse-self-heal-daemon enable</font></div><div class=""><font face="IBMPlexMono" class="">cluster.quorum-reads no</font></div><div class=""><font face="IBMPlexMono" class="">client.bind-insecure (null)</font></div><div class=""><font face="IBMPlexMono" class="">features.shard off</font></div><div class=""><font face="IBMPlexMono" class="">features.shard-block-size 64MB</font></div><div class=""><font face="IBMPlexMono" class="">features.scrub-throttle lazy</font></div><div class=""><font face="IBMPlexMono" class="">features.scrub-freq biweekly</font></div><div class=""><font face="IBMPlexMono" class="">features.scrub false</font></div><div class=""><font face="IBMPlexMono" class="">features.expiry-time 120</font></div><div class=""><font face="IBMPlexMono" class="">features.cache-invalidation true</font></div><div class=""><font face="IBMPlexMono" class="">features.cache-invalidation-timeout 600</font></div><div class=""><font face="IBMPlexMono" class="">features.leases off</font></div><div class=""><font face="IBMPlexMono" class="">features.lease-lock-recall-timeout 60</font></div><div class=""><font face="IBMPlexMono" class="">disperse.background-heals 8</font></div><div class=""><font face="IBMPlexMono" class="">disperse.heal-wait-qlength 128</font></div><div class=""><font face="IBMPlexMono" class="">cluster.heal-timeout 600</font></div><div class=""><font face="IBMPlexMono" class="">dht.force-readdirp on</font></div><div class=""><font face="IBMPlexMono" class="">disperse.read-policy round-robin</font></div><div class=""><font face="IBMPlexMono" class="">cluster.shd-max-threads 1</font></div><div class=""><font face="IBMPlexMono" class="">cluster.shd-wait-qlength 1024</font></div><div class=""><font face="IBMPlexMono" class="">cluster.locking-scheme full</font></div><div class=""><font face="IBMPlexMono" class="">cluster.granular-entry-heal no</font></div><div class=""><font face="IBMPlexMono" class="">features.locks-revocation-secs 0</font></div><div class=""><font face="IBMPlexMono" class="">features.locks-revocation-clear-all false</font></div><div class=""><font face="IBMPlexMono" class="">features.locks-revocation-max-blocked 0</font></div><div class=""><font face="IBMPlexMono" class="">features.locks-monkey-unlocking false</font></div><div class=""><font face="IBMPlexMono" class="">disperse.shd-max-threads 1</font></div><div class=""><font face="IBMPlexMono" class="">disperse.shd-wait-qlength 1024</font></div><div class=""><font face="IBMPlexMono" class="">disperse.cpu-extensions auto</font></div><div class=""><font face="IBMPlexMono" class="">disperse.self-heal-window-size 1</font></div><div class=""><font face="IBMPlexMono" class="">cluster.use-compound-fops true</font></div><div class=""><font face="IBMPlexMono" class="">performance.parallel-readdir off</font></div><div class=""><font face="IBMPlexMono" class="">performance.rda-request-size 131072</font></div><div class=""><font face="IBMPlexMono" class="">performance.rda-low-wmark 4096</font></div><div class=""><font face="IBMPlexMono" class="">performance.rda-high-wmark 128KB</font></div><div class=""><font face="IBMPlexMono" class="">performance.rda-cache-limit 256MB</font></div><div class=""><font face="IBMPlexMono" class=""><a href="http://performance.nl" class="">performance.nl</a>-cache-positive-entry false</font></div><div class=""><font face="IBMPlexMono" class=""><a href="http://performance.nl" class="">performance.nl</a>-cache-limit 10MB</font></div><div class=""><font face="IBMPlexMono" class=""><a href="http://performance.nl" class="">performance.nl</a>-cache-timeout 60</font></div><div class=""><font face="IBMPlexMono" class="">cluster.brick-multiplex off</font></div><div class=""><font face="IBMPlexMono" class="">cluster.max-bricks-per-process 0</font></div><div class=""><font face="IBMPlexMono" class="">disperse.optimistic-change-log on</font></div><div class=""><font face="IBMPlexMono" class="">cluster.halo-enabled False</font></div><div class=""><font face="IBMPlexMono" class="">cluster.halo-shd-max-latency 99999</font></div><div class=""><font face="IBMPlexMono" class="">cluster.halo-nfsd-max-latency 5</font></div><div class=""><font face="IBMPlexMono" class="">cluster.halo-max-latency 5</font></div><div class=""><font face="IBMPlexMono" class="">cluster.halo-max-replicas 99999</font></div><div class=""><font face="IBMPlexMono" class="">cluster.halo-min-replicas 2</font></div><div class=""><font face="IBMPlexMono" class="">debug.delay-gen off</font></div><div class=""><font face="IBMPlexMono" class="">delay-gen.delay-percentage 10%</font></div><div class=""><font face="IBMPlexMono" class="">delay-gen.delay-duration 100000</font></div><div class=""><font face="IBMPlexMono" class="">delay-gen.enable (null)</font></div><div class=""><font face="IBMPlexMono" class="">disperse.parallel-writes on</font></div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div style="font-size: 12px;">--<br class="">Sam McLeod (protoporpoise on IRC)</div><div style="font-size: 12px;"><a href="https://smcleod.net" class="">https://smcleod.net</a><br class="">https://twitter.com/s_mcleod<br class=""><br class="">Words are my own opinions and do not necessarily represent those of my employer or partners.</div></div></body></html>