<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Excellent description, thank you.<div class=""><br class=""></div><div class="">With performance.write-behind-trickling-writes ON (default):</div><div class=""><br class=""></div><div class="">## 4k randwrite</div><div class=""><br class=""></div><div class=""><div class=""><font face="IBMPlexMono" class=""># fio --randrepeat=1 --ioengine=libaio --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=32 --size=256MB --readwrite=randwrite</font></div><div class=""><font face="IBMPlexMono" class="">test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32</font></div><div class=""><font face="IBMPlexMono" class="">fio-3.1</font></div><div class=""><font face="IBMPlexMono" class="">Starting 1 process</font></div><div class=""><font face="IBMPlexMono" class="">Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=17.3MiB/s][r=0,w=4422 IOPS][eta 00m:00s]</font></div><div class=""><font face="IBMPlexMono" class="">test: (groupid=0, jobs=1): err= 0: pid=42701: Tue Mar 20 15:05:23 2018</font></div><div class=""><font face="IBMPlexMono" class=""> write: <b class="">IOPS=4443</b>, <b class="">BW=17.4MiB/s</b> (18.2MB/s)(256MiB/14748msec)</font></div><div class=""><font face="IBMPlexMono" class=""> bw ( KiB/s): min=16384, max=19184, per=99.92%, avg=17760.45, stdev=602.48, samples=29</font></div><div class=""><font face="IBMPlexMono" class=""> iops : min= 4096, max= 4796, avg=4440.07, stdev=150.66, samples=29</font></div><div class=""><font face="IBMPlexMono" class=""> cpu : usr=4.00%, sys=18.02%, ctx=131097, majf=0, minf=7</font></div><div class=""><font face="IBMPlexMono" class=""> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%</font></div><div class=""><font face="IBMPlexMono" class=""> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%</font></div><div class=""><font face="IBMPlexMono" class=""> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%</font></div><div class=""><font face="IBMPlexMono" class=""> issued rwt: total=0,65536,0, short=0,0,0, dropped=0,0,0</font></div><div class=""><font face="IBMPlexMono" class=""> latency : target=0, window=0, percentile=100.00%, depth=32</font></div><div class=""><font face="IBMPlexMono" class=""><br class=""></font></div><div class=""><font face="IBMPlexMono" class="">Run status group 0 (all jobs):</font></div><div class=""><font face="IBMPlexMono" class=""> WRITE: bw=17.4MiB/s (18.2MB/s), 17.4MiB/s-17.4MiB/s (18.2MB/s-18.2MB/s), io=256MiB (268MB), run=14748-14748msec</font></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">## 2k randwrite</div><div class=""><br class="webkit-block-placeholder"></div><div class="">
<div class=""><font face="IBMPlexMono" class=""># fio --randrepeat=1 --ioengine=libaio --gtod_reduce=1 --name=test --filename=test --bs=2k --iodepth=32 --size=256MB --readwrite=randwrite</font></div><div class=""><font face="IBMPlexMono" class="">test: (g=0): rw=randwrite, bs=(R) 2048B-2048B, (W) 2048B-2048B, (T) 2048B-2048B, ioengine=libaio, iodepth=32</font></div><div class=""><font face="IBMPlexMono" class="">fio-3.1</font></div><div class=""><font face="IBMPlexMono" class="">Starting 1 process</font></div><div class=""><font face="IBMPlexMono" class="">Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=8624KiB/s][r=0,w=4312 IOPS][eta 00m:00s]</font></div><div class=""><font face="IBMPlexMono" class="">test: (groupid=0, jobs=1): err= 0: pid=42781: Tue Mar 20 15:05:57 2018</font></div><div class=""><font face="IBMPlexMono" class=""> write: <b class="">IOPS=4439, BW=8880KiB/s</b> (9093kB/s)(256MiB/29522msec)</font></div><div class=""><font face="IBMPlexMono" class=""> bw ( KiB/s): min= 6908, max= 9564, per=99.94%, avg=8874.03, stdev=428.92, samples=59</font></div><div class=""><font face="IBMPlexMono" class=""> iops : min= 3454, max= 4782, avg=4437.00, stdev=214.44, samples=59</font></div><div class=""><font face="IBMPlexMono" class=""> cpu : usr=2.43%, sys=18.18%, ctx=262222, majf=0, minf=8</font></div><div class=""><font face="IBMPlexMono" class=""> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%</font></div><div class=""><font face="IBMPlexMono" class=""> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%</font></div><div class=""><font face="IBMPlexMono" class=""> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%</font></div><div class=""><font face="IBMPlexMono" class=""> issued rwt: total=0,131072,0, short=0,0,0, dropped=0,0,0</font></div><div class=""><font face="IBMPlexMono" class=""> latency : target=0, window=0, percentile=100.00%, depth=32</font></div><div class=""><font face="IBMPlexMono" class=""><br class=""></font></div><div class=""><font face="IBMPlexMono" class="">Run status group 0 (all jobs):</font></div><div class=""><font face="IBMPlexMono" class=""> WRITE: bw=8880KiB/s (9093kB/s), 8880KiB/s-8880KiB/s (9093kB/s-9093kB/s), io=256MiB (268MB), run=29522-29522msec</font></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class="">With performance.write-behind-trickling-writes OFF:</div><div class=""><br class=""></div><div class="">## 4k randwrite - just over half the IOP/s of having it ON.</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class=""><font face="IBMPlexMono" class=""># fio --randrepeat=1 --ioengine=libaio --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=32 --size=256MB --readwrite=randwrite</font></div><div class=""><font face="IBMPlexMono" class="">test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32</font></div><div class=""><font face="IBMPlexMono" class="">fio-3.1</font></div><div class=""><font face="IBMPlexMono" class="">Starting 1 process</font></div><div class=""><font face="IBMPlexMono" class="">Jobs: 1 (f=1): [f(1)][100.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 00m:00s]</font></div><div class=""><font face="IBMPlexMono" class="">test: (groupid=0, jobs=1): err= 0: pid=44225: Tue Mar 20 15:11:04 2018</font></div><div class=""><font face="IBMPlexMono" class=""> write: <b class="">IOPS=2594, BW=10.1MiB/s</b> (10.6MB/s)(256MiB/25259msec)</font></div><div class=""><font face="IBMPlexMono" class=""> bw ( KiB/s): min= 2248, max=18728, per=100.00%, avg=10454.10, stdev=6481.14, samples=50</font></div><div class=""><font face="IBMPlexMono" class=""> iops : min= 562, max= 4682, avg=2613.50, stdev=1620.35, samples=50</font></div><div class=""><font face="IBMPlexMono" class=""> cpu : usr=2.29%, sys=10.09%, ctx=131141, majf=0, minf=7</font></div><div class=""><font face="IBMPlexMono" class=""> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%</font></div><div class=""><font face="IBMPlexMono" class=""> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%</font></div><div class=""><font face="IBMPlexMono" class=""> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%</font></div><div class=""><font face="IBMPlexMono" class=""> issued rwt: total=0,65536,0, short=0,0,0, dropped=0,0,0</font></div><div class=""><font face="IBMPlexMono" class=""> latency : target=0, window=0, percentile=100.00%, depth=32</font></div><div class=""><font face="IBMPlexMono" class=""><br class=""></font></div><div class=""><font face="IBMPlexMono" class="">Run status group 0 (all jobs):</font></div><div class=""><font face="IBMPlexMono" class=""> WRITE: bw=10.1MiB/s (10.6MB/s), 10.1MiB/s-10.1MiB/s (10.6MB/s-10.6MB/s), io=256MiB (268MB), run=25259-25259msec</font></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">## 2k randwrite - no noticable change.</div><div class=""><br class=""></div><div class=""><div class=""><font face="IBMPlexMono" class=""># fio --randrepeat=1 --ioengine=libaio --gtod_reduce=1 --name=test --filename=test --bs=2k --iodepth=32 --size=256MB --readwrite=randwrite</font></div><div class=""><font face="IBMPlexMono" class="">test: (g=0): rw=randwrite, bs=(R) 2048B-2048B, (W) 2048B-2048B, (T) 2048B-2048B, ioengine=libaio, iodepth=32</font></div><div class=""><font face="IBMPlexMono" class="">fio-3.1</font></div><div class=""><font face="IBMPlexMono" class="">Starting 1 process</font></div><div class=""><font face="IBMPlexMono" class="">Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=8662KiB/s][r=0,w=4331 IOPS][eta 00m:00s]</font></div><div class=""><font face="IBMPlexMono" class="">test: (groupid=0, jobs=1): err= 0: pid=45813: Tue Mar 20 15:12:02 2018</font></div><div class=""><font face="IBMPlexMono" class=""> write: <b class="">IOPS=4291, BW=8583KiB/s</b> (8789kB/s)(256MiB/30541msec)</font></div><div class=""><font face="IBMPlexMono" class=""> bw ( KiB/s): min= 7416, max=10264, per=99.94%, avg=8577.66, stdev=618.31, samples=61</font></div><div class=""><font face="IBMPlexMono" class=""> iops : min= 3708, max= 5132, avg=4288.84, stdev=309.15, samples=61</font></div><div class=""><font face="IBMPlexMono" class=""> cpu : usr=2.87%, sys=15.83%, ctx=262236, majf=0, minf=8</font></div><div class=""><font face="IBMPlexMono" class=""> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%</font></div><div class=""><font face="IBMPlexMono" class=""> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%</font></div><div class=""><font face="IBMPlexMono" class=""> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%</font></div><div class=""><font face="IBMPlexMono" class=""> issued rwt: total=0,131072,0, short=0,0,0, dropped=0,0,0</font></div><div class=""><font face="IBMPlexMono" class=""> latency : target=0, window=0, percentile=100.00%, depth=32</font></div><div class=""><font face="IBMPlexMono" class=""><br class=""></font></div><div class=""><font face="IBMPlexMono" class="">Run status group 0 (all jobs):</font></div><div class=""><font face="IBMPlexMono" class=""> WRITE: bw=8583KiB/s (8789kB/s), 8583KiB/s-8583KiB/s (8789kB/s-8789kB/s), io=256MiB (268MB), run=30541-30541msec</font></div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Let me know if you'd recommend any other benchmarks comparing performance.write-behind-trickling-writes ON/OFF (just nothing that'll seriously risk locking up the whole gluster cluster please!).</div><div class=""><br class=""></div><div class=""><br class=""></div></div>--<br class="">Sam McLeod<br class="">Please respond via email when possible.<br class=""><a href="https://smcleod.net" class="">https://smcleod.net</a><br class="">https://twitter.com/s_mcleod
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On 20 Mar 2018, at 2:56 pm, Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" class="">rgowdapp@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Tue, Mar 20, 2018 at 8:57 AM, Sam McLeod <span dir="ltr" class=""><<a href="mailto:mailinglists@smcleod.net" target="_blank" class="">mailinglists@smcleod.net</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class="">Hi Raghavendra,<div class=""><br class=""></div><div class=""><br class=""></div><div class=""><span class="gmail-"><div class=""></div><blockquote type="cite" class=""><div class=""><font class="">On 20 Mar 2018, at 1:55 pm, Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank" class="">rgowdapp@redhat.com</a>> wrote:</font></div><div class=""><div class="gmail_quote"><div class=""><font class=""><br class=""></font></div><div class=""><font class="">Aggregating large number of small writes by write-behind into large writes has been merged on master:<br class=""></font></div><div class=""><a href="https://github.com/gluster/glusterfs/issues/364" target="_blank" class=""><font class="">https://github.com/gluster/<wbr class="">glusterfs/issues/364</font></a></div><div class=""><font class=""><br class=""></font></div><div class=""><font class="">Would like to know whether it helps for this usecase. Note that its not part of any release yet. So you've to build and install from repo.</font></div></div></div></blockquote><div class=""><br class=""></div></span>Sounds interesting, not too keen to build packages at the moment but I've added myself as a watcher to that issue on Github and once it's in a 3.x release I'll try it and let you know.<span class="gmail-"><br class=""><br class=""><blockquote type="cite" class=""><div class=""><div class="gmail_quote"><div class=""><font class="">Another suggestion is to run tests with turning off option performance.write-behind-<wbr class="">trickling-writes.</font></div><div class=""><font class=""><br class=""></font></div><div class=""><font class=""># gluster volume set <volname> performance.write-behind-<wbr class="">trickling-writes off<br class=""></font></div><div class=""><font class=""><br class=""></font></div><div class=""><font class="">A word of caution though is if your files are too small, these suggestions may not have much impact.</font></div></div></div></blockquote></span><div class=""><div class="gmail_quote"><div class=""><br class=""></div><div class="">I'm looking for documentation on this option but all I could really find is in the source for write-behind.c:</div><div class=""><br class=""></div><div class="">if is enabled (which it is), do not hold back writes if there are no outstanding requests.</div></div></div></div></div></blockquote><div class=""><br class=""></div><div class="">Till recently this functionality though was available, couldn't be configured from cli. One could change this option by editing volume configuration file. However, now its configurable through cli:</div><div class=""><br class=""></div><div class=""><a href="https://review.gluster.org/#/c/18719/" class="">https://review.gluster.org/#/c/18719/</a><br class=""></div><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class=""><div class=""><div class="gmail_quote"><div class=""><br class=""></div><div class=""><br class=""></div><div class="">and a note on aggregate-size stating that </div><div class=""><div class=""><br class=""></div><div class=""><i class="">"aggregation won't happen if performance.write-behind-<wbr class="">trickling-writes is turned on"</i></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">What are the potentially negative performance impacts of disabling this?</div></div></div></div></div></div></blockquote><div class=""><br class=""></div><div class="">Even if aggregation option is turned off, write-behind has the capacity to aggregate till a size of 128KB. But, to completely make use of this in case of small write workloads write-behind has to wait for sometime so that there are enough number of write-requests to fill the capacity. With this option enabled, write-behind though aggregates existing requests, won't wait for future writes. This means descendant xlators of write-behind can see writes smaller than 128K. So, for a scenario where small number of large writes are preferred over large number of small sized writes, this can be a problem.<br class=""></div><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><span class="gmail-"><div class=""><br class=""><div class="">
<div dir="auto" style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class=""><div dir="auto" style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class=""><div dir="auto" style="overflow-wrap: break-word;" class=""><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class="">--<br class="">Sam McLeod (protoporpoise on IRC)</div><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" class=""><a href="https://smcleod.net/" target="_blank" class="">https://smcleod.net</a><br class=""><a href="https://twitter.com/s_mcleod" target="_blank" class="">https://twitter.com/s_mcleod</a><br class=""><br class="">Words are my own opinions and do not necessarily represent those of my employer or partners.</div></div></div></div>
</div>
<div class=""><br class=""></div></div></span></div></blockquote></div><br class=""></div></div>
</div></blockquote></div><br class=""></div></body></html>