<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Tony’s performance sounds significantly sub par from my experience. I did some testing with gluster 3.12 and Ovirt 3.9, on my running production cluster when I enabled the glfsapi, even my pre numbers are significantly better than what Tony is reporting:<div class=""><br class=""></div><div class="">———————————————————<span class="Apple-tab-span" style="white-space:pre"> </span></div><div class=""><div class="">Before using gfapi:</div><div class=""><br class=""></div><div class=""><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">]# dd if=/dev/urandom of=test.file bs=1M count=1024</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">1024+0 records in</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">1024+0 records out</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">1073741824 bytes (1.1 GB) copied, 90.1843 s, 11.9 MB/s</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"># echo 3 > /proc/sys/vm/drop_caches</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"># dd if=test.file of=/dev/null </span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">2097152+0 records in</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">2097152+0 records out</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">1073741824 bytes (1.1 GB) copied, 3.94715 s, 272 MB/s</span></div></div><div class=""><br class=""></div></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"># hdparm -tT /dev/vda</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 18px;"><span class="" style="font-variant-ligatures: no-common-ligatures;"></span><br class=""></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">/dev/vda:</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"> Timing cached reads: 17322 MB in 2.00 seconds = 8673.49 MB/sec</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"> Timing buffered disk reads: 996 MB in 3.00 seconds = 331.97 MB/sec</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"><br class=""></span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">#</span>bonnie++ -d . -s 8G -n 0 -m pre-glapi -f -b -u root</div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="color: rgb(0, 200, 255); font-family: "Fantasque Sans Mono"; font-size: 15px; background-color: rgba(0, 0, 0, 0.901961);"><br class=""></span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Version 1.97 ------Sequential Output------ --Sequential Input- --Random-</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">pre-glapi 8G 196245 30 105331 15 962775 49 1638 34</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Latency 1578ms 1383ms 201ms 301ms</span></div><div class=""><span class="" style="font-variant-ligatures: no-common-ligatures;"><br class=""></span></div><div class=""><span class="" style="font-variant-ligatures: no-common-ligatures;"><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Version 1.97 ------Sequential Output------ --Sequential Input- --Random-</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">pre-glapi 8G 155937 27 102899 14 1030285 54 1763 45</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Latency 694ms 1333ms 114ms 229ms</span></div><div class=""><br class=""></div><div class="">(note, sequential reads seem to have been influenced by caching somewhere…)</div></span></div></span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"><br class=""></span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">After switching to gfapi:</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"><br class=""></span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"># dd if=/dev/urandom of=test.file bs=1M count=1024</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">1024+0 records in</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">1024+0 records out</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">1073741824 bytes (1.1 GB) copied, 80.8317 s, 13.3 MB/s</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"># echo 3 > /proc/sys/vm/drop_caches</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"># dd if=test.file of=/dev/null </span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">2097152+0 records in</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">2097152+0 records out</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">1073741824 bytes (1.1 GB) copied, 3.3473 s, 321 MB/s</span></div></span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"><br class=""></span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"># hdparm -tT /dev/vda</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 18px;"><span class="" style="font-variant-ligatures: no-common-ligatures;"></span><br class=""></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">/dev/vda:</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"> Timing cached reads: 17112 MB in 2.00 seconds = 8568.86 MB/sec</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"> Timing buffered disk reads: 1406 MB in 3.01 seconds = 467.70 MB/sec</span></div></span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"><br class=""></span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">#bonnie++ -d . -s 8G -n 0 -m glapi -f -b -u root</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"><br class=""></span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Version 1.97 ------Sequential Output------ --Sequential Input- --Random-</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;"> glapi 8G 359100 59 185289 24 489575 31 2079 67</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Latency 160ms 355ms 36041us 185ms</span></div><div class=""><span class="" style="font-variant-ligatures: no-common-ligatures;"><br class=""></span></div><div class=""><span class="" style="font-variant-ligatures: no-common-ligatures;"><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Version 1.97 ------Sequential Output------ --Sequential Input- --Random-</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;"> glapi 8G 341307 57 180546 24 472572 35 2655 61</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Latency 153ms 394ms 101ms 116ms</span></div><div class=""><span class="" style="font-variant-ligatures: no-common-ligatures;"><br class=""></span></div><div class="">So excellent improvement in write throughput, but the significant improvement in latency is what was most noticed by users. Anecdotal reports of 2x+ performance improvements, with one remarking that it’s like having dedicated disks :)</div><div class=""><br class=""></div></span></div></span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">This system is on my production cluster, so it’s not getting exclusive disk access, but this VM is not doing anything else itself. The cluster is 3 xeon </span>E5-2609 v3 @ 1.90GHz servers w/ 64G ram, SATA2 disks; 2 with 9x spindles each, 1 with 8x slightly faster disks (all spinners). Using ZFS stripes with lz4 compression and 10G connectivity to 8 hosts. Running gluster 3.12.3 at the moment. The cluster itself has about 70 running VMs in varying states of switching to gfapi use, but my main sql servers are using their own volumes and not competing for this one. These have not yet had the spectre/meltdown patches applied.</div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"> </span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;">This will be skewed because I forced it to not steal all the ram on the server (reads will certainly be cached), but an idea of what it can do disk wise, on the volume used above:</span></div><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"><div class="" style="margin: 0px; font-stretch: normal; line-height: normal;"><span class="" style="font-variant-ligatures: no-common-ligatures;"># bonnie++ -d . -s 8G -n 0 -m zfs-server -f -b -u root -r 4096</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Version 1.97 ------Sequential Output------ --Sequential Input- --Random-</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">zfs-server 8G 604940 79 510410 87 1393862 99 3164 91</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; color: rgb(0, 200, 255); background-color: rgba(0, 0, 0, 0.901961);"><span class="" style="font-variant-ligatures: no-common-ligatures;">Latency 99545us 100ms 247us 152ms</span></div><div class=""><span class="" style="font-variant-ligatures: no-common-ligatures;"><br class=""></span></div><div class=""><span class="" style="font-variant-ligatures: no-common-ligatures;">Just for fun from one of the servers showing base load and this testing:</span></div><div class=""><span class="" style="font-variant-ligatures: no-common-ligatures;"><img apple-inline="yes" id="7412EE8A-222F-4005-B266-9CB5BA492E46" class="" src="cid:B68D24F8-81A5-4825-B0DE-39E6D46FA3BA@spaanem.onholyground.com"></span></div><div class=""><span class="" style="font-variant-ligatures: no-common-ligatures;"><br class=""></span></div><div class=""><span class="" style="font-variant-ligatures: no-common-ligatures;">——————————————————————————</span></div><div class=""><span class="" style="font-variant-ligatures: no-common-ligatures;"><br class=""></span></div></span></div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><hr style="border:none;border-top:solid #B5C4DF 1.0pt;padding:0 0 0 0;margin:10px 0 5px 0;" class=""><span style="margin: -1.3px 0.0px 0.0px 0.0px" id="RwhHeaderAttributes" class=""><font face="Helvetica" size="4" color="#000000" style="font: 13.0px Helvetica; color: #000000" class=""><b class="">From:</b> Vincent Royer <<a href="mailto:vincent@epicenergy.ca" class="">vincent@epicenergy.ca</a>></font></span><br class="">
<span style="margin: -1.3px 0.0px 0.0px 0.0px" class=""><font face="Helvetica" size="4" color="#000000" style="font: 13.0px Helvetica; color: #000000" class=""><b class="">Subject:</b> Re: [Gluster-users] Finding performance bottlenecks</font></span><br class="">
<span style="margin: -1.3px 0.0px 0.0px 0.0px" class=""><font face="Helvetica" size="4" color="#000000" style="font: 13.0px Helvetica; color: #000000" class=""><b class="">Date:</b> May 3, 2018 at 1:58:03 PM CDT</font></span><br class="">
<span style="margin: -1.3px 0.0px 0.0px 0.0px" class=""><font face="Helvetica" size="4" color="#000000" style="font: 13.0px Helvetica; color: #000000" class=""><b class="">To:</b> <a href="mailto:tony@hoyle.me.uk" class="">tony@hoyle.me.uk</a></font></span><br class="">
<span style="margin: -1.3px 0.0px 0.0px 0.0px" class=""><font face="Helvetica" size="4" color="#000000" style="font: 13.0px Helvetica; color: #000000" class=""><b class="">Cc:</b> <a href="mailto:gluster-users@gluster.org" class="">gluster-users@gluster.org</a></font></span><br class="">
<br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="">It worries me how many threads talk about low performance. I'm about to build out a replica 3 setup and run Ovirt with a bunch of Windows VMs. </div><div class=""><br class=""></div><div class="">Are the issues Tony is experiencing "normal" for Gluster? Does anyone here have a system with windows VMs and have good performance? </div><div class=""><br class=""></div><div class=""><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div class=""><div dir="ltr" style="font-size:12.8px" class=""><font size="4" face="arial narrow, sans-serif" class=""><b class="">Vincent Royer</b></font></div><div dir="ltr" class=""><div style="font-size:12.8px" class=""><font size="4" face="arial narrow, sans-serif" class=""><b class="">778-825-1057</b></font></div><div style="font-size:12.8px" class=""><font size="4" face="arial narrow, sans-serif" class=""><b class=""><br class=""></b></font></div><div style="font-size:12.8px" class=""><a href="http://www.epicenergy.ca/" target="_blank" class=""><img src="https://lh3.googleusercontent.com/ezD9JxeN67lTscselHjW4ZEqvQRj_i_T_Ky_s__GkR0AtanG1nK52CD1Rdhmyzbk7XRPf-Rhhnr2-SLPmhtd0J8xVMp0xOiR20ebQVb6jU2xwJPKN4lePRq5G2p93ybqA4SnqjCWCvEsNlK37a9FtStdFcTJyIoS2aHKiL69L3bZg1g7mEAMr7IHXVYvHgq3MX9rNaqIQx9T3uPuIHEDIfxrgpCrsBoEamgaRHj5hcpEobnogqCqpa1sRcKy927TSeR5BbVLM1-wn0h-4LvE8CMqj30jpQULxZ9uXEg_S6sR3ZeZjtkkRvgK4l_wiivWlyV_SjCDQ5QREG3jkuCjXC04i1NNmziGZZNWYsaohE6khrN4JGVtvaSOiS2woYJNz3Gt0EucxJp3eUBYspzTl9iZia7bLvFEGIODvmfjRVZJmnCrFucJxGkAtWn_EUIbxJXS5NWAFnvBoixQP563Fbu7dFKL77lDcb4PHwrnJVjqcgU0OTeiMBlMvEt97YgMpMvdi7rx9L4DeEbrIGUovgx4QLfRRoF_9u2mxFu8FHupCxfocSyB3vs8gqOLvVdx42qwiLN0sGLAg8K3G_vFwVNddXQ_yb3cGeCEAOTnk7jfiP4O4jsPIW2RRbnKKL-usL11Tp7gkxbgRouLOoyqpbisjA4vRWDeylj1yDGyBg=w400-h38-no" width="200" height="18" class="hoverZoomLink"><br class=""></a></div><div class=""><b class=""><font face="arial narrow, sans-serif" size="2" class="">SUSTAINABLE MOBILE <font color="#6aa84f" class="">ENERGY </font>SOLUTIONS</font></b></div></div></div><div class=""><br class=""></div><div class=""><br class=""><div class=""><div class=""><br class=""></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="">On Wed, May 2, 2018 at 7:52 AM Tony Hoyle <<a href="mailto:tony@hoyle.me.uk" class="">tony@hoyle.me.uk</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 01/05/2018 02:27, Thing wrote:<br class="">
> Hi,<br class="">
> <br class="">
> So is the KVM or Vmware as the host(s)? I basically have the same setup<br class="">
> ie 3 x 1TB "raid1" nodes and VMs, but 1gb networking. I do notice with<br class="">
> vmware using NFS disk was pretty slow (40% of a single disk) but this<br class="">
> was over 1gb networking which was clearly saturating. Hence I am moving<br class="">
> to KVM to use glusterfs hoping for better performance and bonding, it<br class="">
> will be interesting to see which host type runs faster.<br class="">
<br class="">
1gb will always be the bottleneck in that situation - that's going too<br class="">
max out at the speed of a single disk or lower. You need at minimum to<br class="">
bond interfaces and preferably go to 10gb to do that.<br class="">
<br class="">
Our NFS actually ends up faster than local disk because the read speed<br class="">
of the raid is faster than the read speed of the local disk.<br class="">
<br class="">
> Which operating system is gluster on? <br class="">
<br class="">
Debian Linux. Supermicro motherboards, 24 core i7 with 128GB of RAM on<br class="">
the VM hosts.<br class="">
<br class="">
> Did you do iperf between all nodes?<br class="">
<br class="">
Yes, around 9.7Gb/s<br class="">
<br class="">
It doesn't appear to be raw read speed but iowait. Under nfs load with<br class="">
multiple VMs I get an iowait of around 0.3%. Under gluster, never less<br class="">
than 10% and glusterfsd is often the top of the CPU usage. This causes<br class="">
a load average of ~12 compared to 3 over NFS, and absolutely kills VMs<br class="">
esp. Windows ones - one machine I set booting and it was still booting<br class="">
30 minutes later!<br class="">
<br class="">
Tony<br class="">
_______________________________________________<br class="">
Gluster-users mailing list<br class="">
<a href="mailto:Gluster-users@gluster.org" target="_blank" class="">Gluster-users@gluster.org</a><br class="">
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank" class="">http://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div>
_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">http://lists.gluster.org/mailman/listinfo/gluster-users</div></blockquote></div><br class=""></div></body></html>