<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><font face="Monaco" size="4" class="">Background: 4 identical gluster servers with 15 TB each in 2x2 setup.</font><div class=""><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><font face="Monaco" size="4" class="">CentOS Linux release 7.3.1611 (Core)</font></span></div></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><font face="Monaco" size="4" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">glust</span>erfs-server-3.9.1-1.el7.x86_64</font></span></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><font face="Monaco" size="4" class="">client systems are using:</font></span></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">glusterfs-client<span class="Apple-tab-span" style="white-space:pre"> </span></span> 3.5.2-2+deb8u3</font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class=""><br class=""></font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class="">The cluster has ~12 TB in use with 21 million files. Lots of jpgs. About 12 clients are mounting gluster volumes. </font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class=""><br class=""></font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class="">Network load is light: iftop shows each server has 10-15 Mbit reads and about half that in writes.</font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class=""><br class=""></font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class="">What I’m seeing that concerns me is that one box, gluster4, has roughly twice the CPU utilization and twice or more the load average of the other three servers. gluster4 has a 24 hour average of about 30% CPU utilization, something that seems to me to be way out of line for a couple MB/sec of traffic.</font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class=""><br class=""></font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class="">In running volume top, the odd thing I see is that for gluster1-3 I get latency summaries like this:</font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><div style="margin: 0px; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><font face="Monaco" size="4" class="">Brick: <a href="http://gluster1.publicinteractive.com" class="">gluster1.publicinteractive.com</a>:/gluster/drupal_prod</font></span></div><div style="margin: 0px; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><font face="Monaco" size="4" class="">—————————————————————————————</font></span></div><div style="margin: 0px; line-height: normal;" class=""><font face="Monaco" size="4" class="">%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</font></div><div style="margin: 0px; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><font face="Monaco" size="4" class=""> -------- ----------- ----------- ----------- ------------ ----</font></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><font face="Monaco" size="4" class=""><br class=""></font></span></div></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><div style="margin: 0px; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><font face="Monaco" size="4" class=""> 9.96 675.07 us 15.00 us 1067793.00 us 205060 INODELK </font></span></div><div style="margin: 0px; line-height: normal;" class=""><font face="Monaco" size="4" class="">15.85 3414.20 us 16.00 us 773621.00 us 64494 READ</font></div><div style="margin: 0px; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><font face="Monaco" size="4" class="">51.35 2235.96 us 12.00 us 1093609.00 us 319120 LOOKUP</font></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><font face="Monaco" size="4" class=""><br class=""></font></span></div></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class="">… but my problem server has far more inodelk latency:</font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class=""><br class=""></font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><div style="margin: 0px; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><font face="Monaco" size="4" class="">12.01 4712.03 us 17.00 us 1773590.00 us 47214 READ</font></span></div><div style="margin: 0px; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><font face="Monaco" size="4" class="">27.50 2390.27 us 14.00 us 1877571.00 us 213121 INODELK</font></span></div><div style="margin: 0px; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><font face="Monaco" size="4" class="">28.70 1643.65 us 12.00 us 1837696.00 us 323407 LOOKUP</font></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><font face="Monaco" size="4" class=""><br class=""></font></span></div></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class="">The servers are intended to be identical, and are indeed identical hardware.</font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class=""><br class=""></font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class="">Suggestions on where to look or which FM to RT ver welcome indeed.</font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class=""><br class=""></font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class="">Thanks,</font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class=""><br class=""></font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class="">David</font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class=""><br class=""></font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class=""><br class=""></font></div><div style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);" class=""><font face="Monaco" size="4" class=""><br class=""></font></div><div style="margin: 0px; font-size: 13px; line-height: normal; font-family: Monaco; background-color: rgb(255, 255, 255);" class=""><br class=""></div></body></html>