<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">I’ve had issues like that while playing with different healing config, too many shd threads will drive load higher than it needs to be and keep the boxes busy. I’ve had better luck with<div class=""><br class=""></div><div class=""><div class="">cluster.shd-max-threads: 1</div><div class="">cluster.shd-wait-qlength: 1024</div><div class=""><br class=""></div><div class="">than 8 & 10000. It does heal a little faster with 8/10k than 1/1k, but it keeps the servers too busy to do anything else. In a hyper converged install, I could see it causing problems for the VMs and keeping healing from effectively happening (it would cause disk read issues for me even with a separate server cluster).</div><div class=""><br class=""></div><div class="">Also, what version are you running? If it’s 3.12, I’m finding that .9-.11 have a client side memory leak, just havn’t gotten everything together enough to post about it. .8 seems ok, and .6 is definitely ok though, flat memory use from both, steadily increasing from .9 and .11 clients.</div><div class=""><br class=""></div><div class="">Good luck!</div><div class=""><br class=""></div><div class=""> -Darrell</div><div><blockquote type="cite" class=""><hr style="border:none;border-top:solid #B5C4DF 1.0pt;padding:0 0 0 0;margin:10px 0 5px 0;" class=""><span style="margin: -1.3px 0.0px 0.0px 0.0px" id="RwhHeaderAttributes" class=""><font face="Helvetica" size="4" color="#000000" style="font: 13.0px Helvetica; color: #000000" class=""><b class="">From:</b> Jim Kusznir <<a href="mailto:jim@palousetech.com" class="">jim@palousetech.com</a>></font></span><br class="">
<span style="margin: -1.3px 0.0px 0.0px 0.0px" class=""><font face="Helvetica" size="4" color="#000000" style="font: 13.0px Helvetica; color: #000000" class=""><b class="">Subject:</b> [Gluster-users] Gluster-induced load averages of 50-200</font></span><br class="">
<span style="margin: -1.3px 0.0px 0.0px 0.0px" class=""><font face="Helvetica" size="4" color="#000000" style="font: 13.0px Helvetica; color: #000000" class=""><b class="">Date:</b> July 6, 2018 at 11:43:21 AM CDT</font></span><br class="">
<span style="margin: -1.3px 0.0px 0.0px 0.0px" class=""><font face="Helvetica" size="4" color="#000000" style="font: 13.0px Helvetica; color: #000000" class=""><b class="">To:</b> gluster-user</font></span><br class="">
<br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hello:<div class=""><br class=""></div><div class="">I have a hyperconverged ovirt cluster with 3 replica 2+arb and one replica 3 volume. My replica 3 volume has been having lots of problems for a while now.</div><div class=""><br class=""></div><div class="">First, after a week or two, I notice a lot of RAM usage (half of my physical ram, plus lots of swap) for the replica 3 gluster process. The amount of ram in use varies by machine, but they all have had the problem show up (one at a time). Often the OOM killer kills it off before I catch it.</div><div class=""><br class=""></div><div class="">This last time, I caught it. I migrated all the machines off the problem server, and rebooted it. That was yesterday (thurs at 9am). Shortly after coming back up, ALL 3 of my servers are showing very high load averages (cpu % as reported by top is fairly low, althoguh iowait will spike to 25%). The one process that is using the most CPU is the glusterfsd process for the replica 3 volume. It is consistant on all 3 machines, and it loads down the machine and IO enough to be causing issues with the VMs running on it (even IO to a seprate disk on the same machine) . </div><div class=""><br class=""></div><div class="">I watched network I/O and I can see occasional spikes to 50MB/s (gig link), but its not consistant, and often spends considerable time near zero or in kB/s range. the repica 3 has its own dedicated gig network.</div><div class=""><br class=""></div><div class="">As of last night I still had 4 unhealed items for the server I rebooted. This AM, I'm down to 1, but its still there, and its still causing considerable performance issues. On a server, if I stop that glusterfsd process, then the loadavg returns to normal quickly.</div><div class=""><br class=""></div><div class="">HELP...what do I do? Lately it seems that all my outages and performace issues have been glusterfsd taking down my servers that they are running on. Is gluster ready for production, or did I misunderstand? My reliability has been abismal on my vm cluster, and I still have 3 VMs that won't boot anymore due to disk corruption caused by my last gluster outage (which was the high RAM until it crashed on the server due to kernel OOM killer).</div><div class=""><br class=""></div><div class="">Here's some info on the affected gluster volume:</div><div class=""><div class="">Volume Name: data-hdd</div><div class="">Type: Replicate</div><div class="">Volume ID: d342a3ab-16f3-49f0-bbcf-f788be8ac5f1</div><div class="">Status: Started</div><div class="">Snapshot Count: 0</div><div class="">Number of Bricks: 1 x 3 = 3</div><div class="">Transport-type: tcp</div><div class="">Bricks:</div><div class="">Brick1: 172.172.1.11:/gluster/brick3/data-hdd</div><div class="">Brick2: 172.172.1.12:/gluster/brick3/data-hdd</div><div class="">Brick3: 172.172.1.13:/gluster/brick3/data-hdd</div><div class="">Options Reconfigured:</div><div class="">diagnostics.count-fop-hits: on</div><div class="">diagnostics.latency-measurement: on</div><div class="">transport.address-family: inet</div><div class="">performance.readdir-ahead: on</div><div class="">geo-replication.indexing: on</div><div class="">geo-replication.ignore-pid-check: on</div><div class="">changelog.changelog: on</div><div class="">performance.quick-read: off</div><div class="">performance.read-ahead: off</div><div class=""><a href="http://performance.io" class="">performance.io</a>-cache: off</div><div class="">performance.stat-prefetch: off</div><div class="">performance.low-prio-threads: 32</div><div class="">network.remote-dio: enable</div><div class="">cluster.eager-lock: off</div><div class="">cluster.quorum-type: auto</div><div class="">cluster.server-quorum-type: server</div><div class="">cluster.data-self-heal-algorithm: full</div><div class="">cluster.locking-scheme: granular</div><div class="">cluster.shd-max-threads: 8</div><div class="">cluster.shd-wait-qlength: 10000</div><div class="">features.shard: on</div><div class="">user.cifs: off</div><div class="">storage.owner-uid: 36</div><div class="">storage.owner-gid: 36</div><div class="">server.allow-insecure: on</div><div class="">network.ping-timeout: 30</div></div><div class=""><br class=""></div><div class="">---------------</div><div class=""><br class=""></div><div class=""><div class="">[root@ovirt3 ~]# gluster volume profile data-hdd info</div><div class="">Brick: 172.172.1.13:/gluster/brick3/data-hdd</div><div class="">--------------------------------------------</div><div class="">Cumulative Stats:</div><div class=""> Block Size: 256b+ 512b+ 1024b+</div><div class=""> No. of Reads: 188488 6126 17234</div><div class="">No. of Writes: 80 331655 3899</div><div class=""><br class=""></div><div class=""> Block Size: 2048b+ 4096b+ 8192b+</div><div class=""> No. of Reads: 17228 1371069 265013</div><div class="">No. of Writes: 6365 26617689 15776028</div><div class=""><br class=""></div><div class=""> Block Size: 16384b+ 32768b+ 65536b+</div><div class=""> No. of Reads: 350572 636120 189538</div><div class="">No. of Writes: 4475250 1557330 604455</div><div class=""><br class=""></div><div class=""> Block Size: 131072b+ 262144b+</div><div class=""> No. of Reads: 6442414 0</div><div class="">No. of Writes: 5194964 13</div><div class=""> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div class=""> --------- ----------- ----------- ----------- ------------ ----</div><div class=""> 0.00 0.00 us 0.00 us 0.00 us 63 FORGET</div><div class=""> 0.00 0.00 us 0.00 us 0.00 us 9356 RELEASE</div><div class=""> 0.00 0.00 us 0.00 us 0.00 us 113830 RELEASEDIR</div><div class=""> 0.00 765.33 us 102.00 us 2091.00 us 3 FGETXATTR</div><div class=""> 0.00 352.90 us 263.00 us 1397.00 us 31 TRUNCATE</div><div class=""> 0.00 1094.05 us 360.00 us 5643.00 us 21 MKDIR</div><div class=""> 0.00 647.01 us 260.00 us 4059.00 us 83 MKNOD</div><div class=""> 0.00 899.64 us 68.00 us 27531.00 us 105 SETXATTR</div><div class=""> 0.00 139.19 us 26.00 us 4619.00 us 740 ENTRYLK</div><div class=""> 0.00 2121.54 us 1250.00 us 9726.00 us 63 RENAME</div><div class=""> 0.00 254.93 us 18.00 us 169316.00 us 932 FLUSH</div><div class=""> 0.00 79.37 us 14.00 us 3120.00 us 3210 LK</div><div class=""> 0.00 9377.76 us 374.00 us 197032.00 us 63 CREATE</div><div class=""> 0.00 4777.26 us 91.00 us 757927.00 us 197 FTRUNCATE</div><div class=""> 0.00 1787.09 us 117.00 us 505178.00 us 767 XATTROP</div><div class=""> 0.00 180.43 us 13.00 us 796920.00 us 33471 GETXATTR</div><div class=""> 0.01 398.80 us 72.00 us 2581011.00 us 18792 REMOVEXATTR</div><div class=""> 0.01 378.54 us 21.00 us 264034.00 us 32293 READDIR</div><div class=""> 0.01 152.76 us 1.00 us 952682.00 us 113830 OPENDIR</div><div class=""> 0.02 1460.88 us 64.00 us 4693687.00 us 18918 SETATTR</div><div class=""> 0.02 524.47 us 30.00 us 359603.00 us 56913 READDIRP</div><div class=""> 0.02 3564.23 us 59.00 us 4818020.00 us 9307 OPEN</div><div class=""> 0.04 125.61 us 21.00 us 355789.00 us 400177 STAT</div><div class=""> 0.11 132.13 us 24.00 us 460615.00 us 1143725 STATFS</div><div class=""> 0.26 111.57 us 18.00 us 3533816.00 us 3153747 FSTAT</div><div class=""> 0.81 383.95 us 22.00 us 3896810.00 us 2827323 LOOKUP</div><div class=""> 5.51 74.04 us 13.00 us 1883274.00 us 100195900 FINODELK</div><div class=""> 5.59 258.57 us 13.00 us 11221929.00 us 29091404 INODELK</div><div class=""> 6.50 922.80 us 25.00 us 7548801.00 us 9483802 READ</div><div class=""> 10.20 358.55 us 56.00 us 14993728.00 us 38295759 FXATTROP</div><div class=""> 12.42 306.31 us 47.00 us 30915095.00 us 54567728 WRITE</div><div class=""> 58.45 2495.18 us 43.00 us 21663381.00 us 31520936 FSYNC</div><div class=""><br class=""></div><div class=""> Duration: 1946380 seconds</div><div class=""> Data Read: 894661069137 bytes</div><div class="">Data Written: 1142499928350 bytes</div><div class=""><br class=""></div><div class="">Interval 2 Stats:</div><div class=""> Block Size: 256b+ 512b+ 1024b+</div><div class=""> No. of Reads: 0 3 0</div><div class="">No. of Writes: 15 3702 126</div><div class=""><br class=""></div><div class=""> Block Size: 2048b+ 4096b+ 8192b+</div><div class=""> No. of Reads: 0 9 58</div><div class="">No. of Writes: 198 62716 20361</div><div class=""><br class=""></div><div class=""> Block Size: 16384b+ 32768b+ 65536b+</div><div class=""> No. of Reads: 13 1 1</div><div class="">No. of Writes: 13640 9197 3948</div><div class=""><br class=""></div><div class=""> Block Size: 131072b+ 262144b+</div><div class=""> No. of Reads: 1826585 0</div><div class="">No. of Writes: 7387 5</div><div class=""> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div class=""> --------- ----------- ----------- ----------- ------------ ----</div><div class=""> 0.00 0.00 us 0.00 us 0.00 us 15 FORGET</div><div class=""> 0.00 0.00 us 0.00 us 0.00 us 3648 RELEASE</div><div class=""> 0.00 0.00 us 0.00 us 0.00 us 2421 RELEASEDIR</div><div class=""> 0.00 293.86 us 263.00 us 327.00 us 7 TRUNCATE</div><div class=""> 0.00 72.55 us 30.00 us 152.00 us 90 ENTRYLK</div><div class=""> 0.00 546.00 us 374.00 us 788.00 us 15 CREATE</div><div class=""> 0.00 475.86 us 178.00 us 2572.00 us 51 XATTROP</div><div class=""> 0.00 75.50 us 24.00 us 301.00 us 424 FLUSH</div><div class=""> 0.00 2889.13 us 1557.00 us 9171.00 us 15 RENAME</div><div class=""> 0.00 195.00 us 109.00 us 866.00 us 313 REMOVEXATTR</div><div class=""> 0.00 225.66 us 117.00 us 858.00 us 341 SETATTR</div><div class=""> 0.00 80.26 us 25.00 us 534.00 us 1041 LK</div><div class=""> 0.00 284.35 us 24.00 us 3965.00 us 408 READDIR</div><div class=""> 0.01 134.66 us 2.00 us 17518.00 us 2421 OPENDIR</div><div class=""> 0.01 202.33 us 34.00 us 15815.00 us 2413 FSTAT</div><div class=""> 0.01 624.48 us 23.00 us 279600.00 us 820 GETXATTR</div><div class=""> 0.04 107.65 us 29.00 us 1004.00 us 14106 STATFS</div><div class=""> 0.45 73.59 us 15.00 us 60009.00 us 250150 FINODELK</div><div class=""> 0.45 5080.05 us 62.00 us 1940405.00 us 3631 OPEN</div><div class=""> 0.97 869.22 us 35.00 us 1942304.00 us 45625 LOOKUP</div><div class=""> 1.50 378.77 us 58.00 us 1833454.00 us 163094 FXATTROP</div><div class=""> 1.77 599.60 us 55.00 us 1933975.00 us 121295 WRITE</div><div class=""> 15.58 5494.71 us 50.00 us 1958942.00 us 116418 FSYNC</div><div class=""> 29.74 668.39 us 52.00 us 1945979.00 us 1826670 READ</div><div class=""> 49.46 243.86 us 13.00 us 1945794.00 us 8325886 INODELK</div><div class=""><br class=""></div><div class=""> Duration: 42487 seconds</div><div class=""> Data Read: 239414982144 bytes</div><div class="">Data Written: 2475540892 bytes</div><div class=""><br class=""></div><div class="">Brick: 172.172.1.11:/gluster/brick3/data-hdd</div><div class="">--------------------------------------------</div><div class="">Cumulative Stats:</div><div class=""> Block Size: 256b+ 512b+ 1024b+</div><div class=""> No. of Reads: 7915 524 274</div><div class="">No. of Writes: 31 9934 169</div><div class=""><br class=""></div><div class=""> Block Size: 2048b+ 4096b+ 8192b+</div><div class=""> No. of Reads: 351 36927 5673</div><div class="">No. of Writes: 278 621338 316601</div><div class=""><br class=""></div><div class=""> Block Size: 16384b+ 32768b+ 65536b+</div><div class=""> No. of Reads: 6003 2366 3082</div><div class="">No. of Writes: 110118 45664 16295</div><div class=""><br class=""></div><div class=""> Block Size: 131072b+ 262144b+</div><div class=""> No. of Reads: 3710107 0</div><div class="">No. of Writes: 556659 6</div><div class=""> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div class=""> --------- ----------- ----------- ----------- ------------ ----</div><div class=""> 0.00 0.00 us 0.00 us 0.00 us 29 FORGET</div><div class=""> 0.00 0.00 us 0.00 us 0.00 us 6295 RELEASE</div><div class=""> 0.00 0.00 us 0.00 us 0.00 us 5574 RELEASEDIR</div><div class=""> 0.00 164.00 us 164.00 us 164.00 us 1 SETXATTR</div><div class=""> 0.00 92.33 us 66.00 us 130.00 us 3 FGETXATTR</div><div class=""> 0.00 308.93 us 215.00 us 408.00 us 15 TRUNCATE</div><div class=""> 0.00 70.94 us 22.00 us 271.00 us 186 ENTRYLK</div><div class=""> 0.00 70.06 us 21.00 us 265.00 us 669 FLUSH</div><div class=""> 0.00 1974.90 us 1490.00 us 6096.00 us 31 RENAME</div><div class=""> 0.00 76.58 us 24.00 us 3869.00 us 2582 LK</div><div class=""> 0.00 3252.80 us 175.00 us 244398.00 us 106 XATTROP</div><div class=""> 0.00 377.53 us 22.00 us 33026.00 us 1292 READDIR</div><div class=""> 0.00 940.44 us 85.00 us 563301.00 us 755 REMOVEXATTR</div><div class=""> 0.00 134434.50 us 119.00 us 563308.00 us 6 FTRUNCATE</div><div class=""> 0.00 677.32 us 19.00 us 641166.00 us 2303 GETXATTR</div><div class=""> 0.00 296.94 us 2.00 us 605194.00 us 5575 OPENDIR</div><div class=""> 0.00 3015.49 us 112.00 us 2198121.00 us 793 SETATTR</div><div class=""> 0.00 144086.32 us 461.00 us 1973731.00 us 31 CREATE</div><div class=""> 0.00 122.50 us 21.00 us 470389.00 us 37921 STATFS</div><div class=""> 0.00 322.28 us 25.00 us 2826178.00 us 17238 STAT</div><div class=""> 0.01 151.48 us 22.00 us 460622.00 us 77757 FSTAT</div><div class=""> 0.01 4641.92 us 35.00 us 7009636.00 us 3665 READDIRP</div><div class=""> 0.01 3514.85 us 61.00 us 9109607.00 us 6278 OPEN</div><div class=""> 0.04 800.50 us 23.00 us 8211795.00 us 109977 LOOKUP</div><div class=""> 0.60 1279.23 us 59.00 us 237262387.00 us 1023399 FXATTROP</div><div class=""> 1.50 872.91 us 35.00 us 6929426.00 us 3773222 READ</div><div class=""> 1.61 2096.57 us 50.00 us 237261777.00 us 1677093 WRITE</div><div class=""> 6.17 16780.27 us 51.00 us 22990372.00 us 804858 FSYNC</div><div class=""> 11.35 1890.83 us 14.00 us 1066342526.00 us 13144090 INODELK</div><div class=""> 78.70 73624.16 us 15.00 us 759064386.00 us 2340006 FINODELK</div><div class=""><br class=""></div><div class=""> Duration: 84992 seconds</div><div class=""> Data Read: 486964889723 bytes</div><div class="">Data Written: 84167404788 bytes</div><div class=""><br class=""></div><div class="">Interval 2 Stats:</div><div class=""> Block Size: 256b+ 512b+ 1024b+</div><div class=""> No. of Reads: 3861 337 173</div><div class="">No. of Writes: 15 3702 126</div><div class=""><br class=""></div><div class=""> Block Size: 2048b+ 4096b+ 8192b+</div><div class=""> No. of Reads: 159 5308 857</div><div class="">No. of Writes: 198 62716 20361</div><div class=""><br class=""></div><div class=""> Block Size: 16384b+ 32768b+ 65536b+</div><div class=""> No. of Reads: 1008 1004 621</div><div class="">No. of Writes: 13640 9197 3948</div><div class=""><br class=""></div><div class=""> Block Size: 131072b+ 262144b+</div><div class=""> No. of Reads: 2341344 0</div><div class="">No. of Writes: 7387 5</div><div class=""> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div class=""> --------- ----------- ----------- ----------- ------------ ----</div><div class=""> 0.00 0.00 us 0.00 us 0.00 us 15 FORGET</div><div class=""> 0.00 0.00 us 0.00 us 0.00 us 3647 RELEASE</div><div class=""> 0.00 0.00 us 0.00 us 0.00 us 2541 RELEASEDIR</div><div class=""> 0.00 267.86 us 215.00 us 322.00 us 7 TRUNCATE</div><div class=""> 0.00 66.18 us 22.00 us 144.00 us 90 ENTRYLK</div><div class=""> 0.00 450.02 us 197.00 us 2730.00 us 51 XATTROP</div><div class=""> 0.00 66.45 us 21.00 us 265.00 us 423 FLUSH</div><div class=""> 0.00 2049.80 us 1504.00 us 6096.00 us 15 RENAME</div><div class=""> 0.00 193.90 us 85.00 us 1329.00 us 313 REMOVEXATTR</div><div class=""> 0.00 74.23 us 24.00 us 3869.00 us 1041 LK</div><div class=""> 0.00 258.23 us 112.00 us 12048.00 us 341 SETATTR</div><div class=""> 0.00 384.60 us 22.00 us 2284.00 us 647 READDIR</div><div class=""> 0.00 415.01 us 19.00 us 122660.00 us 1306 GETXATTR</div><div class=""> 0.00 495.05 us 2.00 us 605194.00 us 2542 OPENDIR</div><div class=""> 0.00 98.96 us 30.00 us 2284.00 us 14106 STATFS</div><div class=""> 0.00 194.92 us 25.00 us 628797.00 us 8675 STAT</div><div class=""> 0.00 849.86 us 35.00 us 369113.00 us 2005 READDIRP</div><div class=""> 0.00 585.27 us 61.00 us 279889.00 us 3630 OPEN</div><div class=""> 0.00 141763.87 us 461.00 us 1973731.00 us 15 CREATE</div><div class=""> 0.00 155.27 us 23.00 us 169047.00 us 21344 FSTAT</div><div class=""> 0.02 442.07 us 26.00 us 544143.00 us 45744 LOOKUP</div><div class=""> 0.11 641.69 us 59.00 us 2227486.00 us 163094 FXATTROP</div><div class=""> 0.16 1215.83 us 58.00 us 2628020.00 us 121295 WRITE</div><div class=""> 1.25 9910.79 us 51.00 us 5116269.00 us 116418 FSYNC</div><div class=""> 1.30 511.11 us 35.00 us 2493646.00 us 2354672 READ</div><div class=""> 13.86 1535.67 us 14.00 us 688678811.00 us 8321169 INODELK</div><div class=""> 83.28 309339.72 us 16.00 us 688564320.00 us 248271 FINODELK</div><div class=""><br class=""></div><div class=""> Duration: 42487 seconds</div><div class=""> Data Read: 307016229315 bytes</div><div class="">Data Written: 2475540892 bytes</div><div class=""><br class=""></div><div class="">Brick: 172.172.1.12:/gluster/brick3/data-hdd</div><div class="">--------------------------------------------</div><div class="">Cumulative Stats:</div><div class=""> Block Size: 256b+ 512b+ 1024b+</div><div class=""> No. of Reads: 3706 247 142</div><div class="">No. of Writes: 15 3600 124</div><div class=""><br class=""></div><div class=""> Block Size: 2048b+ 4096b+ 8192b+</div><div class=""> No. of Reads: 94 5848 956</div><div class="">No. of Writes: 194 60615 19772</div><div class=""><br class=""></div><div class=""> Block Size: 16384b+ 32768b+ 65536b+</div><div class=""> No. of Reads: 1287 12947 1248</div><div class="">No. of Writes: 13126 8886 3810</div><div class=""><br class=""></div><div class=""> Block Size: 131072b+ 262144b+</div><div class=""> No. of Reads: 959349 0</div><div class="">No. of Writes: 3178716 5</div><div class=""> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div class=""> --------- ----------- ----------- ----------- ------------ ----</div><div class=""> 0.00 0.00 us 0.00 us 0.00 us 13 FORGET</div><div class=""> 0.00 0.00 us 0.00 us 0.00 us 3509 RELEASE</div><div class=""> 0.00 0.00 us 0.00 us 0.00 us 2425 RELEASEDIR</div><div class=""> 0.00 166.00 us 97.00 us 301.00 us 3 SETXATTR</div><div class=""> 0.00 42.62 us 19.00 us 123.00 us 90 ENTRYLK</div><div class=""> 0.00 431.00 us 80.00 us 8036.00 us 37 FTRUNCATE</div><div class=""> 0.00 160.27 us 17.00 us 63213.00 us 1041 LK</div><div class=""> 0.00 85684.71 us 166.00 us 598415.00 us 7 TRUNCATE</div><div class=""> 0.00 74683.31 us 143.00 us 2544472.00 us 35 XATTROP</div><div class=""> 0.00 22002.58 us 19.00 us 11281413.00 us 520 READDIR</div><div class=""> 0.01 6248116.73 us 1217.00 us 52589867.00 us 15 RENAME</div><div class=""> 0.01 6913389.40 us 332.00 us 34408990.00 us 15 CREATE</div><div class=""> 0.06 294044.50 us 25.00 us 100581145.00 us 1878 READDIRP</div><div class=""> 0.08 2357246.40 us 71.00 us 159200810.00 us 306 REMOVEXATTR</div><div class=""> 0.13 83641.44 us 20.00 us 69913926.00 us 13691 STATFS</div><div class=""> 0.14 1284416.00 us 15.00 us 173040222.00 us 964 GETXATTR</div><div class=""> 0.18 3713522.62 us 14.00 us 372969001.00 us 427 FLUSH</div><div class=""> 0.18 456678.40 us 53.00 us 130997224.00 us 3508 OPEN</div><div class=""> 0.34 1255418.93 us 1.00 us 267727475.00 us 2425 OPENDIR</div><div class=""> 0.64 16601795.78 us 73.00 us 392869613.00 us 347 SETATTR</div><div class=""> 0.74 235518.84 us 19.00 us 384250227.00 us 28134 FSTAT</div><div class=""> 1.10 1203934.76 us 16.00 us 407840076.00 us 8208 STAT</div><div class=""> 2.41 22009.35 us 29.00 us 421083048.00 us 985824 READ</div><div class=""> 2.53 93906.00 us 13.00 us 118521408.00 us 241629 FINODELK</div><div class=""> 3.78 4111.77 us 12.00 us 419936158.00 us 8251369 INODELK</div><div class=""> 3.84 218447.60 us 83.00 us 498122410.00 us 158007 FXATTROP</div><div class=""> 4.02 817162.39 us 19.00 us 409893532.00 us 44246 LOOKUP</div><div class=""> 25.08 68523.67 us 49.00 us 516930780.00 us 3288855 WRITE</div><div class=""> 54.73 4364231.40 us 51.00 us 688577405.00 us 112686 FSYNC</div><div class=""><br class=""></div><div class=""> Duration: 41424 seconds</div><div class=""> Data Read: 126333064115 bytes</div><div class="">Data Written: 418096774044 bytes</div><div class=""><br class=""></div><div class="">Interval 0 Stats:</div><div class=""> Block Size: 256b+ 512b+ 1024b+</div><div class=""> No. of Reads: 3706 247 142</div><div class="">No. of Writes: 15 3600 124</div><div class=""><br class=""></div><div class=""> Block Size: 2048b+ 4096b+ 8192b+</div><div class=""> No. of Reads: 94 5848 956</div><div class="">No. of Writes: 194 60615 19772</div><div class=""><br class=""></div><div class=""> Block Size: 16384b+ 32768b+ 65536b+</div><div class=""> No. of Reads: 1287 12947 1248</div><div class="">No. of Writes: 13126 8886 3810</div><div class=""><br class=""></div><div class=""> Block Size: 131072b+ 262144b+</div><div class=""> No. of Reads: 959349 0</div><div class="">No. of Writes: 3178716 5</div><div class=""> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop</div><div class=""> --------- ----------- ----------- ----------- ------------ ----</div><div class=""> 0.00 0.00 us 0.00 us 0.00 us 13 FORGET</div><div class=""> 0.00 0.00 us 0.00 us 0.00 us 3509 RELEASE</div><div class=""> 0.00 0.00 us 0.00 us 0.00 us 2425 RELEASEDIR</div><div class=""> 0.00 166.00 us 97.00 us 301.00 us 3 SETXATTR</div><div class=""> 0.00 42.62 us 19.00 us 123.00 us 90 ENTRYLK</div><div class=""> 0.00 431.00 us 80.00 us 8036.00 us 37 FTRUNCATE</div><div class=""> 0.00 160.27 us 17.00 us 63213.00 us 1041 LK</div><div class=""> 0.00 85684.71 us 166.00 us 598415.00 us 7 TRUNCATE</div><div class=""> 0.00 74683.31 us 143.00 us 2544472.00 us 35 XATTROP</div><div class=""> 0.00 22002.58 us 19.00 us 11281413.00 us 520 READDIR</div><div class=""> 0.01 6248116.73 us 1217.00 us 52589867.00 us 15 RENAME</div><div class=""> 0.01 6913389.40 us 332.00 us 34408990.00 us 15 CREATE</div><div class=""> 0.06 294044.50 us 25.00 us 100581145.00 us 1878 READDIRP</div><div class=""> 0.08 2357246.40 us 71.00 us 159200810.00 us 306 REMOVEXATTR</div><div class=""> 0.13 83641.44 us 20.00 us 69913926.00 us 13691 STATFS</div><div class=""> 0.14 1284416.00 us 15.00 us 173040222.00 us 964 GETXATTR</div><div class=""> 0.18 3720010.69 us 14.00 us 372969001.00 us 427 FLUSH</div><div class=""> 0.18 456678.40 us 53.00 us 130997224.00 us 3508 OPEN</div><div class=""> 0.34 1255418.93 us 1.00 us 267727475.00 us 2425 OPENDIR</div><div class=""> 0.64 16601795.78 us 73.00 us 392869613.00 us 347 SETATTR</div><div class=""> 0.74 235509.69 us 19.00 us 384250227.00 us 28134 FSTAT</div><div class=""> 1.10 1204610.00 us 16.00 us 407840076.00 us 8208 STAT</div><div class=""> 2.41 22009.63 us 29.00 us 421083048.00 us 985824 READ</div><div class=""> 2.52 93906.03 us 13.00 us 118521408.00 us 241629 FINODELK</div><div class=""> 3.77 4111.77 us 12.00 us 419936158.00 us 8251369 INODELK</div><div class=""> 3.84 218446.23 us 83.00 us 498122410.00 us 158007 FXATTROP</div><div class=""> 4.02 817497.87 us 19.00 us 409893532.00 us 44246 LOOKUP</div><div class=""> 25.07 68548.46 us 49.00 us 516930780.00 us 3288855 WRITE</div><div class=""> 54.76 4369255.44 us 51.00 us 688577405.00 us 112686 FSYNC</div><div class=""><br class=""></div><div class=""> Duration: 41424 seconds</div><div class=""> Data Read: 126333064115 bytes</div><div class="">Data Written: 418096774044 bytes</div></div><div class=""><br class=""></div><div class="">-----------</div><div class="">[this command took a few MINUTES to return]</div><div class=""><br class=""></div><div class=""><div class="">[root@ovirt1 ~]# gluster volume heal data-hdd info</div><div class="">Brick 172.172.1.11:/gluster/brick3/data-hdd</div><div class="">/cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/647be733-f153-4cdc-85bd-ba72544c2631/b453a300-0602-4be1-8310-8bd5abe00971</div><div class="">Status: Connected</div><div class="">Number of entries: 1</div><div class=""><br class=""></div><div class="">Brick 172.172.1.12:/gluster/brick3/data-hdd</div><div class="">Status: Connected</div><div class="">Number of entries: 0</div><div class=""><br class=""></div><div class="">Brick 172.172.1.13:/gluster/brick3/data-hdd</div><div class="">/cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/647be733-f153-4cdc-85bd-ba72544c2631/b453a300-0602-4be1-8310-8bd5abe00971</div><div class="">Status: Connected</div><div class="">Number of entries: 1</div></div><div class=""><br class=""></div><div class="">-----------</div><div class=""><div class=""> 09:43:02 up 16:09, 1 user, load average: 30.75, 39.10, 34.75</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div></div>
_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">https://lists.gluster.org/mailman/listinfo/gluster-users</div></blockquote></div><br class=""></div></body></html>