[Gluster-users] Gluster-induced load averages of 50-200
Darrell Budic
budic at onholyground.com
Fri Jul 6 21:32:18 UTC 2018
I’ve had issues like that while playing with different healing config, too many shd threads will drive load higher than it needs to be and keep the boxes busy. I’ve had better luck with
cluster.shd-max-threads: 1
cluster.shd-wait-qlength: 1024
than 8 & 10000. It does heal a little faster with 8/10k than 1/1k, but it keeps the servers too busy to do anything else. In a hyper converged install, I could see it causing problems for the VMs and keeping healing from effectively happening (it would cause disk read issues for me even with a separate server cluster).
Also, what version are you running? If it’s 3.12, I’m finding that .9-.11 have a client side memory leak, just havn’t gotten everything together enough to post about it. .8 seems ok, and .6 is definitely ok though, flat memory use from both, steadily increasing from .9 and .11 clients.
Good luck!
-Darrell
> From: Jim Kusznir <jim at palousetech.com>
> Subject: [Gluster-users] Gluster-induced load averages of 50-200
> Date: July 6, 2018 at 11:43:21 AM CDT
> To: gluster-user
>
> Hello:
>
> I have a hyperconverged ovirt cluster with 3 replica 2+arb and one replica 3 volume. My replica 3 volume has been having lots of problems for a while now.
>
> First, after a week or two, I notice a lot of RAM usage (half of my physical ram, plus lots of swap) for the replica 3 gluster process. The amount of ram in use varies by machine, but they all have had the problem show up (one at a time). Often the OOM killer kills it off before I catch it.
>
> This last time, I caught it. I migrated all the machines off the problem server, and rebooted it. That was yesterday (thurs at 9am). Shortly after coming back up, ALL 3 of my servers are showing very high load averages (cpu % as reported by top is fairly low, althoguh iowait will spike to 25%). The one process that is using the most CPU is the glusterfsd process for the replica 3 volume. It is consistant on all 3 machines, and it loads down the machine and IO enough to be causing issues with the VMs running on it (even IO to a seprate disk on the same machine) .
>
> I watched network I/O and I can see occasional spikes to 50MB/s (gig link), but its not consistant, and often spends considerable time near zero or in kB/s range. the repica 3 has its own dedicated gig network.
>
> As of last night I still had 4 unhealed items for the server I rebooted. This AM, I'm down to 1, but its still there, and its still causing considerable performance issues. On a server, if I stop that glusterfsd process, then the loadavg returns to normal quickly.
>
> HELP...what do I do? Lately it seems that all my outages and performace issues have been glusterfsd taking down my servers that they are running on. Is gluster ready for production, or did I misunderstand? My reliability has been abismal on my vm cluster, and I still have 3 VMs that won't boot anymore due to disk corruption caused by my last gluster outage (which was the high RAM until it crashed on the server due to kernel OOM killer).
>
> Here's some info on the affected gluster volume:
> Volume Name: data-hdd
> Type: Replicate
> Volume ID: d342a3ab-16f3-49f0-bbcf-f788be8ac5f1
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: 172.172.1.11:/gluster/brick3/data-hdd
> Brick2: 172.172.1.12:/gluster/brick3/data-hdd
> Brick3: 172.172.1.13:/gluster/brick3/data-hdd
> Options Reconfigured:
> diagnostics.count-fop-hits: on
> diagnostics.latency-measurement: on
> transport.address-family: inet
> performance.readdir-ahead: on
> geo-replication.indexing: on
> geo-replication.ignore-pid-check: on
> changelog.changelog: on
> performance.quick-read: off
> performance.read-ahead: off
> performance.io-cache: off
> performance.stat-prefetch: off
> performance.low-prio-threads: 32
> network.remote-dio: enable
> cluster.eager-lock: off
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> cluster.data-self-heal-algorithm: full
> cluster.locking-scheme: granular
> cluster.shd-max-threads: 8
> cluster.shd-wait-qlength: 10000
> features.shard: on
> user.cifs: off
> storage.owner-uid: 36
> storage.owner-gid: 36
> server.allow-insecure: on
> network.ping-timeout: 30
>
> ---------------
>
> [root at ovirt3 ~]# gluster volume profile data-hdd info
> Brick: 172.172.1.13:/gluster/brick3/data-hdd
> --------------------------------------------
> Cumulative Stats:
> Block Size: 256b+ 512b+ 1024b+
> No. of Reads: 188488 6126 17234
> No. of Writes: 80 331655 3899
>
> Block Size: 2048b+ 4096b+ 8192b+
> No. of Reads: 17228 1371069 265013
> No. of Writes: 6365 26617689 15776028
>
> Block Size: 16384b+ 32768b+ 65536b+
> No. of Reads: 350572 636120 189538
> No. of Writes: 4475250 1557330 604455
>
> Block Size: 131072b+ 262144b+
> No. of Reads: 6442414 0
> No. of Writes: 5194964 13
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 63 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 9356 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 113830 RELEASEDIR
> 0.00 765.33 us 102.00 us 2091.00 us 3 FGETXATTR
> 0.00 352.90 us 263.00 us 1397.00 us 31 TRUNCATE
> 0.00 1094.05 us 360.00 us 5643.00 us 21 MKDIR
> 0.00 647.01 us 260.00 us 4059.00 us 83 MKNOD
> 0.00 899.64 us 68.00 us 27531.00 us 105 SETXATTR
> 0.00 139.19 us 26.00 us 4619.00 us 740 ENTRYLK
> 0.00 2121.54 us 1250.00 us 9726.00 us 63 RENAME
> 0.00 254.93 us 18.00 us 169316.00 us 932 FLUSH
> 0.00 79.37 us 14.00 us 3120.00 us 3210 LK
> 0.00 9377.76 us 374.00 us 197032.00 us 63 CREATE
> 0.00 4777.26 us 91.00 us 757927.00 us 197 FTRUNCATE
> 0.00 1787.09 us 117.00 us 505178.00 us 767 XATTROP
> 0.00 180.43 us 13.00 us 796920.00 us 33471 GETXATTR
> 0.01 398.80 us 72.00 us 2581011.00 us 18792 REMOVEXATTR
> 0.01 378.54 us 21.00 us 264034.00 us 32293 READDIR
> 0.01 152.76 us 1.00 us 952682.00 us 113830 OPENDIR
> 0.02 1460.88 us 64.00 us 4693687.00 us 18918 SETATTR
> 0.02 524.47 us 30.00 us 359603.00 us 56913 READDIRP
> 0.02 3564.23 us 59.00 us 4818020.00 us 9307 OPEN
> 0.04 125.61 us 21.00 us 355789.00 us 400177 STAT
> 0.11 132.13 us 24.00 us 460615.00 us 1143725 STATFS
> 0.26 111.57 us 18.00 us 3533816.00 us 3153747 FSTAT
> 0.81 383.95 us 22.00 us 3896810.00 us 2827323 LOOKUP
> 5.51 74.04 us 13.00 us 1883274.00 us 100195900 FINODELK
> 5.59 258.57 us 13.00 us 11221929.00 us 29091404 INODELK
> 6.50 922.80 us 25.00 us 7548801.00 us 9483802 READ
> 10.20 358.55 us 56.00 us 14993728.00 us 38295759 FXATTROP
> 12.42 306.31 us 47.00 us 30915095.00 us 54567728 WRITE
> 58.45 2495.18 us 43.00 us 21663381.00 us 31520936 FSYNC
>
> Duration: 1946380 seconds
> Data Read: 894661069137 bytes
> Data Written: 1142499928350 bytes
>
> Interval 2 Stats:
> Block Size: 256b+ 512b+ 1024b+
> No. of Reads: 0 3 0
> No. of Writes: 15 3702 126
>
> Block Size: 2048b+ 4096b+ 8192b+
> No. of Reads: 0 9 58
> No. of Writes: 198 62716 20361
>
> Block Size: 16384b+ 32768b+ 65536b+
> No. of Reads: 13 1 1
> No. of Writes: 13640 9197 3948
>
> Block Size: 131072b+ 262144b+
> No. of Reads: 1826585 0
> No. of Writes: 7387 5
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 15 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 3648 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 2421 RELEASEDIR
> 0.00 293.86 us 263.00 us 327.00 us 7 TRUNCATE
> 0.00 72.55 us 30.00 us 152.00 us 90 ENTRYLK
> 0.00 546.00 us 374.00 us 788.00 us 15 CREATE
> 0.00 475.86 us 178.00 us 2572.00 us 51 XATTROP
> 0.00 75.50 us 24.00 us 301.00 us 424 FLUSH
> 0.00 2889.13 us 1557.00 us 9171.00 us 15 RENAME
> 0.00 195.00 us 109.00 us 866.00 us 313 REMOVEXATTR
> 0.00 225.66 us 117.00 us 858.00 us 341 SETATTR
> 0.00 80.26 us 25.00 us 534.00 us 1041 LK
> 0.00 284.35 us 24.00 us 3965.00 us 408 READDIR
> 0.01 134.66 us 2.00 us 17518.00 us 2421 OPENDIR
> 0.01 202.33 us 34.00 us 15815.00 us 2413 FSTAT
> 0.01 624.48 us 23.00 us 279600.00 us 820 GETXATTR
> 0.04 107.65 us 29.00 us 1004.00 us 14106 STATFS
> 0.45 73.59 us 15.00 us 60009.00 us 250150 FINODELK
> 0.45 5080.05 us 62.00 us 1940405.00 us 3631 OPEN
> 0.97 869.22 us 35.00 us 1942304.00 us 45625 LOOKUP
> 1.50 378.77 us 58.00 us 1833454.00 us 163094 FXATTROP
> 1.77 599.60 us 55.00 us 1933975.00 us 121295 WRITE
> 15.58 5494.71 us 50.00 us 1958942.00 us 116418 FSYNC
> 29.74 668.39 us 52.00 us 1945979.00 us 1826670 READ
> 49.46 243.86 us 13.00 us 1945794.00 us 8325886 INODELK
>
> Duration: 42487 seconds
> Data Read: 239414982144 bytes
> Data Written: 2475540892 bytes
>
> Brick: 172.172.1.11:/gluster/brick3/data-hdd
> --------------------------------------------
> Cumulative Stats:
> Block Size: 256b+ 512b+ 1024b+
> No. of Reads: 7915 524 274
> No. of Writes: 31 9934 169
>
> Block Size: 2048b+ 4096b+ 8192b+
> No. of Reads: 351 36927 5673
> No. of Writes: 278 621338 316601
>
> Block Size: 16384b+ 32768b+ 65536b+
> No. of Reads: 6003 2366 3082
> No. of Writes: 110118 45664 16295
>
> Block Size: 131072b+ 262144b+
> No. of Reads: 3710107 0
> No. of Writes: 556659 6
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 29 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 6295 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 5574 RELEASEDIR
> 0.00 164.00 us 164.00 us 164.00 us 1 SETXATTR
> 0.00 92.33 us 66.00 us 130.00 us 3 FGETXATTR
> 0.00 308.93 us 215.00 us 408.00 us 15 TRUNCATE
> 0.00 70.94 us 22.00 us 271.00 us 186 ENTRYLK
> 0.00 70.06 us 21.00 us 265.00 us 669 FLUSH
> 0.00 1974.90 us 1490.00 us 6096.00 us 31 RENAME
> 0.00 76.58 us 24.00 us 3869.00 us 2582 LK
> 0.00 3252.80 us 175.00 us 244398.00 us 106 XATTROP
> 0.00 377.53 us 22.00 us 33026.00 us 1292 READDIR
> 0.00 940.44 us 85.00 us 563301.00 us 755 REMOVEXATTR
> 0.00 134434.50 us 119.00 us 563308.00 us 6 FTRUNCATE
> 0.00 677.32 us 19.00 us 641166.00 us 2303 GETXATTR
> 0.00 296.94 us 2.00 us 605194.00 us 5575 OPENDIR
> 0.00 3015.49 us 112.00 us 2198121.00 us 793 SETATTR
> 0.00 144086.32 us 461.00 us 1973731.00 us 31 CREATE
> 0.00 122.50 us 21.00 us 470389.00 us 37921 STATFS
> 0.00 322.28 us 25.00 us 2826178.00 us 17238 STAT
> 0.01 151.48 us 22.00 us 460622.00 us 77757 FSTAT
> 0.01 4641.92 us 35.00 us 7009636.00 us 3665 READDIRP
> 0.01 3514.85 us 61.00 us 9109607.00 us 6278 OPEN
> 0.04 800.50 us 23.00 us 8211795.00 us 109977 LOOKUP
> 0.60 1279.23 us 59.00 us 237262387.00 us 1023399 FXATTROP
> 1.50 872.91 us 35.00 us 6929426.00 us 3773222 READ
> 1.61 2096.57 us 50.00 us 237261777.00 us 1677093 WRITE
> 6.17 16780.27 us 51.00 us 22990372.00 us 804858 FSYNC
> 11.35 1890.83 us 14.00 us 1066342526.00 us 13144090 INODELK
> 78.70 73624.16 us 15.00 us 759064386.00 us 2340006 FINODELK
>
> Duration: 84992 seconds
> Data Read: 486964889723 bytes
> Data Written: 84167404788 bytes
>
> Interval 2 Stats:
> Block Size: 256b+ 512b+ 1024b+
> No. of Reads: 3861 337 173
> No. of Writes: 15 3702 126
>
> Block Size: 2048b+ 4096b+ 8192b+
> No. of Reads: 159 5308 857
> No. of Writes: 198 62716 20361
>
> Block Size: 16384b+ 32768b+ 65536b+
> No. of Reads: 1008 1004 621
> No. of Writes: 13640 9197 3948
>
> Block Size: 131072b+ 262144b+
> No. of Reads: 2341344 0
> No. of Writes: 7387 5
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 15 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 3647 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 2541 RELEASEDIR
> 0.00 267.86 us 215.00 us 322.00 us 7 TRUNCATE
> 0.00 66.18 us 22.00 us 144.00 us 90 ENTRYLK
> 0.00 450.02 us 197.00 us 2730.00 us 51 XATTROP
> 0.00 66.45 us 21.00 us 265.00 us 423 FLUSH
> 0.00 2049.80 us 1504.00 us 6096.00 us 15 RENAME
> 0.00 193.90 us 85.00 us 1329.00 us 313 REMOVEXATTR
> 0.00 74.23 us 24.00 us 3869.00 us 1041 LK
> 0.00 258.23 us 112.00 us 12048.00 us 341 SETATTR
> 0.00 384.60 us 22.00 us 2284.00 us 647 READDIR
> 0.00 415.01 us 19.00 us 122660.00 us 1306 GETXATTR
> 0.00 495.05 us 2.00 us 605194.00 us 2542 OPENDIR
> 0.00 98.96 us 30.00 us 2284.00 us 14106 STATFS
> 0.00 194.92 us 25.00 us 628797.00 us 8675 STAT
> 0.00 849.86 us 35.00 us 369113.00 us 2005 READDIRP
> 0.00 585.27 us 61.00 us 279889.00 us 3630 OPEN
> 0.00 141763.87 us 461.00 us 1973731.00 us 15 CREATE
> 0.00 155.27 us 23.00 us 169047.00 us 21344 FSTAT
> 0.02 442.07 us 26.00 us 544143.00 us 45744 LOOKUP
> 0.11 641.69 us 59.00 us 2227486.00 us 163094 FXATTROP
> 0.16 1215.83 us 58.00 us 2628020.00 us 121295 WRITE
> 1.25 9910.79 us 51.00 us 5116269.00 us 116418 FSYNC
> 1.30 511.11 us 35.00 us 2493646.00 us 2354672 READ
> 13.86 1535.67 us 14.00 us 688678811.00 us 8321169 INODELK
> 83.28 309339.72 us 16.00 us 688564320.00 us 248271 FINODELK
>
> Duration: 42487 seconds
> Data Read: 307016229315 bytes
> Data Written: 2475540892 bytes
>
> Brick: 172.172.1.12:/gluster/brick3/data-hdd
> --------------------------------------------
> Cumulative Stats:
> Block Size: 256b+ 512b+ 1024b+
> No. of Reads: 3706 247 142
> No. of Writes: 15 3600 124
>
> Block Size: 2048b+ 4096b+ 8192b+
> No. of Reads: 94 5848 956
> No. of Writes: 194 60615 19772
>
> Block Size: 16384b+ 32768b+ 65536b+
> No. of Reads: 1287 12947 1248
> No. of Writes: 13126 8886 3810
>
> Block Size: 131072b+ 262144b+
> No. of Reads: 959349 0
> No. of Writes: 3178716 5
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 13 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 3509 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 2425 RELEASEDIR
> 0.00 166.00 us 97.00 us 301.00 us 3 SETXATTR
> 0.00 42.62 us 19.00 us 123.00 us 90 ENTRYLK
> 0.00 431.00 us 80.00 us 8036.00 us 37 FTRUNCATE
> 0.00 160.27 us 17.00 us 63213.00 us 1041 LK
> 0.00 85684.71 us 166.00 us 598415.00 us 7 TRUNCATE
> 0.00 74683.31 us 143.00 us 2544472.00 us 35 XATTROP
> 0.00 22002.58 us 19.00 us 11281413.00 us 520 READDIR
> 0.01 6248116.73 us 1217.00 us 52589867.00 us 15 RENAME
> 0.01 6913389.40 us 332.00 us 34408990.00 us 15 CREATE
> 0.06 294044.50 us 25.00 us 100581145.00 us 1878 READDIRP
> 0.08 2357246.40 us 71.00 us 159200810.00 us 306 REMOVEXATTR
> 0.13 83641.44 us 20.00 us 69913926.00 us 13691 STATFS
> 0.14 1284416.00 us 15.00 us 173040222.00 us 964 GETXATTR
> 0.18 3713522.62 us 14.00 us 372969001.00 us 427 FLUSH
> 0.18 456678.40 us 53.00 us 130997224.00 us 3508 OPEN
> 0.34 1255418.93 us 1.00 us 267727475.00 us 2425 OPENDIR
> 0.64 16601795.78 us 73.00 us 392869613.00 us 347 SETATTR
> 0.74 235518.84 us 19.00 us 384250227.00 us 28134 FSTAT
> 1.10 1203934.76 us 16.00 us 407840076.00 us 8208 STAT
> 2.41 22009.35 us 29.00 us 421083048.00 us 985824 READ
> 2.53 93906.00 us 13.00 us 118521408.00 us 241629 FINODELK
> 3.78 4111.77 us 12.00 us 419936158.00 us 8251369 INODELK
> 3.84 218447.60 us 83.00 us 498122410.00 us 158007 FXATTROP
> 4.02 817162.39 us 19.00 us 409893532.00 us 44246 LOOKUP
> 25.08 68523.67 us 49.00 us 516930780.00 us 3288855 WRITE
> 54.73 4364231.40 us 51.00 us 688577405.00 us 112686 FSYNC
>
> Duration: 41424 seconds
> Data Read: 126333064115 bytes
> Data Written: 418096774044 bytes
>
> Interval 0 Stats:
> Block Size: 256b+ 512b+ 1024b+
> No. of Reads: 3706 247 142
> No. of Writes: 15 3600 124
>
> Block Size: 2048b+ 4096b+ 8192b+
> No. of Reads: 94 5848 956
> No. of Writes: 194 60615 19772
>
> Block Size: 16384b+ 32768b+ 65536b+
> No. of Reads: 1287 12947 1248
> No. of Writes: 13126 8886 3810
>
> Block Size: 131072b+ 262144b+
> No. of Reads: 959349 0
> No. of Writes: 3178716 5
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 13 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 3509 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 2425 RELEASEDIR
> 0.00 166.00 us 97.00 us 301.00 us 3 SETXATTR
> 0.00 42.62 us 19.00 us 123.00 us 90 ENTRYLK
> 0.00 431.00 us 80.00 us 8036.00 us 37 FTRUNCATE
> 0.00 160.27 us 17.00 us 63213.00 us 1041 LK
> 0.00 85684.71 us 166.00 us 598415.00 us 7 TRUNCATE
> 0.00 74683.31 us 143.00 us 2544472.00 us 35 XATTROP
> 0.00 22002.58 us 19.00 us 11281413.00 us 520 READDIR
> 0.01 6248116.73 us 1217.00 us 52589867.00 us 15 RENAME
> 0.01 6913389.40 us 332.00 us 34408990.00 us 15 CREATE
> 0.06 294044.50 us 25.00 us 100581145.00 us 1878 READDIRP
> 0.08 2357246.40 us 71.00 us 159200810.00 us 306 REMOVEXATTR
> 0.13 83641.44 us 20.00 us 69913926.00 us 13691 STATFS
> 0.14 1284416.00 us 15.00 us 173040222.00 us 964 GETXATTR
> 0.18 3720010.69 us 14.00 us 372969001.00 us 427 FLUSH
> 0.18 456678.40 us 53.00 us 130997224.00 us 3508 OPEN
> 0.34 1255418.93 us 1.00 us 267727475.00 us 2425 OPENDIR
> 0.64 16601795.78 us 73.00 us 392869613.00 us 347 SETATTR
> 0.74 235509.69 us 19.00 us 384250227.00 us 28134 FSTAT
> 1.10 1204610.00 us 16.00 us 407840076.00 us 8208 STAT
> 2.41 22009.63 us 29.00 us 421083048.00 us 985824 READ
> 2.52 93906.03 us 13.00 us 118521408.00 us 241629 FINODELK
> 3.77 4111.77 us 12.00 us 419936158.00 us 8251369 INODELK
> 3.84 218446.23 us 83.00 us 498122410.00 us 158007 FXATTROP
> 4.02 817497.87 us 19.00 us 409893532.00 us 44246 LOOKUP
> 25.07 68548.46 us 49.00 us 516930780.00 us 3288855 WRITE
> 54.76 4369255.44 us 51.00 us 688577405.00 us 112686 FSYNC
>
> Duration: 41424 seconds
> Data Read: 126333064115 bytes
> Data Written: 418096774044 bytes
>
> -----------
> [this command took a few MINUTES to return]
>
> [root at ovirt1 ~]# gluster volume heal data-hdd info
> Brick 172.172.1.11:/gluster/brick3/data-hdd
> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/647be733-f153-4cdc-85bd-ba72544c2631/b453a300-0602-4be1-8310-8bd5abe00971
> Status: Connected
> Number of entries: 1
>
> Brick 172.172.1.12:/gluster/brick3/data-hdd
> Status: Connected
> Number of entries: 0
>
> Brick 172.172.1.13:/gluster/brick3/data-hdd
> /cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/647be733-f153-4cdc-85bd-ba72544c2631/b453a300-0602-4be1-8310-8bd5abe00971
> Status: Connected
> Number of entries: 1
>
> -----------
> 09:43:02 up 16:09, 1 user, load average: 30.75, 39.10, 34.75
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180706/37a13d50/attachment.html>
More information about the Gluster-users
mailing list