[Gluster-users] Gluster-induced load averages of 50-200
Jim Kusznir
jim at palousetech.com
Fri Jul 6 16:43:21 UTC 2018
Hello:
I have a hyperconverged ovirt cluster with 3 replica 2+arb and one replica
3 volume. My replica 3 volume has been having lots of problems for a while
now.
First, after a week or two, I notice a lot of RAM usage (half of my
physical ram, plus lots of swap) for the replica 3 gluster process. The
amount of ram in use varies by machine, but they all have had the problem
show up (one at a time). Often the OOM killer kills it off before I catch
it.
This last time, I caught it. I migrated all the machines off the problem
server, and rebooted it. That was yesterday (thurs at 9am). Shortly after
coming back up, ALL 3 of my servers are showing very high load averages
(cpu % as reported by top is fairly low, althoguh iowait will spike to
25%). The one process that is using the most CPU is the glusterfsd process
for the replica 3 volume. It is consistant on all 3 machines, and it loads
down the machine and IO enough to be causing issues with the VMs running on
it (even IO to a seprate disk on the same machine) .
I watched network I/O and I can see occasional spikes to 50MB/s (gig link),
but its not consistant, and often spends considerable time near zero or in
kB/s range. the repica 3 has its own dedicated gig network.
As of last night I still had 4 unhealed items for the server I rebooted.
This AM, I'm down to 1, but its still there, and its still causing
considerable performance issues. On a server, if I stop that glusterfsd
process, then the loadavg returns to normal quickly.
HELP...what do I do? Lately it seems that all my outages and performace
issues have been glusterfsd taking down my servers that they are running
on. Is gluster ready for production, or did I misunderstand? My
reliability has been abismal on my vm cluster, and I still have 3 VMs that
won't boot anymore due to disk corruption caused by my last gluster outage
(which was the high RAM until it crashed on the server due to kernel OOM
killer).
Here's some info on the affected gluster volume:
Volume Name: data-hdd
Type: Replicate
Volume ID: d342a3ab-16f3-49f0-bbcf-f788be8ac5f1
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 172.172.1.11:/gluster/brick3/data-hdd
Brick2: 172.172.1.12:/gluster/brick3/data-hdd
Brick3: 172.172.1.13:/gluster/brick3/data-hdd
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
transport.address-family: inet
performance.readdir-ahead: on
geo-replication.indexing: on
geo-replication.ignore-pid-check: on
changelog.changelog: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
performance.low-prio-threads: 32
network.remote-dio: enable
cluster.eager-lock: off
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
features.shard: on
user.cifs: off
storage.owner-uid: 36
storage.owner-gid: 36
server.allow-insecure: on
network.ping-timeout: 30
---------------
[root at ovirt3 ~]# gluster volume profile data-hdd info
Brick: 172.172.1.13:/gluster/brick3/data-hdd
--------------------------------------------
Cumulative Stats:
Block Size: 256b+ 512b+
1024b+
No. of Reads: 188488 6126
17234
No. of Writes: 80 331655
3899
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 17228 1371069
265013
No. of Writes: 6365 26617689
15776028
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 350572 636120
189538
No. of Writes: 4475250 1557330
604455
Block Size: 131072b+ 262144b+
No. of Reads: 6442414 0
No. of Writes: 5194964 13
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 63
FORGET
0.00 0.00 us 0.00 us 0.00 us 9356
RELEASE
0.00 0.00 us 0.00 us 0.00 us 113830
RELEASEDIR
0.00 765.33 us 102.00 us 2091.00 us 3
FGETXATTR
0.00 352.90 us 263.00 us 1397.00 us 31
TRUNCATE
0.00 1094.05 us 360.00 us 5643.00 us 21
MKDIR
0.00 647.01 us 260.00 us 4059.00 us 83
MKNOD
0.00 899.64 us 68.00 us 27531.00 us 105
SETXATTR
0.00 139.19 us 26.00 us 4619.00 us 740
ENTRYLK
0.00 2121.54 us 1250.00 us 9726.00 us 63
RENAME
0.00 254.93 us 18.00 us 169316.00 us 932
FLUSH
0.00 79.37 us 14.00 us 3120.00 us 3210
LK
0.00 9377.76 us 374.00 us 197032.00 us 63
CREATE
0.00 4777.26 us 91.00 us 757927.00 us 197
FTRUNCATE
0.00 1787.09 us 117.00 us 505178.00 us 767
XATTROP
0.00 180.43 us 13.00 us 796920.00 us 33471
GETXATTR
0.01 398.80 us 72.00 us 2581011.00 us 18792
REMOVEXATTR
0.01 378.54 us 21.00 us 264034.00 us 32293
READDIR
0.01 152.76 us 1.00 us 952682.00 us 113830
OPENDIR
0.02 1460.88 us 64.00 us 4693687.00 us 18918
SETATTR
0.02 524.47 us 30.00 us 359603.00 us 56913
READDIRP
0.02 3564.23 us 59.00 us 4818020.00 us 9307
OPEN
0.04 125.61 us 21.00 us 355789.00 us 400177
STAT
0.11 132.13 us 24.00 us 460615.00 us 1143725
STATFS
0.26 111.57 us 18.00 us 3533816.00 us 3153747
FSTAT
0.81 383.95 us 22.00 us 3896810.00 us 2827323
LOOKUP
5.51 74.04 us 13.00 us 1883274.00 us 100195900
FINODELK
5.59 258.57 us 13.00 us 11221929.00 us 29091404
INODELK
6.50 922.80 us 25.00 us 7548801.00 us 9483802
READ
10.20 358.55 us 56.00 us 14993728.00 us 38295759
FXATTROP
12.42 306.31 us 47.00 us 30915095.00 us 54567728
WRITE
58.45 2495.18 us 43.00 us 21663381.00 us 31520936
FSYNC
Duration: 1946380 seconds
Data Read: 894661069137 bytes
Data Written: 1142499928350 bytes
Interval 2 Stats:
Block Size: 256b+ 512b+
1024b+
No. of Reads: 0 3
0
No. of Writes: 15 3702
126
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 0 9
58
No. of Writes: 198 62716
20361
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 13 1
1
No. of Writes: 13640 9197
3948
Block Size: 131072b+ 262144b+
No. of Reads: 1826585 0
No. of Writes: 7387 5
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 15
FORGET
0.00 0.00 us 0.00 us 0.00 us 3648
RELEASE
0.00 0.00 us 0.00 us 0.00 us 2421
RELEASEDIR
0.00 293.86 us 263.00 us 327.00 us 7
TRUNCATE
0.00 72.55 us 30.00 us 152.00 us 90
ENTRYLK
0.00 546.00 us 374.00 us 788.00 us 15
CREATE
0.00 475.86 us 178.00 us 2572.00 us 51
XATTROP
0.00 75.50 us 24.00 us 301.00 us 424
FLUSH
0.00 2889.13 us 1557.00 us 9171.00 us 15
RENAME
0.00 195.00 us 109.00 us 866.00 us 313
REMOVEXATTR
0.00 225.66 us 117.00 us 858.00 us 341
SETATTR
0.00 80.26 us 25.00 us 534.00 us 1041
LK
0.00 284.35 us 24.00 us 3965.00 us 408
READDIR
0.01 134.66 us 2.00 us 17518.00 us 2421
OPENDIR
0.01 202.33 us 34.00 us 15815.00 us 2413
FSTAT
0.01 624.48 us 23.00 us 279600.00 us 820
GETXATTR
0.04 107.65 us 29.00 us 1004.00 us 14106
STATFS
0.45 73.59 us 15.00 us 60009.00 us 250150
FINODELK
0.45 5080.05 us 62.00 us 1940405.00 us 3631
OPEN
0.97 869.22 us 35.00 us 1942304.00 us 45625
LOOKUP
1.50 378.77 us 58.00 us 1833454.00 us 163094
FXATTROP
1.77 599.60 us 55.00 us 1933975.00 us 121295
WRITE
15.58 5494.71 us 50.00 us 1958942.00 us 116418
FSYNC
29.74 668.39 us 52.00 us 1945979.00 us 1826670
READ
49.46 243.86 us 13.00 us 1945794.00 us 8325886
INODELK
Duration: 42487 seconds
Data Read: 239414982144 bytes
Data Written: 2475540892 bytes
Brick: 172.172.1.11:/gluster/brick3/data-hdd
--------------------------------------------
Cumulative Stats:
Block Size: 256b+ 512b+
1024b+
No. of Reads: 7915 524
274
No. of Writes: 31 9934
169
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 351 36927
5673
No. of Writes: 278 621338
316601
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 6003 2366
3082
No. of Writes: 110118 45664
16295
Block Size: 131072b+ 262144b+
No. of Reads: 3710107 0
No. of Writes: 556659 6
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 29
FORGET
0.00 0.00 us 0.00 us 0.00 us 6295
RELEASE
0.00 0.00 us 0.00 us 0.00 us 5574
RELEASEDIR
0.00 164.00 us 164.00 us 164.00 us 1
SETXATTR
0.00 92.33 us 66.00 us 130.00 us 3
FGETXATTR
0.00 308.93 us 215.00 us 408.00 us 15
TRUNCATE
0.00 70.94 us 22.00 us 271.00 us 186
ENTRYLK
0.00 70.06 us 21.00 us 265.00 us 669
FLUSH
0.00 1974.90 us 1490.00 us 6096.00 us 31
RENAME
0.00 76.58 us 24.00 us 3869.00 us 2582
LK
0.00 3252.80 us 175.00 us 244398.00 us 106
XATTROP
0.00 377.53 us 22.00 us 33026.00 us 1292
READDIR
0.00 940.44 us 85.00 us 563301.00 us 755
REMOVEXATTR
0.00 134434.50 us 119.00 us 563308.00 us 6
FTRUNCATE
0.00 677.32 us 19.00 us 641166.00 us 2303
GETXATTR
0.00 296.94 us 2.00 us 605194.00 us 5575
OPENDIR
0.00 3015.49 us 112.00 us 2198121.00 us 793
SETATTR
0.00 144086.32 us 461.00 us 1973731.00 us 31
CREATE
0.00 122.50 us 21.00 us 470389.00 us 37921
STATFS
0.00 322.28 us 25.00 us 2826178.00 us 17238
STAT
0.01 151.48 us 22.00 us 460622.00 us 77757
FSTAT
0.01 4641.92 us 35.00 us 7009636.00 us 3665
READDIRP
0.01 3514.85 us 61.00 us 9109607.00 us 6278
OPEN
0.04 800.50 us 23.00 us 8211795.00 us 109977
LOOKUP
0.60 1279.23 us 59.00 us 237262387.00 us 1023399
FXATTROP
1.50 872.91 us 35.00 us 6929426.00 us 3773222
READ
1.61 2096.57 us 50.00 us 237261777.00 us 1677093
WRITE
6.17 16780.27 us 51.00 us 22990372.00 us 804858
FSYNC
11.35 1890.83 us 14.00 us 1066342526.00 us 13144090
INODELK
78.70 73624.16 us 15.00 us 759064386.00 us 2340006
FINODELK
Duration: 84992 seconds
Data Read: 486964889723 bytes
Data Written: 84167404788 bytes
Interval 2 Stats:
Block Size: 256b+ 512b+
1024b+
No. of Reads: 3861 337
173
No. of Writes: 15 3702
126
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 159 5308
857
No. of Writes: 198 62716
20361
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 1008 1004
621
No. of Writes: 13640 9197
3948
Block Size: 131072b+ 262144b+
No. of Reads: 2341344 0
No. of Writes: 7387 5
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 15
FORGET
0.00 0.00 us 0.00 us 0.00 us 3647
RELEASE
0.00 0.00 us 0.00 us 0.00 us 2541
RELEASEDIR
0.00 267.86 us 215.00 us 322.00 us 7
TRUNCATE
0.00 66.18 us 22.00 us 144.00 us 90
ENTRYLK
0.00 450.02 us 197.00 us 2730.00 us 51
XATTROP
0.00 66.45 us 21.00 us 265.00 us 423
FLUSH
0.00 2049.80 us 1504.00 us 6096.00 us 15
RENAME
0.00 193.90 us 85.00 us 1329.00 us 313
REMOVEXATTR
0.00 74.23 us 24.00 us 3869.00 us 1041
LK
0.00 258.23 us 112.00 us 12048.00 us 341
SETATTR
0.00 384.60 us 22.00 us 2284.00 us 647
READDIR
0.00 415.01 us 19.00 us 122660.00 us 1306
GETXATTR
0.00 495.05 us 2.00 us 605194.00 us 2542
OPENDIR
0.00 98.96 us 30.00 us 2284.00 us 14106
STATFS
0.00 194.92 us 25.00 us 628797.00 us 8675
STAT
0.00 849.86 us 35.00 us 369113.00 us 2005
READDIRP
0.00 585.27 us 61.00 us 279889.00 us 3630
OPEN
0.00 141763.87 us 461.00 us 1973731.00 us 15
CREATE
0.00 155.27 us 23.00 us 169047.00 us 21344
FSTAT
0.02 442.07 us 26.00 us 544143.00 us 45744
LOOKUP
0.11 641.69 us 59.00 us 2227486.00 us 163094
FXATTROP
0.16 1215.83 us 58.00 us 2628020.00 us 121295
WRITE
1.25 9910.79 us 51.00 us 5116269.00 us 116418
FSYNC
1.30 511.11 us 35.00 us 2493646.00 us 2354672
READ
13.86 1535.67 us 14.00 us 688678811.00 us 8321169
INODELK
83.28 309339.72 us 16.00 us 688564320.00 us 248271
FINODELK
Duration: 42487 seconds
Data Read: 307016229315 bytes
Data Written: 2475540892 bytes
Brick: 172.172.1.12:/gluster/brick3/data-hdd
--------------------------------------------
Cumulative Stats:
Block Size: 256b+ 512b+
1024b+
No. of Reads: 3706 247
142
No. of Writes: 15 3600
124
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 94 5848
956
No. of Writes: 194 60615
19772
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 1287 12947
1248
No. of Writes: 13126 8886
3810
Block Size: 131072b+ 262144b+
No. of Reads: 959349 0
No. of Writes: 3178716 5
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 13
FORGET
0.00 0.00 us 0.00 us 0.00 us 3509
RELEASE
0.00 0.00 us 0.00 us 0.00 us 2425
RELEASEDIR
0.00 166.00 us 97.00 us 301.00 us 3
SETXATTR
0.00 42.62 us 19.00 us 123.00 us 90
ENTRYLK
0.00 431.00 us 80.00 us 8036.00 us 37
FTRUNCATE
0.00 160.27 us 17.00 us 63213.00 us 1041
LK
0.00 85684.71 us 166.00 us 598415.00 us 7
TRUNCATE
0.00 74683.31 us 143.00 us 2544472.00 us 35
XATTROP
0.00 22002.58 us 19.00 us 11281413.00 us 520
READDIR
0.01 6248116.73 us 1217.00 us 52589867.00 us 15
RENAME
0.01 6913389.40 us 332.00 us 34408990.00 us 15
CREATE
0.06 294044.50 us 25.00 us 100581145.00 us 1878
READDIRP
0.08 2357246.40 us 71.00 us 159200810.00 us 306
REMOVEXATTR
0.13 83641.44 us 20.00 us 69913926.00 us 13691
STATFS
0.14 1284416.00 us 15.00 us 173040222.00 us 964
GETXATTR
0.18 3713522.62 us 14.00 us 372969001.00 us 427
FLUSH
0.18 456678.40 us 53.00 us 130997224.00 us 3508
OPEN
0.34 1255418.93 us 1.00 us 267727475.00 us 2425
OPENDIR
0.64 16601795.78 us 73.00 us 392869613.00 us 347
SETATTR
0.74 235518.84 us 19.00 us 384250227.00 us 28134
FSTAT
1.10 1203934.76 us 16.00 us 407840076.00 us 8208
STAT
2.41 22009.35 us 29.00 us 421083048.00 us 985824
READ
2.53 93906.00 us 13.00 us 118521408.00 us 241629
FINODELK
3.78 4111.77 us 12.00 us 419936158.00 us 8251369
INODELK
3.84 218447.60 us 83.00 us 498122410.00 us 158007
FXATTROP
4.02 817162.39 us 19.00 us 409893532.00 us 44246
LOOKUP
25.08 68523.67 us 49.00 us 516930780.00 us 3288855
WRITE
54.73 4364231.40 us 51.00 us 688577405.00 us 112686
FSYNC
Duration: 41424 seconds
Data Read: 126333064115 bytes
Data Written: 418096774044 bytes
Interval 0 Stats:
Block Size: 256b+ 512b+
1024b+
No. of Reads: 3706 247
142
No. of Writes: 15 3600
124
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 94 5848
956
No. of Writes: 194 60615
19772
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 1287 12947
1248
No. of Writes: 13126 8886
3810
Block Size: 131072b+ 262144b+
No. of Reads: 959349 0
No. of Writes: 3178716 5
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 13
FORGET
0.00 0.00 us 0.00 us 0.00 us 3509
RELEASE
0.00 0.00 us 0.00 us 0.00 us 2425
RELEASEDIR
0.00 166.00 us 97.00 us 301.00 us 3
SETXATTR
0.00 42.62 us 19.00 us 123.00 us 90
ENTRYLK
0.00 431.00 us 80.00 us 8036.00 us 37
FTRUNCATE
0.00 160.27 us 17.00 us 63213.00 us 1041
LK
0.00 85684.71 us 166.00 us 598415.00 us 7
TRUNCATE
0.00 74683.31 us 143.00 us 2544472.00 us 35
XATTROP
0.00 22002.58 us 19.00 us 11281413.00 us 520
READDIR
0.01 6248116.73 us 1217.00 us 52589867.00 us 15
RENAME
0.01 6913389.40 us 332.00 us 34408990.00 us 15
CREATE
0.06 294044.50 us 25.00 us 100581145.00 us 1878
READDIRP
0.08 2357246.40 us 71.00 us 159200810.00 us 306
REMOVEXATTR
0.13 83641.44 us 20.00 us 69913926.00 us 13691
STATFS
0.14 1284416.00 us 15.00 us 173040222.00 us 964
GETXATTR
0.18 3720010.69 us 14.00 us 372969001.00 us 427
FLUSH
0.18 456678.40 us 53.00 us 130997224.00 us 3508
OPEN
0.34 1255418.93 us 1.00 us 267727475.00 us 2425
OPENDIR
0.64 16601795.78 us 73.00 us 392869613.00 us 347
SETATTR
0.74 235509.69 us 19.00 us 384250227.00 us 28134
FSTAT
1.10 1204610.00 us 16.00 us 407840076.00 us 8208
STAT
2.41 22009.63 us 29.00 us 421083048.00 us 985824
READ
2.52 93906.03 us 13.00 us 118521408.00 us 241629
FINODELK
3.77 4111.77 us 12.00 us 419936158.00 us 8251369
INODELK
3.84 218446.23 us 83.00 us 498122410.00 us 158007
FXATTROP
4.02 817497.87 us 19.00 us 409893532.00 us 44246
LOOKUP
25.07 68548.46 us 49.00 us 516930780.00 us 3288855
WRITE
54.76 4369255.44 us 51.00 us 688577405.00 us 112686
FSYNC
Duration: 41424 seconds
Data Read: 126333064115 bytes
Data Written: 418096774044 bytes
-----------
[this command took a few MINUTES to return]
[root at ovirt1 ~]# gluster volume heal data-hdd info
Brick 172.172.1.11:/gluster/brick3/data-hdd
/cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/647be733-f153-4cdc-85bd-ba72544c2631/b453a300-0602-4be1-8310-8bd5abe00971
Status: Connected
Number of entries: 1
Brick 172.172.1.12:/gluster/brick3/data-hdd
Status: Connected
Number of entries: 0
Brick 172.172.1.13:/gluster/brick3/data-hdd
/cc65f671-3377-494a-a7d4-1d9f7c3ae46c/images/647be733-f153-4cdc-85bd-ba72544c2631/b453a300-0602-4be1-8310-8bd5abe00971
Status: Connected
Number of entries: 1
-----------
09:43:02 up 16:09, 1 user, load average: 30.75, 39.10, 34.75
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180706/1654089b/attachment.html>
More information about the Gluster-users
mailing list