[Gluster-users] glusterfs and glusterfsd process utilization
Kyle Harris
kyle.harris98 at gmail.com
Mon Dec 22 02:57:09 UTC 2014
I haven't heard anything from anyone re this post yet so I have a bit more
info to add. The last time I posted about extremely high process
utilization for glusterfs and glusterfsd, I was asked to see if I could
capture the output of a profile. I did so and here is the output of 2
captures (they appear very similar). Perhaps this will assist. Note that
previously I was mounting with the fuse client. This time I tried NFS.
Profile Output 1:
Brick: 192.168.10.92:/export/brick1
-----------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+
4b+
No. of Reads: 0 0
0
No. of Writes: 133 6
37
Block Size: 8b+ 16b+
32b+
No. of Reads: 28 93
31
No. of Writes: 4576 6494
22228
Block Size: 64b+ 128b+
256b+
No. of Reads: 217 216
254
No. of Writes: 37627 41711
20519
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 484 978
684
No. of Writes: 39094 42589
203747
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 1073 1800
2838
No. of Writes: 220154 229725
215083
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 4224 14941
57812
No. of Writes: 60938 61967
793400
Block Size: 262144b+ 524288b+
1048576b+
No. of Reads: 4 4
2
No. of Writes: 5 0
0
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 1835064
FORGET
0.00 0.00 us 0.00 us 0.00 us 2639321
RELEASE
0.00 0.00 us 0.00 us 0.00 us 1439759
RELEASEDIR
0.00 153.50 us 142.00 us 165.00 us 2
XATTROP
0.00 102.67 us 97.00 us 110.00 us 3
SETATTR
0.00 93.20 us 53.00 us 129.00 us 5
GETXATTR
0.00 55.07 us 33.00 us 100.00 us 15
INODELK
0.00 46.61 us 35.00 us 69.00 us 18
STATFS
0.01 48.40 us 31.00 us 64.00 us 25
LK
0.01 50.72 us 32.00 us 88.00 us 47
FINODELK
0.01 35.49 us 17.00 us 68.00 us 86
FLUSH
0.02 50.59 us 27.00 us 76.00 us 68
ENTRYLK
0.03 75.72 us 47.00 us 112.00 us 74
OPEN
0.06 58.79 us 32.00 us 100.00 us 222
OPENDIR
0.08 1535.92 us 129.00 us 16583.00 us 12
UNLINK
0.11 24814.00 us 24814.00 us 24814.00 us 1
TRUNCATE
0.12 34.25 us 11.00 us 1950.00 us 784
ACCESS
0.13 603.50 us 90.00 us 21829.00 us 46
FXATTROP
0.16 7225.40 us 63.00 us 14563.00 us 5
READLINK
0.17 437.90 us 54.00 us 19042.00 us 83
READ
0.36 85.92 us 56.00 us 313.00 us 912
FSTAT
0.42 59.86 us 22.00 us 1443.00 us 1533
STAT
0.45 4318.65 us 70.00 us 56839.00 us 23
WRITE
0.84 8405.18 us 161.00 us 37688.00 us 22
CREATE
8.33 2005.77 us 361.00 us 2796.00 us 912
READDIRP
88.68 921.10 us 34.00 us 159393.00 us 21137
LOOKUP
Duration: 338151 seconds
Data Read: 9290348715 bytes
Data Written: 122127653133 bytes
Interval 1 Stats:
Block Size: 8b+ 32b+
64b+
No. of Reads: 2 0
1
No. of Writes: 0 1
4
Block Size: 128b+ 256b+
512b+
No. of Reads: 1 2
5
No. of Writes: 3 0
0
Block Size: 1024b+ 2048b+
4096b+
No. of Reads: 7 13
11
No. of Writes: 6 0
0
Block Size: 8192b+ 16384b+
32768b+
No. of Reads: 10 2
9
No. of Writes: 0 0
0
Block Size: 65536b+ 131072b+
262144b+
No. of Reads: 1 1
1
No. of Writes: 0 0
0
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 7
FORGET
0.00 0.00 us 0.00 us 0.00 us 45
RELEASE
0.00 0.00 us 0.00 us 0.00 us 177
RELEASEDIR
0.00 109.00 us 109.00 us 109.00 us 1
GETXATTR
0.00 110.00 us 110.00 us 110.00 us 1
SETATTR
0.00 52.78 us 33.00 us 81.00 us 9
INODELK
0.00 48.92 us 36.00 us 69.00 us 12
STATFS
0.01 146.80 us 129.00 us 171.00 us 5
UNLINK
0.01 47.89 us 31.00 us 63.00 us 18
LK
0.01 34.25 us 17.00 us 61.00 us 40
FLUSH
0.01 49.89 us 35.00 us 64.00 us 28
FINODELK
0.02 49.12 us 27.00 us 76.00 us 42
ENTRYLK
0.02 78.14 us 47.00 us 99.00 us 29
OPEN
0.08 58.01 us 34.00 us 100.00 us 177
OPENDIR
0.13 241.08 us 54.00 us 10214.00 us 66
READ
0.18 34.76 us 11.00 us 1950.00 us 649
ACCESS
0.21 906.82 us 94.00 us 21829.00 us 28
FXATTROP
0.29 9014.75 us 63.00 us 14563.00 us 4
READLINK
0.32 85.78 us 56.00 us 313.00 us 455
FSTAT
0.59 60.01 us 22.00 us 1443.00 us 1215
STAT
0.67 5906.21 us 70.00 us 56839.00 us 14
WRITE
1.02 7854.81 us 161.00 us 27303.00 us 16
CREATE
7.42 2008.99 us 361.00 us 2796.00 us 455
READDIRP
89.00 1052.36 us 37.00 us 132924.00 us 10418
LOOKUP
Duration: 18 seconds
Data Read: 1285736 bytes
Data Written: 8465 bytes
Brick: 192.168.10.90:/export/brick1
-----------------------------------
Cumulative Stats:
Block Size: 2b+ 8b+
16b+
No. of Reads: 0 6
32
No. of Writes: 1 1928
2604
Block Size: 32b+ 64b+
128b+
No. of Reads: 21 112
169
No. of Writes: 6983 12066
26008
Block Size: 256b+ 512b+
1024b+
No. of Reads: 85 299
809
No. of Writes: 743 18584
7209
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 790 1418
2884
No. of Writes: 1650 10647
16901
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 4588 5856
32428
No. of Writes: 2974 5758
3387
Block Size: 131072b+
No. of Reads: 455199
No. of Writes: 4601
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 604780
FORGET
0.00 0.00 us 0.00 us 0.00 us 1410794
RELEASE
0.00 0.00 us 0.00 us 0.00 us 989700
RELEASEDIR
0.00 45.50 us 29.00 us 62.00 us 2
READDIR
0.00 58.00 us 44.00 us 77.00 us 4
GETXATTR
0.00 92.00 us 86.00 us 101.00 us 3
SETATTR
0.00 147.50 us 136.00 us 159.00 us 2
XATTROP
0.00 55.89 us 41.00 us 84.00 us 9
FSTAT
0.00 54.22 us 33.00 us 76.00 us 18
STATFS
0.00 46.04 us 30.00 us 60.00 us 25
LK
0.01 172.17 us 115.00 us 222.00 us 12
UNLINK
0.01 38.75 us 15.00 us 150.00 us 56
FLUSH
0.01 54.46 us 30.00 us 101.00 us 46
FINODELK
0.01 52.78 us 27.00 us 91.00 us 68
ENTRYLK
0.01 93.74 us 49.00 us 352.00 us 43
OPEN
0.01 204.50 us 150.00 us 511.00 us 22
CREATE
0.03 58.25 us 3.00 us 94.00 us 183
OPENDIR
0.03 247.33 us 73.00 us 5551.00 us 45
FXATTROP
0.11 35786.00 us 35786.00 us 35786.00 us 1
TRUNCATE
0.15 669.42 us 41.00 us 23631.00 us 76
READLINK
0.17 3501.75 us 60.00 us 27637.00 us 16
READ
0.34 5093.86 us 60.00 us 69136.00 us 22
WRITE
4.56 100248.87 us 25.00 us 1502989.00 us 15
INODELK
9.64 1930.07 us 30.00 us 3404.00 us 1646
READDIRP
84.89 1367.04 us 34.00 us 828308.00 us 20463
LOOKUP
Duration: 76033 seconds
Data Read: 63112532714 bytes
Data Written: 1489661828 bytes
Interval 1 Stats:
Block Size: 32b+ 64b+
128b+
No. of Reads: 0 0
0
No. of Writes: 1 4
3
Block Size: 1024b+ 65536b+
131072b+
No. of Reads: 0 2
1
No. of Writes: 6 0
0
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 9
FORGET
0.00 0.00 us 0.00 us 0.00 us 45
RELEASE
0.00 0.00 us 0.00 us 0.00 us 177
RELEASEDIR
0.00 86.00 us 86.00 us 86.00 us 1
SETATTR
0.00 55.89 us 41.00 us 84.00 us 9
FSTAT
0.00 53.50 us 40.00 us 76.00 us 12
STATFS
0.00 160.00 us 130.00 us 184.00 us 5
UNLINK
0.00 46.83 us 30.00 us 60.00 us 18
LK
0.01 51.11 us 30.00 us 88.00 us 28
FINODELK
0.01 37.88 us 15.00 us 150.00 us 40
FLUSH
0.01 52.14 us 27.00 us 83.00 us 42
ENTRYLK
0.01 101.34 us 52.00 us 352.00 us 29
OPEN
0.02 209.44 us 150.00 us 511.00 us 16
CREATE
0.04 319.57 us 73.00 us 5551.00 us 28
FXATTROP
0.05 58.58 us 29.00 us 94.00 us 177
OPENDIR
0.23 669.42 us 41.00 us 23631.00 us 76
READLINK
0.26 3728.27 us 60.00 us 27637.00 us 15
READ
0.37 5742.93 us 60.00 us 69136.00 us 14
WRITE
6.88 167043.89 us 25.00 us 1502989.00 us 9
INODELK
10.95 1908.32 us 30.00 us 3404.00 us 1254
READDIRP
81.16 1701.62 us 34.00 us 828308.00 us 10421
LOOKUP
Duration: 18 seconds
Data Read: 310621 bytes
Data Written: 8465 bytes
Brick: 192.168.10.91:/export/brick1
-----------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+
4b+
No. of Reads: 0 0
0
No. of Writes: 3 3
6
Block Size: 8b+ 16b+
32b+
No. of Reads: 13 173
58
No. of Writes: 3370 5495
13458
Block Size: 64b+ 128b+
256b+
No. of Reads: 194 280
500
No. of Writes: 33703 34265
7020
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 1302 1888
3596
No. of Writes: 26022 17086
12495
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 5031 13204
21838
No. of Writes: 21889 32294
18289
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 73578 190508
2710663
No. of Writes: 17017 13980
144272
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 817553
FORGET
0.00 0.00 us 0.00 us 0.00 us 1663792
RELEASE
0.00 0.00 us 0.00 us 0.00 us 1438610
RELEASEDIR
0.00 85.00 us 85.00 us 85.00 us 1
READDIR
0.00 58.50 us 56.00 us 60.00 us 4
FSTAT
0.00 34.86 us 24.00 us 46.00 us 7
STAT
0.00 89.67 us 81.00 us 100.00 us 3
SETATTR
0.00 136.50 us 108.00 us 165.00 us 2
XATTROP
0.00 63.20 us 35.00 us 87.00 us 5
GETXATTR
0.00 54.73 us 39.00 us 66.00 us 15
INODELK
0.01 53.39 us 33.00 us 88.00 us 18
STATFS
0.01 58.00 us 32.00 us 259.00 us 25
LK
0.01 208.50 us 112.00 us 691.00 us 12
UNLINK
0.02 60.76 us 40.00 us 125.00 us 46
FINODELK
0.02 176.63 us 74.00 us 302.00 us 19
READ
0.02 52.51 us 23.00 us 97.00 us 68
ENTRYLK
0.02 48.00 us 18.00 us 579.00 us 81
FLUSH
0.03 206.95 us 163.00 us 497.00 us 22
CREATE
0.04 85.16 us 51.00 us 168.00 us 70
OPEN
0.04 145.02 us 92.00 us 344.00 us 45
FXATTROP
0.07 59.91 us 3.00 us 145.00 us 184
OPENDIR
0.13 1024.41 us 61.00 us 12251.00 us 22
WRITE
0.14 444.38 us 44.00 us 11184.00 us 52
READLINK
0.27 45104.00 us 45104.00 us 45104.00 us 1
TRUNCATE
20.33 1895.33 us 27.00 us 6083.00 us 1819
READDIRP
78.84 645.59 us 35.00 us 209082.00 us 20706
LOOKUP
Duration: 269212 seconds
Data Read: 376538217873 bytes
Data Written: 21968843126 bytes
Interval 1 Stats:
Block Size: 32b+ 64b+
128b+
No. of Reads: 0 0
0
No. of Writes: 1 4
3
Block Size: 1024b+ 32768b+
131072b+
No. of Reads: 0 1
10
No. of Writes: 6 0
0
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 8
FORGET
0.00 0.00 us 0.00 us 0.00 us 45
RELEASE
0.00 0.00 us 0.00 us 0.00 us 178
RELEASEDIR
0.00 81.00 us 81.00 us 81.00 us 1
SETATTR
0.00 85.00 us 85.00 us 85.00 us 1
READDIR
0.00 63.33 us 35.00 us 87.00 us 3
GETXATTR
0.00 58.50 us 56.00 us 60.00 us 4
FSTAT
0.00 34.86 us 24.00 us 46.00 us 7
STAT
0.00 54.00 us 47.00 us 66.00 us 9
INODELK
0.01 51.25 us 33.00 us 78.00 us 12
STATFS
0.01 156.20 us 112.00 us 202.00 us 5
UNLINK
0.01 61.33 us 34.00 us 259.00 us 18
LK
0.02 61.64 us 40.00 us 125.00 us 28
FINODELK
0.02 54.65 us 21.00 us 579.00 us 40
FLUSH
0.02 54.12 us 28.00 us 97.00 us 42
ENTRYLK
0.02 161.93 us 74.00 us 302.00 us 15
READ
0.02 83.97 us 68.00 us 107.00 us 29
OPEN
0.03 214.88 us 163.00 us 497.00 us 16
CREATE
0.04 144.21 us 92.00 us 344.00 us 28
FXATTROP
0.10 60.01 us 3.00 us 145.00 us 178
OPENDIR
0.10 316.62 us 44.00 us 8512.00 us 34
READLINK
0.15 1121.14 us 61.00 us 12251.00 us 14
WRITE
32.97 1895.33 us 27.00 us 6083.00 us 1819
READDIRP
66.46 667.09 us 35.00 us 67882.00 us 10418
LOOKUP
Duration: 18 seconds
Data Read: 1352163 bytes
Data Written: 8465 bytes
Profile 2:
Brick: 192.168.10.92:/export/brick1
-----------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+
4b+
No. of Reads: 0 0
0
No. of Writes: 133 6
37
Block Size: 8b+ 16b+
32b+
No. of Reads: 28 93
31
No. of Writes: 4577 6495
22231
Block Size: 64b+ 128b+
256b+
No. of Reads: 219 218
254
No. of Writes: 37641 41723
20519
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 484 981
696
No. of Writes: 39094 42613
203748
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 1079 1802
2840
No. of Writes: 220154 229725
215083
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 4227 14944
57816
No. of Writes: 60938 61967
793400
Block Size: 262144b+ 524288b+
1048576b+
No. of Reads: 6 5
2
No. of Writes: 5 0
0
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 1835089
FORGET
0.00 0.00 us 0.00 us 0.00 us 2639439
RELEASE
0.00 0.00 us 0.00 us 0.00 us 1440418
RELEASEDIR
0.00 53.50 us 43.00 us 64.00 us 2
READDIR
0.00 153.50 us 142.00 us 165.00 us 2
XATTROP
0.00 113.00 us 97.00 us 144.00 us 4
SETATTR
0.00 114.60 us 49.00 us 339.00 us 10
GETXATTR
0.00 55.00 us 33.00 us 100.00 us 24
INODELK
0.00 48.85 us 25.00 us 81.00 us 71
STATFS
0.01 49.47 us 31.00 us 76.00 us 92
LK
0.01 36.07 us 16.00 us 68.00 us 180
FLUSH
0.01 52.45 us 32.00 us 88.00 us 159
FINODELK
0.02 79.16 us 47.00 us 112.00 us 137
OPEN
0.02 51.06 us 26.00 us 95.00 us 216
ENTRYLK
0.03 24814.00 us 24814.00 us 24814.00 us 1
TRUNCATE
0.06 271.36 us 86.00 us 21829.00 us 158
FXATTROP
0.07 59.02 us 2.00 us 100.00 us 881
OPENDIR
0.07 5910.00 us 56.00 us 16873.00 us 9
READLINK
0.11 33.81 us 11.00 us 1950.00 us 2348
ACCESS
0.17 4181.53 us 129.00 us 41541.00 us 30
UNLINK
0.31 2788.77 us 59.00 us 56839.00 us 79
WRITE
0.37 2096.99 us 54.00 us 61234.00 us 125
READ
0.38 84.58 us 53.00 us 313.00 us 3191
FSTAT
0.50 61.21 us 20.00 us 14904.00 us 5873
STAT
0.76 7023.69 us 145.00 us 102353.00 us 78
CREATE
8.90 2002.80 us 361.00 us 3545.00 us 3191
READDIRP
88.19 1112.81 us 34.00 us 307904.00 us 56876
LOOKUP
Duration: 338222 seconds
Data Read: 9292898491 bytes
Data Written: 122127690284 bytes
Interval 4 Stats:
Block Size: 16b+ 32b+
64b+
No. of Reads: 0 0
2
No. of Writes: 1 3
12
Block Size: 128b+ 1024b+
2048b+
No. of Reads: 2 3
10
No. of Writes: 8 16
0
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 4 2
1
No. of Writes: 0 0
0
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 2 3
1
No. of Writes: 0 0
0
Block Size: 262144b+
No. of Reads: 2
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 21
FORGET
0.00 0.00 us 0.00 us 0.00 us 76
RELEASE
0.00 0.00 us 0.00 us 0.00 us 407
RELEASEDIR
0.00 144.00 us 144.00 us 144.00 us 1
SETATTR
0.00 63.33 us 56.00 us 69.00 us 3
READLINK
0.00 226.00 us 113.00 us 339.00 us 2
GETXATTR
0.00 54.89 us 39.00 us 70.00 us 9
INODELK
0.01 48.03 us 25.00 us 72.00 us 37
STATFS
0.01 36.75 us 16.00 us 65.00 us 59
FLUSH
0.01 50.44 us 33.00 us 76.00 us 48
LK
0.01 83.73 us 55.00 us 111.00 us 37
OPEN
0.01 53.79 us 32.00 us 80.00 us 80
FINODELK
0.02 50.55 us 26.00 us 83.00 us 104
ENTRYLK
0.03 135.21 us 86.00 us 228.00 us 80
FXATTROP
0.07 58.75 us 35.00 us 98.00 us 407
OPENDIR
0.10 1044.22 us 55.00 us 30268.00 us 32
READ
0.14 33.74 us 13.00 us 1044.00 us 1384
ACCESS
0.19 5076.42 us 141.00 us 41541.00 us 12
UNLINK
0.36 2860.70 us 59.00 us 29520.00 us 40
WRITE
0.36 84.16 us 53.00 us 182.00 us 1369
FSTAT
0.65 5225.18 us 145.00 us 28783.00 us 40
CREATE
0.73 61.98 us 20.00 us 14904.00 us 3771
STAT
8.53 2009.18 us 518.00 us 2532.00 us 1369
READDIRP
88.77 1156.14 us 36.00 us 232288.00 us 24747
LOOKUP
Duration: 48 seconds
Data Read: 1424095 bytes
Data Written: 22682 bytes
Brick: 192.168.10.91:/export/brick1
-----------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+
4b+
No. of Reads: 0 0
0
No. of Writes: 3 3
6
Block Size: 8b+ 16b+
32b+
No. of Reads: 13 173
58
No. of Writes: 3371 5496
13461
Block Size: 64b+ 128b+
256b+
No. of Reads: 194 280
500
No. of Writes: 33717 34277
7020
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 1302 1888
3596
No. of Writes: 26022 17110
12496
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 5031 13204
21838
No. of Writes: 21889 32294
18289
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 73579 190511
2710676
No. of Writes: 17017 13980
144272
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 817592
FORGET
0.00 0.00 us 0.00 us 0.00 us 1663910
RELEASE
0.00 0.00 us 0.00 us 0.00 us 1439269
RELEASEDIR
0.00 61.67 us 39.00 us 85.00 us 3
READDIR
0.00 136.50 us 108.00 us 165.00 us 2
XATTROP
0.00 91.00 us 81.00 us 100.00 us 4
SETATTR
0.00 58.86 us 35.00 us 87.00 us 7
GETXATTR
0.00 60.57 us 55.00 us 69.00 us 7
FSTAT
0.00 35.53 us 24.00 us 53.00 us 15
STAT
0.00 57.12 us 39.00 us 72.00 us 24
INODELK
0.01 51.25 us 33.00 us 90.00 us 71
STATFS
0.01 52.09 us 32.00 us 259.00 us 92
LK
0.01 175.87 us 112.00 us 691.00 us 30
UNLINK
0.01 41.84 us 16.00 us 579.00 us 175
FLUSH
0.02 56.23 us 25.00 us 125.00 us 158
FINODELK
0.02 51.59 us 23.00 us 97.00 us 216
ENTRYLK
0.02 85.10 us 51.00 us 168.00 us 133
OPEN
0.03 200.55 us 150.00 us 554.00 us 78
CREATE
0.06 163.03 us 38.00 us 11184.00 us 204
READLINK
0.07 1022.60 us 67.00 us 24446.00 us 40
READ
0.08 45104.00 us 45104.00 us 45104.00 us 1
TRUNCATE
0.09 59.52 us 3.00 us 145.00 us 843
OPENDIR
0.13 936.73 us 56.00 us 12251.00 us 78
WRITE
0.19 687.17 us 88.00 us 85925.00 us 157
FXATTROP
22.66 1881.29 us 23.00 us 6083.00 us 6698
READDIRP
76.59 754.63 us 33.00 us 209082.00 us 56449
LOOKUP
Duration: 269283 seconds
Data Read: 376540270509 bytes
Data Written: 21968880277 bytes
Interval 4 Stats:
Block Size: 16b+ 32b+
64b+
No. of Reads: 0 0
0
No. of Writes: 1 3
12
Block Size: 128b+ 1024b+
32768b+
No. of Reads: 0 0
1
No. of Writes: 8 16
0
Block Size: 65536b+ 131072b+
No. of Reads: 2 9
No. of Writes: 0 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 36
FORGET
0.00 0.00 us 0.00 us 0.00 us 76
RELEASE
0.00 0.00 us 0.00 us 0.00 us 408
RELEASEDIR
0.00 95.00 us 95.00 us 95.00 us 1
SETATTR
0.00 48.00 us 39.00 us 57.00 us 2
GETXATTR
0.00 50.00 us 39.00 us 61.00 us 2
READDIR
0.00 35.50 us 27.00 us 53.00 us 4
STAT
0.00 63.33 us 55.00 us 69.00 us 3
FSTAT
0.00 61.11 us 50.00 us 72.00 us 9
INODELK
0.01 50.41 us 34.00 us 90.00 us 37
STATFS
0.01 160.08 us 125.00 us 213.00 us 12
UNLINK
0.01 37.19 us 18.00 us 71.00 us 59
FLUSH
0.01 48.46 us 32.00 us 78.00 us 48
LK
0.01 84.92 us 57.00 us 118.00 us 37
OPEN
0.02 54.89 us 25.00 us 89.00 us 80
FINODELK
0.02 50.57 us 27.00 us 74.00 us 104
ENTRYLK
0.03 66.88 us 38.00 us 100.00 us 113
READLINK
0.03 196.90 us 150.00 us 554.00 us 40
CREATE
0.04 139.06 us 88.00 us 233.00 us 80
FXATTROP
0.10 58.95 us 8.00 us 93.00 us 408
OPENDIR
0.11 1675.56 us 67.00 us 24446.00 us 16
READ
0.17 1066.62 us 62.00 us 8506.00 us 40
WRITE
23.14 1889.02 us 23.00 us 3775.00 us 3073
READDIRP
76.29 773.40 us 33.00 us 135928.00 us 24746
LOOKUP
Duration: 48 seconds
Data Read: 1446611 bytes
Data Written: 22682 bytes
Brick: 192.168.10.90:/export/brick1
-----------------------------------
Cumulative Stats:
Block Size: 2b+ 8b+
16b+
No. of Reads: 0 6
32
No. of Writes: 1 1929
2605
Block Size: 32b+ 64b+
128b+
No. of Reads: 21 112
169
No. of Writes: 6986 12080
26020
Block Size: 256b+ 512b+
1024b+
No. of Reads: 85 299
809
No. of Writes: 743 18584
7233
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 790 1419
2885
No. of Writes: 1651 10647
16901
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 4588 5858
32434
No. of Writes: 2974 5758
3387
Block Size: 131072b+
No. of Reads: 455218
No. of Writes: 4601
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 604813
FORGET
0.00 0.00 us 0.00 us 0.00 us 1410912
RELEASE
0.00 0.00 us 0.00 us 0.00 us 990360
RELEASEDIR
0.00 44.33 us 16.00 us 62.00 us 6
READDIR
0.00 147.50 us 136.00 us 159.00 us 2
XATTROP
0.00 100.25 us 86.00 us 125.00 us 4
SETATTR
0.00 60.11 us 39.00 us 92.00 us 9
GETXATTR
0.00 60.19 us 41.00 us 84.00 us 16
FSTAT
0.00 45.50 us 17.00 us 182.00 us 38
STAT
0.00 51.37 us 31.00 us 81.00 us 71
STATFS
0.00 49.05 us 30.00 us 79.00 us 92
LK
0.00 168.87 us 115.00 us 287.00 us 30
UNLINK
0.01 38.74 us 15.00 us 150.00 us 150
FLUSH
0.01 54.76 us 30.00 us 236.00 us 158
FINODELK
0.01 87.17 us 49.00 us 352.00 us 106
OPEN
0.01 52.56 us 27.00 us 91.00 us 216
ENTRYLK
0.02 174.25 us 73.00 us 5551.00 us 157
FXATTROP
0.03 35786.00 us 35786.00 us 35786.00 us 1
TRUNCATE
0.05 62.12 us 3.00 us 1628.00 us 843
OPENDIR
0.10 685.58 us 41.00 us 51432.00 us 157
READLINK
0.36 6483.54 us 59.00 us 47225.00 us 63
READ
0.58 8304.77 us 60.00 us 185503.00 us 78
WRITE
0.67 9646.06 us 150.00 us 686579.00 us 78
CREATE
1.82 85338.88 us 25.00 us 1502989.00 us 24
INODELK
14.62 1920.48 us 23.00 us 9822.00 us 8567
READDIRP
81.71 1635.58 us 30.00 us 828308.00 us 56208
LOOKUP
Duration: 76104 seconds
Data Read: 63115678019 bytes
Data Written: 1489698979 bytes
Interval 4 Stats:
Block Size: 16b+ 32b+
64b+
No. of Reads: 0 0
0
No. of Writes: 1 3
12
Block Size: 128b+ 1024b+
4096b+
No. of Reads: 0 0
1
No. of Writes: 8 16
0
Block Size: 8192b+ 32768b+
65536b+
No. of Reads: 1 1
6
No. of Writes: 0 0
0
Block Size: 131072b+
No. of Reads: 10
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 29
FORGET
0.00 0.00 us 0.00 us 0.00 us 76
RELEASE
0.00 0.00 us 0.00 us 0.00 us 409
RELEASEDIR
0.00 125.00 us 125.00 us 125.00 us 1
SETATTR
0.00 43.75 us 16.00 us 60.00 us 4
READDIR
0.00 61.80 us 39.00 us 92.00 us 5
GETXATTR
0.00 65.71 us 44.00 us 76.00 us 7
FSTAT
0.00 48.50 us 24.00 us 182.00 us 28
STAT
0.00 50.08 us 31.00 us 70.00 us 37
STATFS
0.00 165.25 us 119.00 us 238.00 us 12
UNLINK
0.00 39.66 us 19.00 us 62.00 us 59
FLUSH
0.00 51.12 us 33.00 us 79.00 us 48
LK
0.01 82.78 us 64.00 us 104.00 us 37
OPEN
0.01 56.45 us 30.00 us 236.00 us 80
FINODELK
0.01 52.17 us 28.00 us 84.00 us 104
ENTRYLK
0.02 154.12 us 88.00 us 1577.00 us 80
FXATTROP
0.05 65.00 us 3.00 us 1628.00 us 409
OPENDIR
0.09 909.64 us 43.00 us 51432.00 us 61
READLINK
0.60 9748.22 us 59.00 us 47225.00 us 36
READ
0.86 12688.17 us 60.00 us 185503.00 us 40
WRITE
0.92 60488.89 us 50.00 us 543898.00 us 9
INODELK
1.26 18623.40 us 161.00 us 686579.00 us 40
CREATE
19.51 1951.60 us 25.00 us 9822.00 us 5889
READDIRP
76.64 1823.96 us 30.00 us 318395.00 us 24747
LOOKUP
Duration: 48 seconds
Data Read: 1931486 bytes
Data Written: 22682 bytes
Again, any assistance on this issue would be much appreciated.
Thank you.
On Sat, Dec 20, 2014 at 11:58 PM, Kyle Harris <kyle.harris98 at gmail.com>
wrote:
> Hello All,
>
> I have been experiencing difficulty with high process utilization on the
> glusterfs and glusterfsd processes. I thought I had it tracked down to
> Apache looking for files on the cluster drive that don't exist. I have
> been working to clean that up and now I have discovered another problem.
>
> When I reboot one of the nodes in my 3-node replicated cluster, as soon as
> it comes back up I see a bunch of entries such as follows:
>
> [2014-12-21 05:33:49.873989] I [dict.c:370:dict_get] (-->
> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7fe3bd0d2420] (-->
> /usr/lib64/libglusterfs.so.0(dict_get+0x4f)[0x7fe3bd0cbbdf] (-->
> /usr/lib64/glusterfs/3.6.1/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x1e1)[0x7fe3aad3ada1]
> (-->
> /usr/lib64/glusterfs/3.6.1/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x113)[0x7fe3aaf524e3]
> (-->
> /usr/lib64/glusterfs/3.6.1/xlator/performance/md-cache.so(mdc_lookup+0x300)[0x7fe3ab167310]
> ))))) 0-dict: !this || key=system.posix_acl_access
>
> While this is occurring, the process utilization for glusterfs and
> glusterfsd go sky high. Eventually they calm down on their own but the
> servers are virtually unusable until they do. Note this is on Centos 6.6
> with Gluster 3.6.1 and kernel 2.6.32 using a fuse mount with each of the 3
> servers having the client mounted for hosting web sites via Apache. There
> are many small files and some are php. I have not done any tuning to the
> default gluster parameters.
>
> Can someone tell me why this occurs after one of the nodes is rebooted and
> perhaps lend some guidance on what can be done to resolve this issue?
>
> Thank you.
>
> --
> Kyle
>
>
--
Kyle A. Harris
Kyle at TheHarrisHome.com
615-364-6752
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20141221/a843bf8e/attachment.html>
More information about the Gluster-users
mailing list