[Gluster-users] Self heal issues
Prasun Gera
prasun.gera at gmail.com
Fri Aug 7 07:57:35 UTC 2015
All the volume commands are taking several minutes to complete. Here's the
profiler's output:
Node3's brick is the one that was replaced. It's replica is node6
Brick: node1:/bricks/brickname
---------------------------------------------------
Cumulative Stats:
Block Size: 8b+ 16b+
32b+
No. of Reads: 0 13
2
No. of Writes: 60 0
141
Block Size: 64b+ 128b+
256b+
No. of Reads: 3 26
17
No. of Writes: 325 87
738
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 99 114
222
No. of Writes: 877 343
128
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 110 5
2
No. of Writes: 29401 78829
1448
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 19 33
34679
No. of Writes: 6233 22903
41202
Block Size: 262144b+ 524288b+
1048576b+
No. of Reads: 0 1
513
No. of Writes: 1 0
105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 126138
FORGET
0.00 0.00 us 0.00 us 0.00 us 141671
RELEASE
0.00 0.00 us 0.00 us 0.00 us 117718
RELEASEDIR
0.02 43.00 us 21.00 us 80.00 us 17
STAT
0.02 51.62 us 27.00 us 131.00 us 21
STATFS
0.09 42.78 us 11.00 us 1640.00 us 95
FLUSH
0.21 50.01 us 23.00 us 567.00 us 189
ENTRYLK
0.21 50.01 us 19.00 us 291.00 us 190
FINODELK
0.56 8578.33 us 53.00 us 25625.00 us 3
GETXATTR
0.62 148.89 us 71.00 us 2761.00 us 190
XATTROP
0.70 168.11 us 83.00 us 1019.00 us 190
FXATTROP
0.91 219.84 us 47.00 us 13732.00 us 190
SETATTR
1.21 71.74 us 16.00 us 11516.00 us 775
INODELK
1.47 354.88 us 56.00 us 22669.00 us 190
REMOVEXATTR
2.60 1254.69 us 122.00 us 12514.00 us 95
WRITE
3.25 194.10 us 51.00 us 48823.00 us 770
LOOKUP
88.15 43068.81 us 265.00 us 418819.00 us 94
CREATE
Duration: 644070 seconds
Data Read: 5089537361 bytes
Data Written: 9083513756 bytes
Interval 0 Stats:
Block Size: 8b+ 16b+
32b+
No. of Reads: 0 13
2
No. of Writes: 60 0
141
Block Size: 64b+ 128b+
256b+
No. of Reads: 3 26
17
No. of Writes: 325 87
738
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 99 114
222
No. of Writes: 877 343
128
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 110 5
2
No. of Writes: 29401 78829
1448
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 19 33
34679
No. of Writes: 6233 22903
41202
Block Size: 262144b+ 524288b+
1048576b+
No. of Reads: 0 1
513
No. of Writes: 1 0
105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 126138
FORGET
0.00 0.00 us 0.00 us 0.00 us 141671
RELEASE
0.00 0.00 us 0.00 us 0.00 us 117718
RELEASEDIR
0.02 43.00 us 21.00 us 80.00 us 17
STAT
0.02 51.62 us 27.00 us 131.00 us 21
STATFS
0.09 42.78 us 11.00 us 1640.00 us 95
FLUSH
0.21 50.01 us 23.00 us 567.00 us 189
ENTRYLK
0.21 50.01 us 19.00 us 291.00 us 190
FINODELK
0.56 8578.33 us 53.00 us 25625.00 us 3
GETXATTR
0.62 148.89 us 71.00 us 2761.00 us 190
XATTROP
0.70 168.11 us 83.00 us 1019.00 us 190
FXATTROP
0.91 219.84 us 47.00 us 13732.00 us 190
SETATTR
1.21 71.74 us 16.00 us 11516.00 us 775
INODELK
1.47 354.88 us 56.00 us 22669.00 us 190
REMOVEXATTR
2.60 1254.69 us 122.00 us 12514.00 us 95
WRITE
3.25 194.10 us 51.00 us 48823.00 us 770
LOOKUP
88.15 43068.81 us 265.00 us 418819.00 us 94
CREATE
Duration: 644070 seconds
Data Read: 5089537361 bytes
Data Written: 9083513756 bytes
Brick: node2:/bricks/brickname
-----------------------------------------------
Cumulative Stats:
Block Size: 8b+ 16b+
32b+
No. of Reads: 0 1
23
No. of Writes: 60 0
141
Block Size: 64b+ 128b+
256b+
No. of Reads: 45 47
363
No. of Writes: 325 87
738
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 515 37
42
No. of Writes: 877 343
128
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 17 0
1
No. of Writes: 29401 78829
1448
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 15 39
67031
No. of Writes: 6233 22903
41202
Block Size: 262144b+ 1048576b+
No. of Reads: 1 105
No. of Writes: 1 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 126136
FORGET
0.00 0.00 us 0.00 us 0.00 us 141671
RELEASE
0.00 0.00 us 0.00 us 0.00 us 117718
RELEASEDIR
0.02 74.00 us 74.00 us 74.00 us 1
STAT
0.15 196.00 us 141.00 us 288.00 us 3
GETXATTR
0.59 105.23 us 44.00 us 146.00 us 22
STATFS
2.05 83.88 us 11.00 us 137.00 us 96
FLUSH
4.79 98.61 us 20.00 us 146.00 us 191
ENTRYLK
5.03 102.93 us 22.00 us 158.00 us 192
FINODELK
5.52 226.07 us 136.00 us 295.00 us 96
WRITE
6.31 261.08 us 150.00 us 345.00 us 95
CREATE
6.72 137.53 us 50.00 us 214.00 us 192
SETATTR
7.68 157.14 us 75.00 us 237.00 us 192
REMOVEXATTR
8.13 166.49 us 81.00 us 282.00 us 192
XATTROP
8.26 169.01 us 76.00 us 275.00 us 192
FXATTROP
17.22 86.46 us 16.00 us 216.00 us 783
INODELK
27.54 138.09 us 43.00 us 266.00 us 784
LOOKUP
Duration: 644071 seconds
Data Read: 8902589511 bytes
Data Written: 9083513756 bytes
Interval 0 Stats:
Block Size: 8b+ 16b+
32b+
No. of Reads: 0 1
23
No. of Writes: 60 0
141
Block Size: 64b+ 128b+
256b+
No. of Reads: 45 47
363
No. of Writes: 325 87
738
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 515 37
42
No. of Writes: 877 343
128
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 17 0
1
No. of Writes: 29401 78829
1448
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 15 39
67031
No. of Writes: 6233 22903
41202
Block Size: 262144b+ 1048576b+
No. of Reads: 1 105
No. of Writes: 1 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 126136
FORGET
0.00 0.00 us 0.00 us 0.00 us 141671
RELEASE
0.00 0.00 us 0.00 us 0.00 us 117718
RELEASEDIR
0.02 74.00 us 74.00 us 74.00 us 1
STAT
0.15 196.00 us 141.00 us 288.00 us 3
GETXATTR
0.59 105.23 us 44.00 us 146.00 us 22
STATFS
2.05 83.88 us 11.00 us 137.00 us 96
FLUSH
4.79 98.61 us 20.00 us 146.00 us 191
ENTRYLK
5.03 102.93 us 22.00 us 158.00 us 192
FINODELK
5.52 226.07 us 136.00 us 295.00 us 96
WRITE
6.31 261.08 us 150.00 us 345.00 us 95
CREATE
6.72 137.53 us 50.00 us 214.00 us 192
SETATTR
7.68 157.14 us 75.00 us 237.00 us 192
REMOVEXATTR
8.13 166.49 us 81.00 us 282.00 us 192
XATTROP
8.26 169.01 us 76.00 us 275.00 us 192
FXATTROP
17.22 86.46 us 16.00 us 216.00 us 783
INODELK
27.54 138.09 us 43.00 us 266.00 us 784
LOOKUP
Duration: 644071 seconds
Data Read: 8902589511 bytes
Data Written: 9083513756 bytes
Brick: node3(sink):/bricks/brickname
------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+
4b+
No. of Reads: 0 0
0
No. of Writes: 11 26
125
Block Size: 8b+ 16b+
32b+
No. of Reads: 0 0
0
No. of Writes: 829 2341
9599
Block Size: 64b+ 128b+
256b+
No. of Reads: 0 0
0
No. of Writes: 12674 9229
27346
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 2 10
0
No. of Writes: 23414 28727
18372
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 1 0
0
No. of Writes: 48347 92134
9675
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 2 11
50
No. of Writes: 11717 24948
1022216
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 13805186
FORGET
0.00 0.00 us 0.00 us 0.00 us 17674891
RELEASE
0.00 0.00 us 0.00 us 0.00 us 218068
RELEASEDIR
0.00 24.00 us 24.00 us 24.00 us 1
OPENDIR
0.00 13.06 us 8.00 us 36.00 us 16
STAT
0.00 19.22 us 9.00 us 46.00 us 18
STATFS
0.00 45.54 us 22.00 us 82.00 us 13
SETXATTR
0.00 120.93 us 77.00 us 156.00 us 14
XATTROP
0.00 11.01 us 7.00 us 68.00 us 156
ENTRYLK
0.01 283.15 us 246.00 us 504.00 us 59
READDIR
0.02 899.19 us 39.00 us 17518.00 us 26
SETATTR
0.02 2004.85 us 38.00 us 10406.00 us 13
WRITE
0.02 2022.77 us 24.00 us 21677.00 us 13
REMOVEXATTR
0.03 2965.85 us 34.00 us 37695.00 us 13
FTRUNCATE
0.04 3691.62 us 31.00 us 18386.00 us 13
FLUSH
0.31 2105.65 us 23.00 us 57417.00 us 177
OPEN
0.43 2603.12 us 57.00 us 73929.00 us 202
FXATTROP
0.46 3030.94 us 7.00 us 87892.00 us 186
FSTAT
1.07 33.17 us 18.00 us 17545.00 us 39491
GETXATTR
1.31 123033.46 us 75610.00 us 269227.00 us 13
FSYNC
46.14 704.24 us 6.00 us 268597.00 us 79866
INODELK
50.13 699.93 us 20.00 us 267607.00 us 87307
LOOKUP
Duration: 112674 seconds
Data Read: 7441454 bytes
Data Written: 138577629032 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+
4b+
No. of Reads: 0 0
0
No. of Writes: 11 26
125
Block Size: 8b+ 16b+
32b+
No. of Reads: 0 0
0
No. of Writes: 829 2341
9599
Block Size: 64b+ 128b+
256b+
No. of Reads: 0 0
0
No. of Writes: 12674 9229
27346
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 2 10
0
No. of Writes: 23414 28727
18372
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 1 0
0
No. of Writes: 48347 92134
9675
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 2 11
50
No. of Writes: 11717 24948
1022216
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 13805186
FORGET
0.00 0.00 us 0.00 us 0.00 us 17674862
RELEASE
0.00 0.00 us 0.00 us 0.00 us 218068
RELEASEDIR
0.00 24.00 us 24.00 us 24.00 us 1
OPENDIR
0.00 13.06 us 8.00 us 36.00 us 16
STAT
0.00 19.22 us 9.00 us 46.00 us 18
STATFS
0.00 45.54 us 22.00 us 82.00 us 13
SETXATTR
0.00 120.93 us 77.00 us 156.00 us 14
XATTROP
0.00 11.01 us 7.00 us 68.00 us 156
ENTRYLK
0.01 283.15 us 246.00 us 504.00 us 59
READDIR
0.02 899.19 us 39.00 us 17518.00 us 26
SETATTR
0.02 2004.85 us 38.00 us 10406.00 us 13
WRITE
0.02 2022.77 us 24.00 us 21677.00 us 13
REMOVEXATTR
0.03 2965.85 us 34.00 us 37695.00 us 13
FTRUNCATE
0.04 3691.62 us 31.00 us 18386.00 us 13
FLUSH
0.31 2105.65 us 23.00 us 57417.00 us 177
OPEN
0.43 2603.12 us 57.00 us 73929.00 us 202
FXATTROP
0.46 3030.94 us 7.00 us 87892.00 us 186
FSTAT
1.07 33.17 us 18.00 us 17545.00 us 39491
GETXATTR
1.31 123033.46 us 75610.00 us 269227.00 us 13
FSYNC
46.14 704.24 us 6.00 us 268597.00 us 79866
INODELK
50.13 699.93 us 20.00 us 267607.00 us 87307
LOOKUP
Duration: 112674 seconds
Data Read: 7441454 bytes
Data Written: 138577629032 bytes
Brick: node4:/bricks/brickname
-----------------------------------------------
Cumulative Stats:
Block Size: 8b+ 32b+
64b+
No. of Reads: 0 9
24
No. of Writes: 62 128
335
Block Size: 128b+ 256b+
512b+
No. of Reads: 21 177
257
No. of Writes: 186 779
885
Block Size: 1024b+ 2048b+
4096b+
No. of Reads: 30 14
7
No. of Writes: 286 101
29410
Block Size: 8192b+ 16384b+
32768b+
No. of Reads: 0 9
0
No. of Writes: 79662 1379
6187
Block Size: 65536b+ 131072b+
262144b+
No. of Reads: 29 3924
0
No. of Writes: 22467 32424
1
Block Size: 1048576b+
No. of Reads: 0
No. of Writes: 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 126295
FORGET
0.00 0.00 us 0.00 us 0.00 us 141875
RELEASE
0.00 0.00 us 0.00 us 0.00 us 117220
RELEASEDIR
0.12 119.50 us 102.00 us 147.00 us 4
GETXATTR
0.19 68.18 us 42.00 us 109.00 us 11
STAT
0.44 92.16 us 19.00 us 141.00 us 19
STATFS
1.20 68.03 us 11.00 us 120.00 us 71
FLUSH
2.93 83.02 us 18.00 us 136.00 us 142
ENTRYLK
3.15 89.18 us 16.00 us 160.00 us 142
FINODELK
3.40 192.82 us 76.00 us 271.00 us 71
WRITE
4.04 114.43 us 35.00 us 204.00 us 142
SETATTR
4.87 138.05 us 49.00 us 222.00 us 142
REMOVEXATTR
5.63 159.52 us 56.00 us 262.00 us 142
FXATTROP
10.68 73.81 us 11.00 us 202.00 us 582
INODELK
19.90 1127.35 us 116.00 us 27717.00 us 71
CREATE
21.67 613.82 us 46.00 us 65260.00 us 142
XATTROP
21.80 130.68 us 35.00 us 241.00 us 671
LOOKUP
Duration: 458509 seconds
Data Read: 517180943 bytes
Data Written: 7895152670 bytes
Interval 0 Stats:
Block Size: 8b+ 32b+
64b+
No. of Reads: 0 9
24
No. of Writes: 62 128
335
Block Size: 128b+ 256b+
512b+
No. of Reads: 21 177
257
No. of Writes: 186 779
885
Block Size: 1024b+ 2048b+
4096b+
No. of Reads: 30 14
7
No. of Writes: 286 101
29410
Block Size: 8192b+ 16384b+
32768b+
No. of Reads: 0 9
0
No. of Writes: 79662 1379
6187
Block Size: 65536b+ 131072b+
262144b+
No. of Reads: 29 3924
0
No. of Writes: 22467 32424
1
Block Size: 1048576b+
No. of Reads: 0
No. of Writes: 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 126295
FORGET
0.00 0.00 us 0.00 us 0.00 us 141875
RELEASE
0.00 0.00 us 0.00 us 0.00 us 117220
RELEASEDIR
0.12 119.50 us 102.00 us 147.00 us 4
GETXATTR
0.19 68.18 us 42.00 us 109.00 us 11
STAT
0.44 92.16 us 19.00 us 141.00 us 19
STATFS
1.20 68.03 us 11.00 us 120.00 us 71
FLUSH
2.93 83.02 us 18.00 us 136.00 us 142
ENTRYLK
3.15 89.18 us 16.00 us 160.00 us 142
FINODELK
3.40 192.82 us 76.00 us 271.00 us 71
WRITE
4.04 114.43 us 35.00 us 204.00 us 142
SETATTR
4.87 138.05 us 49.00 us 222.00 us 142
REMOVEXATTR
5.63 159.52 us 56.00 us 262.00 us 142
FXATTROP
10.68 73.81 us 11.00 us 202.00 us 582
INODELK
19.90 1127.35 us 116.00 us 27717.00 us 71
CREATE
21.67 613.82 us 46.00 us 65260.00 us 142
XATTROP
21.80 130.68 us 35.00 us 241.00 us 671
LOOKUP
Duration: 458509 seconds
Data Read: 517180943 bytes
Data Written: 7895152670 bytes
Brick: node5:/bricks/brickname
------------------------------------------------
Cumulative Stats:
Block Size: 8b+ 16b+
32b+
No. of Reads: 0 14
9
No. of Writes: 62 0
128
Block Size: 64b+ 128b+
256b+
No. of Reads: 23 56
225
No. of Writes: 335 186
779
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 357 106
233
No. of Writes: 885 286
102
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 128 11
15
No. of Writes: 29410 79662
1379
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 16 34
28965
No. of Writes: 6191 22467
32424
Block Size: 262144b+ 524288b+
1048576b+
No. of Reads: 5 3
984
No. of Writes: 1 0
105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 126301
FORGET
0.00 0.00 us 0.00 us 0.00 us 141880
RELEASE
0.00 0.00 us 0.00 us 0.00 us 117718
RELEASEDIR
0.01 51.75 us 41.00 us 69.00 us 4
GETXATTR
0.02 59.50 us 35.00 us 108.00 us 4
STAT
0.05 44.06 us 28.00 us 118.00 us 16
STATFS
0.10 24.78 us 15.00 us 99.00 us 59
FLUSH
0.34 41.01 us 24.00 us 107.00 us 118
ENTRYLK
0.37 44.43 us 25.00 us 156.00 us 118
FINODELK
0.76 90.88 us 70.00 us 183.00 us 118
SETATTR
0.81 193.88 us 162.00 us 283.00 us 59
WRITE
0.98 116.89 us 85.00 us 212.00 us 118
REMOVEXATTR
1.10 131.18 us 86.00 us 219.00 us 118
FXATTROP
1.52 44.02 us 24.00 us 1004.00 us 487
INODELK
3.35 86.29 us 52.00 us 183.00 us 549
LOOKUP
4.21 504.03 us 83.00 us 43099.00 us 118
XATTROP
86.39 20696.02 us 207.00 us 69802.00 us 59
CREATE
Duration: 644071 seconds
Data Read: 4837930222 bytes
Data Written: 7895351133 bytes
Interval 0 Stats:
Block Size: 8b+ 16b+
32b+
No. of Reads: 0 14
9
No. of Writes: 62 0
128
Block Size: 64b+ 128b+
256b+
No. of Reads: 23 56
225
No. of Writes: 335 186
779
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 357 106
233
No. of Writes: 885 286
102
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 128 11
15
No. of Writes: 29410 79662
1379
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 16 34
28965
No. of Writes: 6191 22467
32424
Block Size: 262144b+ 524288b+
1048576b+
No. of Reads: 5 3
984
No. of Writes: 1 0
105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 126301
FORGET
0.00 0.00 us 0.00 us 0.00 us 141880
RELEASE
0.00 0.00 us 0.00 us 0.00 us 117718
RELEASEDIR
0.01 51.75 us 41.00 us 69.00 us 4
GETXATTR
0.02 59.50 us 35.00 us 108.00 us 4
STAT
0.05 44.06 us 28.00 us 118.00 us 16
STATFS
0.10 24.78 us 15.00 us 99.00 us 59
FLUSH
0.34 41.01 us 24.00 us 107.00 us 118
ENTRYLK
0.37 44.43 us 25.00 us 156.00 us 118
FINODELK
0.76 90.88 us 70.00 us 183.00 us 118
SETATTR
0.81 193.88 us 162.00 us 283.00 us 59
WRITE
0.98 116.89 us 85.00 us 212.00 us 118
REMOVEXATTR
1.10 131.18 us 86.00 us 219.00 us 118
FXATTROP
1.52 44.02 us 24.00 us 1004.00 us 487
INODELK
3.35 86.29 us 52.00 us 183.00 us 549
LOOKUP
4.21 504.03 us 83.00 us 43099.00 us 118
XATTROP
86.39 20696.02 us 207.00 us 69802.00 us 59
CREATE
Duration: 644071 seconds
Data Read: 4837930222 bytes
Data Written: 7895351133 bytes
Brick: node6(source):/bricks/brickname
--------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+
4b+
No. of Reads: 7 18
89
No. of Writes: 4 8
37
Block Size: 8b+ 16b+
32b+
No. of Reads: 727 2325
9459
No. of Writes: 108 54
188
Block Size: 64b+ 128b+
256b+
No. of Reads: 12419 9313
27616
No. of Writes: 360 85
772
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 23708 28691
18594
No. of Writes: 847 313
138
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 19484 12596
8458
No. of Writes: 29185 79632
1431
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 5695 5755
1062899
No. of Writes: 6168 19435
32017
Block Size: 262144b+ 1048576b+
No. of Reads: 0 0
No. of Writes: 1 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 13806534
FORGET
0.00 0.00 us 0.00 us 0.00 us 17813646
RELEASE
0.00 0.00 us 0.00 us 0.00 us 223324
RELEASEDIR
0.00 560.00 us 560.00 us 560.00 us 1
ENTRYLK
0.00 3901.00 us 3901.00 us 3901.00 us 1
SETXATTR
0.00 4010.00 us 4010.00 us 4010.00 us 1
REMOVEXATTR
0.01 62446.08 us 8.00 us 365433.00 us 13
FLUSH
0.01 93887.77 us 52.00 us 588566.00 us 13
SETATTR
0.03 10772.83 us 28.00 us 1121761.00 us 253
GETXATTR
0.04 3190096.00 us 3190096.00 us 3190096.00 us 1
READDIR
0.09 558307.69 us 179931.00 us 3188951.00 us 13
READ
0.11 616756.00 us 74.00 us 7307745.00 us 14
XATTROP
0.12 4754785.50 us 48.00 us 9509523.00 us 2
OPENDIR
0.15 1799185.00 us 2310.00 us 5023537.00 us 7
STATFS
0.16 68757.98 us 10.00 us 872148.00 us 189
FSTAT
0.31 143533.93 us 42.00 us 7002195.00 us 174
OPEN
0.40 160262.95 us 661.00 us 2825083.00 us 202
READDIRP
1.55 624450.87 us 31.00 us 7397432.00 us 203
FXATTROP
22.43 212161.62 us 12.00 us 7397413.00 us 8639
INODELK
74.60 541421.09 us 63.00 us 14463033.00 us 11261
LOOKUP
Duration: 644071 seconds
Data Read: 140706386722 bytes
Data Written: 7549422894 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+
4b+
No. of Reads: 7 18
89
No. of Writes: 4 8
37
Block Size: 8b+ 16b+
32b+
No. of Reads: 727 2325
9459
No. of Writes: 108 54
188
Block Size: 64b+ 128b+
256b+
No. of Reads: 12419 9313
27616
No. of Writes: 360 85
772
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 23708 28691
18594
No. of Writes: 847 313
138
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 19484 12596
8458
No. of Writes: 29185 79632
1431
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 5695 5755
1062899
No. of Writes: 6168 19435
32017
Block Size: 262144b+ 1048576b+
No. of Reads: 0 0
No. of Writes: 1 105
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 13806534
FORGET
0.00 0.00 us 0.00 us 0.00 us 17813657
RELEASE
0.00 0.00 us 0.00 us 0.00 us 223324
RELEASEDIR
0.00 560.00 us 560.00 us 560.00 us 1
ENTRYLK
0.00 3901.00 us 3901.00 us 3901.00 us 1
SETXATTR
0.00 4010.00 us 4010.00 us 4010.00 us 1
REMOVEXATTR
0.01 62446.08 us 8.00 us 365433.00 us 13
FLUSH
0.01 93887.77 us 52.00 us 588566.00 us 13
SETATTR
0.03 10772.83 us 28.00 us 1121761.00 us 253
GETXATTR
0.04 3190096.00 us 3190096.00 us 3190096.00 us 1
READDIR
0.09 558307.69 us 179931.00 us 3188951.00 us 13
READ
0.11 616756.00 us 74.00 us 7307745.00 us 14
XATTROP
0.12 4754785.50 us 48.00 us 9509523.00 us 2
OPENDIR
0.15 1799185.00 us 2310.00 us 5023537.00 us 7
STATFS
0.16 68757.98 us 10.00 us 872148.00 us 189
FSTAT
0.31 143533.93 us 42.00 us 7002195.00 us 174
OPEN
0.40 160262.95 us 661.00 us 2825083.00 us 202
READDIRP
1.55 624450.87 us 31.00 us 7397432.00 us 203
FXATTROP
22.43 212161.62 us 12.00 us 7397413.00 us 8639
INODELK
74.60 541421.09 us 63.00 us 14463033.00 us 11261
LOOKUP
Duration: 644071 seconds
Data Read: 140706386722 bytes
Data Written: 7549422894 bytes
On Fri, Aug 7, 2015 at 12:17 AM, Ravishankar N <ravishankar at redhat.com>
wrote:
>
>
> On 08/07/2015 12:11 PM, Prasun Gera wrote:
>
> No, no noticeable difference. Still very high, possibly higher than
> before.
>
>
> I was guessing that the cpu usage could be because of the diff algorithm
> which computes checksums (which is a cpu intensive task). That doesn't
> seem to be the case. Could you do a volume profile and see the FOPS that
> are happening on the bricks and share the result?
> 1.gluster volume profile <volname> start
> 2. gluster volume profile <volname> info
> 3. wait 10-15 seconds
> 4.gluster volume profile <volname> info
>
>
>
> The system has come down to a crawl. It's difficult to even ssh or run any
> commands on the terminal. Do you make anything of the logs ? The brick log
> is just a giant alternating stream of those two lines I mentioned earlier.
>
>
>
>
> On Thu, Aug 6, 2015 at 10:10 PM, Ravishankar N <ravishankar at redhat.com>
> wrote:
>
>>
>>
>> On 08/07/2015 01:33 AM, Prasun Gera wrote:
>>
>>> I replaced the brick in a node in my 3x2 dist+repl volume (RHS 3). I'm
>>> seeing that the heal process, which should essentially be a dump from the
>>> working replica to the newly added one is taking exceptionally long. It has
>>> moved ~100 G over a day on a 1Gigabit network. The CPU usage on both the
>>> nodes of the replica has been pretty high.
>>>
>>
>> Does setting `cluster.data-self-heal-algorithm` to full make a difference
>> in the cpu usage?
>>
>>
>> I also think that nagios is making it worse. The heal is slow enough as
>>> it is, and nagios keeps triggering heal info, which I think never
>>> completes. I also see my logs filling up These are some of the log contents
>>> which I got by running tail on them:
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150807/e1290e72/attachment.html>
More information about the Gluster-users
mailing list