[Gluster-users] Self heal issues
Ravishankar N
ravishankar at redhat.com
Fri Aug 7 10:04:29 UTC 2015
So the nodes 3 and 6 seem to indicate inode-locks and lookups are of the
highest latency. This only seems to confirm self-heals are happening.
If you are unable to use the system because of this, you could try
killing the self-heal daemons on both these nodes (kill `pgrep -f
glustershd`) to stop heals. You can then do a lookup of the files from
the mount, which will also trigger heals.
Restart the selfheal daemons (with `gluster vol start volname force`)
when you think you can spare the volume for heals again. The sooner the
better though.
For the brick log errors, we are suspecting it could be something
related to selinux.
Can you tell what kind of data is there in your volume? - no. of files,
avg. file size, brick size, network connection speed etc? Perhaps we can
try to reproduce the issue and identify the bottle neck.
Thanks,
Ravi
On 08/07/2015 01:27 PM, Prasun Gera wrote:
> All the volume commands are taking several minutes to complete. Here's
> the profiler's output:
> Node3's brick is the one that was replaced. It's replica is node6
>
>
> Brick: node1:/bricks/brickname
> ---------------------------------------------------
> Cumulative Stats:
> Block Size: 8b+ 16b+ 32b+
> No. of Reads: 0 13 2
> No. of Writes: 60 0 141
> Block Size: 64b+ 128b+ 256b+
> No. of Reads: 3 26 17
> No. of Writes: 325 87 738
> Block Size: 512b+ 1024b+ 2048b+
> No. of Reads: 99 114 222
> No. of Writes: 877 343 128
> Block Size: 4096b+ 8192b+ 16384b+
> No. of Reads: 110 5 2
> No. of Writes: 29401 78829 1448
> Block Size: 32768b+ 65536b+ 131072b+
> No. of Reads: 19 33 34679
> No. of Writes: 6233 22903 41202
> Block Size: 262144b+ 524288b+ 1048576b+
> No. of Reads: 0 1 513
> No. of Writes: 1 0 105
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 126138
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 141671
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 117718
> RELEASEDIR
> 0.02 43.00 us 21.00 us 80.00 us 17
> STAT
> 0.02 51.62 us 27.00 us 131.00 us 21
> STATFS
> 0.09 42.78 us 11.00 us 1640.00 us 95
> FLUSH
> 0.21 50.01 us 23.00 us 567.00 us 189
> ENTRYLK
> 0.21 50.01 us 19.00 us 291.00 us 190
> FINODELK
> 0.56 8578.33 us 53.00 us 25625.00 us 3
> GETXATTR
> 0.62 148.89 us 71.00 us 2761.00 us 190
> XATTROP
> 0.70 168.11 us 83.00 us 1019.00 us 190
> FXATTROP
> 0.91 219.84 us 47.00 us 13732.00 us 190
> SETATTR
> 1.21 71.74 us 16.00 us 11516.00 us 775
> INODELK
> 1.47 354.88 us 56.00 us 22669.00 us 190
> REMOVEXATTR
> 2.60 1254.69 us 122.00 us 12514.00 us 95
> WRITE
> 3.25 194.10 us 51.00 us 48823.00 us 770
> LOOKUP
> 88.15 43068.81 us 265.00 us 418819.00 us 94
> CREATE
> Duration: 644070 seconds
> Data Read: 5089537361 bytes
> Data Written: 9083513756 bytes
> Interval 0 Stats:
> Block Size: 8b+ 16b+ 32b+
> No. of Reads: 0 13 2
> No. of Writes: 60 0 141
> Block Size: 64b+ 128b+ 256b+
> No. of Reads: 3 26 17
> No. of Writes: 325 87 738
> Block Size: 512b+ 1024b+ 2048b+
> No. of Reads: 99 114 222
> No. of Writes: 877 343 128
> Block Size: 4096b+ 8192b+ 16384b+
> No. of Reads: 110 5 2
> No. of Writes: 29401 78829 1448
> Block Size: 32768b+ 65536b+ 131072b+
> No. of Reads: 19 33 34679
> No. of Writes: 6233 22903 41202
> Block Size: 262144b+ 524288b+ 1048576b+
> No. of Reads: 0 1 513
> No. of Writes: 1 0 105
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 126138
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 141671
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 117718
> RELEASEDIR
> 0.02 43.00 us 21.00 us 80.00 us 17
> STAT
> 0.02 51.62 us 27.00 us 131.00 us 21
> STATFS
> 0.09 42.78 us 11.00 us 1640.00 us 95
> FLUSH
> 0.21 50.01 us 23.00 us 567.00 us 189
> ENTRYLK
> 0.21 50.01 us 19.00 us 291.00 us 190
> FINODELK
> 0.56 8578.33 us 53.00 us 25625.00 us 3
> GETXATTR
> 0.62 148.89 us 71.00 us 2761.00 us 190
> XATTROP
> 0.70 168.11 us 83.00 us 1019.00 us 190
> FXATTROP
> 0.91 219.84 us 47.00 us 13732.00 us 190
> SETATTR
> 1.21 71.74 us 16.00 us 11516.00 us 775
> INODELK
> 1.47 354.88 us 56.00 us 22669.00 us 190
> REMOVEXATTR
> 2.60 1254.69 us 122.00 us 12514.00 us 95
> WRITE
> 3.25 194.10 us 51.00 us 48823.00 us 770
> LOOKUP
> 88.15 43068.81 us 265.00 us 418819.00 us 94
> CREATE
> Duration: 644070 seconds
> Data Read: 5089537361 bytes
> Data Written: 9083513756 bytes
> Brick: node2:/bricks/brickname
> -----------------------------------------------
> Cumulative Stats:
> Block Size: 8b+ 16b+ 32b+
> No. of Reads: 0 1 23
> No. of Writes: 60 0 141
> Block Size: 64b+ 128b+ 256b+
> No. of Reads: 45 47 363
> No. of Writes: 325 87 738
> Block Size: 512b+ 1024b+ 2048b+
> No. of Reads: 515 37 42
> No. of Writes: 877 343 128
> Block Size: 4096b+ 8192b+ 16384b+
> No. of Reads: 17 0 1
> No. of Writes: 29401 78829 1448
> Block Size: 32768b+ 65536b+ 131072b+
> No. of Reads: 15 39 67031
> No. of Writes: 6233 22903 41202
> Block Size: 262144b+ 1048576b+
> No. of Reads: 1 105
> No. of Writes: 1 105
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 126136
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 141671
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 117718
> RELEASEDIR
> 0.02 74.00 us 74.00 us 74.00 us 1
> STAT
> 0.15 196.00 us 141.00 us 288.00 us 3
> GETXATTR
> 0.59 105.23 us 44.00 us 146.00 us 22
> STATFS
> 2.05 83.88 us 11.00 us 137.00 us 96
> FLUSH
> 4.79 98.61 us 20.00 us 146.00 us 191
> ENTRYLK
> 5.03 102.93 us 22.00 us 158.00 us 192
> FINODELK
> 5.52 226.07 us 136.00 us 295.00 us 96
> WRITE
> 6.31 261.08 us 150.00 us 345.00 us 95
> CREATE
> 6.72 137.53 us 50.00 us 214.00 us 192
> SETATTR
> 7.68 157.14 us 75.00 us 237.00 us 192
> REMOVEXATTR
> 8.13 166.49 us 81.00 us 282.00 us 192
> XATTROP
> 8.26 169.01 us 76.00 us 275.00 us 192
> FXATTROP
> 17.22 86.46 us 16.00 us 216.00 us 783
> INODELK
> 27.54 138.09 us 43.00 us 266.00 us 784
> LOOKUP
> Duration: 644071 seconds
> Data Read: 8902589511 bytes
> Data Written: 9083513756 bytes
> Interval 0 Stats:
> Block Size: 8b+ 16b+ 32b+
> No. of Reads: 0 1 23
> No. of Writes: 60 0 141
> Block Size: 64b+ 128b+ 256b+
> No. of Reads: 45 47 363
> No. of Writes: 325 87 738
> Block Size: 512b+ 1024b+ 2048b+
> No. of Reads: 515 37 42
> No. of Writes: 877 343 128
> Block Size: 4096b+ 8192b+ 16384b+
> No. of Reads: 17 0 1
> No. of Writes: 29401 78829 1448
> Block Size: 32768b+ 65536b+ 131072b+
> No. of Reads: 15 39 67031
> No. of Writes: 6233 22903 41202
> Block Size: 262144b+ 1048576b+
> No. of Reads: 1 105
> No. of Writes: 1 105
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 126136
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 141671
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 117718
> RELEASEDIR
> 0.02 74.00 us 74.00 us 74.00 us 1
> STAT
> 0.15 196.00 us 141.00 us 288.00 us 3
> GETXATTR
> 0.59 105.23 us 44.00 us 146.00 us 22
> STATFS
> 2.05 83.88 us 11.00 us 137.00 us 96
> FLUSH
> 4.79 98.61 us 20.00 us 146.00 us 191
> ENTRYLK
> 5.03 102.93 us 22.00 us 158.00 us 192
> FINODELK
> 5.52 226.07 us 136.00 us 295.00 us 96
> WRITE
> 6.31 261.08 us 150.00 us 345.00 us 95
> CREATE
> 6.72 137.53 us 50.00 us 214.00 us 192
> SETATTR
> 7.68 157.14 us 75.00 us 237.00 us 192
> REMOVEXATTR
> 8.13 166.49 us 81.00 us 282.00 us 192
> XATTROP
> 8.26 169.01 us 76.00 us 275.00 us 192
> FXATTROP
> 17.22 86.46 us 16.00 us 216.00 us 783
> INODELK
> 27.54 138.09 us 43.00 us 266.00 us 784
> LOOKUP
> Duration: 644071 seconds
> Data Read: 8902589511 bytes
> Data Written: 9083513756 bytes
> Brick: node3(sink):/bricks/brickname
> ------------------------------------------------
> Cumulative Stats:
> Block Size: 1b+ 2b+ 4b+
> No. of Reads: 0 0 0
> No. of Writes: 11 26 125
> Block Size: 8b+ 16b+ 32b+
> No. of Reads: 0 0 0
> No. of Writes: 829 2341 9599
> Block Size: 64b+ 128b+ 256b+
> No. of Reads: 0 0 0
> No. of Writes: 12674 9229 27346
> Block Size: 512b+ 1024b+ 2048b+
> No. of Reads: 2 10 0
> No. of Writes: 23414 28727 18372
> Block Size: 4096b+ 8192b+ 16384b+
> No. of Reads: 1 0 0
> No. of Writes: 48347 92134 9675
> Block Size: 32768b+ 65536b+ 131072b+
> No. of Reads: 2 11 50
> No. of Writes: 11717 24948 1022216
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 13805186
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 17674891
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 218068
> RELEASEDIR
> 0.00 24.00 us 24.00 us 24.00 us 1
> OPENDIR
> 0.00 13.06 us 8.00 us 36.00 us 16
> STAT
> 0.00 19.22 us 9.00 us 46.00 us 18
> STATFS
> 0.00 45.54 us 22.00 us 82.00 us 13
> SETXATTR
> 0.00 120.93 us 77.00 us 156.00 us 14
> XATTROP
> 0.00 11.01 us 7.00 us 68.00 us 156
> ENTRYLK
> 0.01 283.15 us 246.00 us 504.00 us 59
> READDIR
> 0.02 899.19 us 39.00 us 17518.00 us 26
> SETATTR
> 0.02 2004.85 us 38.00 us 10406.00 us 13
> WRITE
> 0.02 2022.77 us 24.00 us 21677.00 us 13
> REMOVEXATTR
> 0.03 2965.85 us 34.00 us 37695.00 us 13
> FTRUNCATE
> 0.04 3691.62 us 31.00 us 18386.00 us 13
> FLUSH
> 0.31 2105.65 us 23.00 us 57417.00 us 177
> OPEN
> 0.43 2603.12 us 57.00 us 73929.00 us 202
> FXATTROP
> 0.46 3030.94 us 7.00 us 87892.00 us 186
> FSTAT
> 1.07 33.17 us 18.00 us 17545.00 us 39491
> GETXATTR
> 1.31 123033.46 us 75610.00 us 269227.00 us 13
> FSYNC
> 46.14 704.24 us 6.00 us 268597.00 us 79866
> INODELK
> 50.13 699.93 us 20.00 us 267607.00 us 87307
> LOOKUP
> Duration: 112674 seconds
> Data Read: 7441454 bytes
> Data Written: 138577629032 bytes
> Interval 0 Stats:
> Block Size: 1b+ 2b+ 4b+
> No. of Reads: 0 0 0
> No. of Writes: 11 26 125
> Block Size: 8b+ 16b+ 32b+
> No. of Reads: 0 0 0
> No. of Writes: 829 2341 9599
> Block Size: 64b+ 128b+ 256b+
> No. of Reads: 0 0 0
> No. of Writes: 12674 9229 27346
> Block Size: 512b+ 1024b+ 2048b+
> No. of Reads: 2 10 0
> No. of Writes: 23414 28727 18372
> Block Size: 4096b+ 8192b+ 16384b+
> No. of Reads: 1 0 0
> No. of Writes: 48347 92134 9675
> Block Size: 32768b+ 65536b+ 131072b+
> No. of Reads: 2 11 50
> No. of Writes: 11717 24948 1022216
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 13805186
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 17674862
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 218068
> RELEASEDIR
> 0.00 24.00 us 24.00 us 24.00 us 1
> OPENDIR
> 0.00 13.06 us 8.00 us 36.00 us 16
> STAT
> 0.00 19.22 us 9.00 us 46.00 us 18
> STATFS
> 0.00 45.54 us 22.00 us 82.00 us 13
> SETXATTR
> 0.00 120.93 us 77.00 us 156.00 us 14
> XATTROP
> 0.00 11.01 us 7.00 us 68.00 us 156
> ENTRYLK
> 0.01 283.15 us 246.00 us 504.00 us 59
> READDIR
> 0.02 899.19 us 39.00 us 17518.00 us 26
> SETATTR
> 0.02 2004.85 us 38.00 us 10406.00 us 13
> WRITE
> 0.02 2022.77 us 24.00 us 21677.00 us 13
> REMOVEXATTR
> 0.03 2965.85 us 34.00 us 37695.00 us 13
> FTRUNCATE
> 0.04 3691.62 us 31.00 us 18386.00 us 13
> FLUSH
> 0.31 2105.65 us 23.00 us 57417.00 us 177
> OPEN
> 0.43 2603.12 us 57.00 us 73929.00 us 202
> FXATTROP
> 0.46 3030.94 us 7.00 us 87892.00 us 186
> FSTAT
> 1.07 33.17 us 18.00 us 17545.00 us 39491
> GETXATTR
> 1.31 123033.46 us 75610.00 us 269227.00 us 13
> FSYNC
> 46.14 704.24 us 6.00 us 268597.00 us 79866
> INODELK
> 50.13 699.93 us 20.00 us 267607.00 us 87307
> LOOKUP
> Duration: 112674 seconds
> Data Read: 7441454 bytes
> Data Written: 138577629032 bytes
> Brick: node4:/bricks/brickname
> -----------------------------------------------
> Cumulative Stats:
> Block Size: 8b+ 32b+ 64b+
> No. of Reads: 0 9 24
> No. of Writes: 62 128 335
> Block Size: 128b+ 256b+ 512b+
> No. of Reads: 21 177 257
> No. of Writes: 186 779 885
> Block Size: 1024b+ 2048b+ 4096b+
> No. of Reads: 30 14 7
> No. of Writes: 286 101 29410
> Block Size: 8192b+ 16384b+ 32768b+
> No. of Reads: 0 9 0
> No. of Writes: 79662 1379 6187
> Block Size: 65536b+ 131072b+ 262144b+
> No. of Reads: 29 3924 0
> No. of Writes: 22467 32424 1
> Block Size: 1048576b+
> No. of Reads: 0
> No. of Writes: 105
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 126295
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 141875
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 117220
> RELEASEDIR
> 0.12 119.50 us 102.00 us 147.00 us 4
> GETXATTR
> 0.19 68.18 us 42.00 us 109.00 us 11
> STAT
> 0.44 92.16 us 19.00 us 141.00 us 19
> STATFS
> 1.20 68.03 us 11.00 us 120.00 us 71
> FLUSH
> 2.93 83.02 us 18.00 us 136.00 us 142
> ENTRYLK
> 3.15 89.18 us 16.00 us 160.00 us 142
> FINODELK
> 3.40 192.82 us 76.00 us 271.00 us 71
> WRITE
> 4.04 114.43 us 35.00 us 204.00 us 142
> SETATTR
> 4.87 138.05 us 49.00 us 222.00 us 142
> REMOVEXATTR
> 5.63 159.52 us 56.00 us 262.00 us 142
> FXATTROP
> 10.68 73.81 us 11.00 us 202.00 us 582
> INODELK
> 19.90 1127.35 us 116.00 us 27717.00 us 71
> CREATE
> 21.67 613.82 us 46.00 us 65260.00 us 142
> XATTROP
> 21.80 130.68 us 35.00 us 241.00 us 671
> LOOKUP
> Duration: 458509 seconds
> Data Read: 517180943 bytes
> Data Written: 7895152670 bytes
> Interval 0 Stats:
> Block Size: 8b+ 32b+ 64b+
> No. of Reads: 0 9 24
> No. of Writes: 62 128 335
> Block Size: 128b+ 256b+ 512b+
> No. of Reads: 21 177 257
> No. of Writes: 186 779 885
> Block Size: 1024b+ 2048b+ 4096b+
> No. of Reads: 30 14 7
> No. of Writes: 286 101 29410
> Block Size: 8192b+ 16384b+ 32768b+
> No. of Reads: 0 9 0
> No. of Writes: 79662 1379 6187
> Block Size: 65536b+ 131072b+ 262144b+
> No. of Reads: 29 3924 0
> No. of Writes: 22467 32424 1
> Block Size: 1048576b+
> No. of Reads: 0
> No. of Writes: 105
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 126295
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 141875
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 117220
> RELEASEDIR
> 0.12 119.50 us 102.00 us 147.00 us 4
> GETXATTR
> 0.19 68.18 us 42.00 us 109.00 us 11
> STAT
> 0.44 92.16 us 19.00 us 141.00 us 19
> STATFS
> 1.20 68.03 us 11.00 us 120.00 us 71
> FLUSH
> 2.93 83.02 us 18.00 us 136.00 us 142
> ENTRYLK
> 3.15 89.18 us 16.00 us 160.00 us 142
> FINODELK
> 3.40 192.82 us 76.00 us 271.00 us 71
> WRITE
> 4.04 114.43 us 35.00 us 204.00 us 142
> SETATTR
> 4.87 138.05 us 49.00 us 222.00 us 142
> REMOVEXATTR
> 5.63 159.52 us 56.00 us 262.00 us 142
> FXATTROP
> 10.68 73.81 us 11.00 us 202.00 us 582
> INODELK
> 19.90 1127.35 us 116.00 us 27717.00 us 71
> CREATE
> 21.67 613.82 us 46.00 us 65260.00 us 142
> XATTROP
> 21.80 130.68 us 35.00 us 241.00 us 671
> LOOKUP
> Duration: 458509 seconds
> Data Read: 517180943 bytes
> Data Written: 7895152670 bytes
> Brick: node5:/bricks/brickname
> ------------------------------------------------
> Cumulative Stats:
> Block Size: 8b+ 16b+ 32b+
> No. of Reads: 0 14 9
> No. of Writes: 62 0 128
> Block Size: 64b+ 128b+ 256b+
> No. of Reads: 23 56 225
> No. of Writes: 335 186 779
> Block Size: 512b+ 1024b+ 2048b+
> No. of Reads: 357 106 233
> No. of Writes: 885 286 102
> Block Size: 4096b+ 8192b+ 16384b+
> No. of Reads: 128 11 15
> No. of Writes: 29410 79662 1379
> Block Size: 32768b+ 65536b+ 131072b+
> No. of Reads: 16 34 28965
> No. of Writes: 6191 22467 32424
> Block Size: 262144b+ 524288b+ 1048576b+
> No. of Reads: 5 3 984
> No. of Writes: 1 0 105
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 126301
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 141880
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 117718
> RELEASEDIR
> 0.01 51.75 us 41.00 us 69.00 us 4
> GETXATTR
> 0.02 59.50 us 35.00 us 108.00 us 4
> STAT
> 0.05 44.06 us 28.00 us 118.00 us 16
> STATFS
> 0.10 24.78 us 15.00 us 99.00 us 59
> FLUSH
> 0.34 41.01 us 24.00 us 107.00 us 118
> ENTRYLK
> 0.37 44.43 us 25.00 us 156.00 us 118
> FINODELK
> 0.76 90.88 us 70.00 us 183.00 us 118
> SETATTR
> 0.81 193.88 us 162.00 us 283.00 us 59
> WRITE
> 0.98 116.89 us 85.00 us 212.00 us 118
> REMOVEXATTR
> 1.10 131.18 us 86.00 us 219.00 us 118
> FXATTROP
> 1.52 44.02 us 24.00 us 1004.00 us 487
> INODELK
> 3.35 86.29 us 52.00 us 183.00 us 549
> LOOKUP
> 4.21 504.03 us 83.00 us 43099.00 us 118
> XATTROP
> 86.39 20696.02 us 207.00 us 69802.00 us 59
> CREATE
> Duration: 644071 seconds
> Data Read: 4837930222 bytes
> Data Written: 7895351133 bytes
> Interval 0 Stats:
> Block Size: 8b+ 16b+ 32b+
> No. of Reads: 0 14 9
> No. of Writes: 62 0 128
> Block Size: 64b+ 128b+ 256b+
> No. of Reads: 23 56 225
> No. of Writes: 335 186 779
> Block Size: 512b+ 1024b+ 2048b+
> No. of Reads: 357 106 233
> No. of Writes: 885 286 102
> Block Size: 4096b+ 8192b+ 16384b+
> No. of Reads: 128 11 15
> No. of Writes: 29410 79662 1379
> Block Size: 32768b+ 65536b+ 131072b+
> No. of Reads: 16 34 28965
> No. of Writes: 6191 22467 32424
> Block Size: 262144b+ 524288b+ 1048576b+
> No. of Reads: 5 3 984
> No. of Writes: 1 0 105
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 126301
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 141880
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 117718
> RELEASEDIR
> 0.01 51.75 us 41.00 us 69.00 us 4
> GETXATTR
> 0.02 59.50 us 35.00 us 108.00 us 4
> STAT
> 0.05 44.06 us 28.00 us 118.00 us 16
> STATFS
> 0.10 24.78 us 15.00 us 99.00 us 59
> FLUSH
> 0.34 41.01 us 24.00 us 107.00 us 118
> ENTRYLK
> 0.37 44.43 us 25.00 us 156.00 us 118
> FINODELK
> 0.76 90.88 us 70.00 us 183.00 us 118
> SETATTR
> 0.81 193.88 us 162.00 us 283.00 us 59
> WRITE
> 0.98 116.89 us 85.00 us 212.00 us 118
> REMOVEXATTR
> 1.10 131.18 us 86.00 us 219.00 us 118
> FXATTROP
> 1.52 44.02 us 24.00 us 1004.00 us 487
> INODELK
> 3.35 86.29 us 52.00 us 183.00 us 549
> LOOKUP
> 4.21 504.03 us 83.00 us 43099.00 us 118
> XATTROP
> 86.39 20696.02 us 207.00 us 69802.00 us 59
> CREATE
> Duration: 644071 seconds
> Data Read: 4837930222 bytes
> Data Written: 7895351133 bytes
> Brick: node6(source):/bricks/brickname
> --------------------------------------------------
> Cumulative Stats:
> Block Size: 1b+ 2b+ 4b+
> No. of Reads: 7 18 89
> No. of Writes: 4 8 37
> Block Size: 8b+ 16b+ 32b+
> No. of Reads: 727 2325 9459
> No. of Writes: 108 54 188
> Block Size: 64b+ 128b+ 256b+
> No. of Reads: 12419 9313 27616
> No. of Writes: 360 85 772
> Block Size: 512b+ 1024b+ 2048b+
> No. of Reads: 23708 28691 18594
> No. of Writes: 847 313 138
> Block Size: 4096b+ 8192b+ 16384b+
> No. of Reads: 19484 12596 8458
> No. of Writes: 29185 79632 1431
> Block Size: 32768b+ 65536b+ 131072b+
> No. of Reads: 5695 5755 1062899
> No. of Writes: 6168 19435 32017
> Block Size: 262144b+ 1048576b+
> No. of Reads: 0 0
> No. of Writes: 1 105
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 13806534
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 17813646
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 223324
> RELEASEDIR
> 0.00 560.00 us 560.00 us 560.00 us 1
> ENTRYLK
> 0.00 3901.00 us 3901.00 us 3901.00 us 1
> SETXATTR
> 0.00 4010.00 us 4010.00 us 4010.00 us 1
> REMOVEXATTR
> 0.01 62446.08 us 8.00 us 365433.00 us 13
> FLUSH
> 0.01 93887.77 us 52.00 us 588566.00 us 13
> SETATTR
> 0.03 10772.83 us 28.00 us 1121761.00 us 253
> GETXATTR
> 0.04 3190096.00 us 3190096.00 us 3190096.00 us 1
> READDIR
> 0.09 558307.69 us 179931.00 us 3188951.00 us 13
> READ
> 0.11 616756.00 us 74.00 us 7307745.00 us 14
> XATTROP
> 0.12 4754785.50 us 48.00 us 9509523.00 us 2
> OPENDIR
> 0.15 1799185.00 us 2310.00 us 5023537.00 us 7
> STATFS
> 0.16 68757.98 us 10.00 us 872148.00 us 189
> FSTAT
> 0.31 143533.93 us 42.00 us 7002195.00 us 174
> OPEN
> 0.40 160262.95 us 661.00 us 2825083.00 us 202
> READDIRP
> 1.55 624450.87 us 31.00 us 7397432.00 us 203
> FXATTROP
> 22.43 212161.62 us 12.00 us 7397413.00 us 8639
> INODELK
> 74.60 541421.09 us 63.00 us 14463033.00 us 11261
> LOOKUP
> Duration: 644071 seconds
> Data Read: 140706386722 bytes
> Data Written: 7549422894 bytes
> Interval 0 Stats:
> Block Size: 1b+ 2b+ 4b+
> No. of Reads: 7 18 89
> No. of Writes: 4 8 37
> Block Size: 8b+ 16b+ 32b+
> No. of Reads: 727 2325 9459
> No. of Writes: 108 54 188
> Block Size: 64b+ 128b+ 256b+
> No. of Reads: 12419 9313 27616
> No. of Writes: 360 85 772
> Block Size: 512b+ 1024b+ 2048b+
> No. of Reads: 23708 28691 18594
> No. of Writes: 847 313 138
> Block Size: 4096b+ 8192b+ 16384b+
> No. of Reads: 19484 12596 8458
> No. of Writes: 29185 79632 1431
> Block Size: 32768b+ 65536b+ 131072b+
> No. of Reads: 5695 5755 1062899
> No. of Writes: 6168 19435 32017
> Block Size: 262144b+ 1048576b+
> No. of Reads: 0 0
> No. of Writes: 1 105
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 13806534
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 17813657
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 223324
> RELEASEDIR
> 0.00 560.00 us 560.00 us 560.00 us 1
> ENTRYLK
> 0.00 3901.00 us 3901.00 us 3901.00 us 1
> SETXATTR
> 0.00 4010.00 us 4010.00 us 4010.00 us 1
> REMOVEXATTR
> 0.01 62446.08 us 8.00 us 365433.00 us 13
> FLUSH
> 0.01 93887.77 us 52.00 us 588566.00 us 13
> SETATTR
> 0.03 10772.83 us 28.00 us 1121761.00 us 253
> GETXATTR
> 0.04 3190096.00 us 3190096.00 us 3190096.00 us 1
> READDIR
> 0.09 558307.69 us 179931.00 us 3188951.00 us 13
> READ
> 0.11 616756.00 us 74.00 us 7307745.00 us 14
> XATTROP
> 0.12 4754785.50 us 48.00 us 9509523.00 us 2
> OPENDIR
> 0.15 1799185.00 us 2310.00 us 5023537.00 us 7
> STATFS
> 0.16 68757.98 us 10.00 us 872148.00 us 189
> FSTAT
> 0.31 143533.93 us 42.00 us 7002195.00 us 174
> OPEN
> 0.40 160262.95 us 661.00 us 2825083.00 us 202
> READDIRP
> 1.55 624450.87 us 31.00 us 7397432.00 us 203
> FXATTROP
> 22.43 212161.62 us 12.00 us 7397413.00 us 8639
> INODELK
> 74.60 541421.09 us 63.00 us 14463033.00 us 11261
> LOOKUP
> Duration: 644071 seconds
> Data Read: 140706386722 bytes
> Data Written: 7549422894 bytes
>
> On Fri, Aug 7, 2015 at 12:17 AM, Ravishankar N <ravishankar at redhat.com
> <mailto:ravishankar at redhat.com>> wrote:
>
>
>
> On 08/07/2015 12:11 PM, Prasun Gera wrote:
>> No, no noticeable difference. Still very high, possibly higher
>> than before.
>
> I was guessing that the cpu usage could be because of the diff
> algorithm which computes checksums (which is a cpu intensive
> task). That doesn't seem to be the case. Could you do a volume
> profile and see the FOPS that are happening on the bricks and
> share the result?
> 1.gluster volume profile <volname> start
> 2. gluster volume profile <volname> info
> 3. wait 10-15 seconds
> 4.gluster volume profile <volname> info
>
>
>
>> The system has come down to a crawl. It's difficult to even ssh
>> or run any commands on the terminal. Do you make anything of the
>> logs ? The brick log is just a giant alternating stream of those
>> two lines I mentioned earlier.
>
>
>>
>> On Thu, Aug 6, 2015 at 10:10 PM, Ravishankar N
>> <ravishankar at redhat.com <mailto:ravishankar at redhat.com>> wrote:
>>
>>
>>
>> On 08/07/2015 01:33 AM, Prasun Gera wrote:
>>
>> I replaced the brick in a node in my 3x2 dist+repl volume
>> (RHS 3). I'm seeing that the heal process, which should
>> essentially be a dump from the working replica to the
>> newly added one is taking exceptionally long. It has
>> moved ~100 G over a day on a 1Gigabit network. The CPU
>> usage on both the nodes of the replica has been pretty high.
>>
>>
>> Does setting `cluster.data-self-heal-algorithm` to full make
>> a difference in the cpu usage?
>>
>>
>> I also think that nagios is making it worse. The heal is
>> slow enough as it is, and nagios keeps triggering heal
>> info, which I think never completes. I also see my logs
>> filling up These are some of the log contents which I got
>> by running tail on them:
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150807/93118fdf/attachment.html>
More information about the Gluster-users
mailing list