[Gluster-users] Self heal issues
Prasun Gera
prasun.gera at gmail.com
Fri Aug 7 20:37:12 UTC 2015
So I let the heal complete, and it sped up later on. The total data that
needed to be transferred to the brick was about 400G. It took about 2.5
days to finish. However, most of the time was spent in transferring a few
GBs. Once it was through the rough patch, the rest of it transferred at
acceptable speeds. That also corelates with the errors in the brick logs.
It was really slow with high CPU usage when those errors were thrown in the
brick log. Later one, the errors went away and the speed also became
normal. Each brick is 1.8 TB. All the nodes have 2 TB SATA hard drives with
200GB reserved for OS, and rest as bricks. Some of the systems are old with
low memory (4 GB). Not sure if that played a part in the heal. I did see
spikes for kswapd0 when the CPU was high. The usage is a regular file
server with most files ranging in the KBs to low MBs range. The network is
a stock gigabit network without any tweaks for bonding, MTU etc. I can
generate more specific stats if there are commands.
On Fri, Aug 7, 2015 at 3:04 AM, Ravishankar N <ravishankar at redhat.com>
wrote:
> So the nodes 3 and 6 seem to indicate inode-locks and lookups are of the
> highest latency. This only seems to confirm self-heals are happening.
> If you are unable to use the system because of this, you could try killing
> the self-heal daemons on both these nodes (kill `pgrep -f glustershd`) to
> stop heals. You can then do a lookup of the files from the mount, which
> will also trigger heals.
> Restart the selfheal daemons (with `gluster vol start volname force`)
> when you think you can spare the volume for heals again. The sooner the
> better though.
> For the brick log errors, we are suspecting it could be something related
> to selinux.
>
> Can you tell what kind of data is there in your volume? - no. of files,
> avg. file size, brick size, network connection speed etc? Perhaps we can
> try to reproduce the issue and identify the bottle neck.
>
> Thanks,
> Ravi
>
>
> On 08/07/2015 01:27 PM, Prasun Gera wrote:
>
> All the volume commands are taking several minutes to complete. Here's the
> profiler's output:
> Node3's brick is the one that was replaced. It's replica is node6
>
>
> Brick: node1:/bricks/brickname
> ---------------------------------------------------
> Cumulative Stats:
> Block Size: 8b+ 16b+
> 32b+
> No. of Reads: 0 13
> 2
> No. of Writes: 60 0
> 141
>
> Block Size: 64b+ 128b+
> 256b+
> No. of Reads: 3 26
> 17
> No. of Writes: 325 87
> 738
>
> Block Size: 512b+ 1024b+
> 2048b+
> No. of Reads: 99 114
> 222
> No. of Writes: 877 343
> 128
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 110 5
> 2
> No. of Writes: 29401 78829
> 1448
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 19 33
> 34679
> No. of Writes: 6233 22903
> 41202
>
> Block Size: 262144b+ 524288b+
> 1048576b+
> No. of Reads: 0 1
> 513
> No. of Writes: 1 0
> 105
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 126138
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 141671
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 117718
> RELEASEDIR
> 0.02 43.00 us 21.00 us 80.00 us 17
> STAT
> 0.02 51.62 us 27.00 us 131.00 us 21
> STATFS
> 0.09 42.78 us 11.00 us 1640.00 us 95
> FLUSH
> 0.21 50.01 us 23.00 us 567.00 us 189
> ENTRYLK
> 0.21 50.01 us 19.00 us 291.00 us 190
> FINODELK
> 0.56 8578.33 us 53.00 us 25625.00 us 3
> GETXATTR
> 0.62 148.89 us 71.00 us 2761.00 us 190
> XATTROP
> 0.70 168.11 us 83.00 us 1019.00 us 190
> FXATTROP
> 0.91 219.84 us 47.00 us 13732.00 us 190
> SETATTR
> 1.21 71.74 us 16.00 us 11516.00 us 775
> INODELK
> 1.47 354.88 us 56.00 us 22669.00 us 190
> REMOVEXATTR
> 2.60 1254.69 us 122.00 us 12514.00 us 95
> WRITE
> 3.25 194.10 us 51.00 us 48823.00 us 770
> LOOKUP
> 88.15 43068.81 us 265.00 us 418819.00 us 94
> CREATE
>
> Duration: 644070 seconds
> Data Read: 5089537361 bytes
> Data Written: 9083513756 bytes
>
> Interval 0 Stats:
> Block Size: 8b+ 16b+
> 32b+
> No. of Reads: 0 13
> 2
> No. of Writes: 60 0
> 141
>
> Block Size: 64b+ 128b+
> 256b+
> No. of Reads: 3 26
> 17
> No. of Writes: 325 87
> 738
>
> Block Size: 512b+ 1024b+
> 2048b+
> No. of Reads: 99 114
> 222
> No. of Writes: 877 343
> 128
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 110 5
> 2
> No. of Writes: 29401 78829
> 1448
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 19 33
> 34679
> No. of Writes: 6233 22903
> 41202
>
> Block Size: 262144b+ 524288b+
> 1048576b+
> No. of Reads: 0 1
> 513
> No. of Writes: 1 0
> 105
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 126138
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 141671
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 117718
> RELEASEDIR
> 0.02 43.00 us 21.00 us 80.00 us 17
> STAT
> 0.02 51.62 us 27.00 us 131.00 us 21
> STATFS
> 0.09 42.78 us 11.00 us 1640.00 us 95
> FLUSH
> 0.21 50.01 us 23.00 us 567.00 us 189
> ENTRYLK
> 0.21 50.01 us 19.00 us 291.00 us 190
> FINODELK
> 0.56 8578.33 us 53.00 us 25625.00 us 3
> GETXATTR
> 0.62 148.89 us 71.00 us 2761.00 us 190
> XATTROP
> 0.70 168.11 us 83.00 us 1019.00 us 190
> FXATTROP
> 0.91 219.84 us 47.00 us 13732.00 us 190
> SETATTR
> 1.21 71.74 us 16.00 us 11516.00 us 775
> INODELK
> 1.47 354.88 us 56.00 us 22669.00 us 190
> REMOVEXATTR
> 2.60 1254.69 us 122.00 us 12514.00 us 95
> WRITE
> 3.25 194.10 us 51.00 us 48823.00 us 770
> LOOKUP
> 88.15 43068.81 us 265.00 us 418819.00 us 94
> CREATE
>
> Duration: 644070 seconds
> Data Read: 5089537361 bytes
> Data Written: 9083513756 bytes
>
> Brick: node2:/bricks/brickname
> -----------------------------------------------
> Cumulative Stats:
> Block Size: 8b+ 16b+
> 32b+
> No. of Reads: 0 1
> 23
> No. of Writes: 60 0
> 141
>
> Block Size: 64b+ 128b+
> 256b+
> No. of Reads: 45 47
> 363
> No. of Writes: 325 87
> 738
>
> Block Size: 512b+ 1024b+
> 2048b+
> No. of Reads: 515 37
> 42
> No. of Writes: 877 343
> 128
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 17 0
> 1
> No. of Writes: 29401 78829
> 1448
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 15 39
> 67031
> No. of Writes: 6233 22903
> 41202
>
> Block Size: 262144b+ 1048576b+
> No. of Reads: 1 105
> No. of Writes: 1 105
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 126136
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 141671
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 117718
> RELEASEDIR
> 0.02 74.00 us 74.00 us 74.00 us 1
> STAT
> 0.15 196.00 us 141.00 us 288.00 us 3
> GETXATTR
> 0.59 105.23 us 44.00 us 146.00 us 22
> STATFS
> 2.05 83.88 us 11.00 us 137.00 us 96
> FLUSH
> 4.79 98.61 us 20.00 us 146.00 us 191
> ENTRYLK
> 5.03 102.93 us 22.00 us 158.00 us 192
> FINODELK
> 5.52 226.07 us 136.00 us 295.00 us 96
> WRITE
> 6.31 261.08 us 150.00 us 345.00 us 95
> CREATE
> 6.72 137.53 us 50.00 us 214.00 us 192
> SETATTR
> 7.68 157.14 us 75.00 us 237.00 us 192
> REMOVEXATTR
> 8.13 166.49 us 81.00 us 282.00 us 192
> XATTROP
> 8.26 169.01 us 76.00 us 275.00 us 192
> FXATTROP
> 17.22 86.46 us 16.00 us 216.00 us 783
> INODELK
> 27.54 138.09 us 43.00 us 266.00 us 784
> LOOKUP
>
> Duration: 644071 seconds
> Data Read: 8902589511 bytes
> Data Written: 9083513756 bytes
>
> Interval 0 Stats:
> Block Size: 8b+ 16b+
> 32b+
> No. of Reads: 0 1
> 23
> No. of Writes: 60 0
> 141
>
> Block Size: 64b+ 128b+
> 256b+
> No. of Reads: 45 47
> 363
> No. of Writes: 325 87
> 738
>
> Block Size: 512b+ 1024b+
> 2048b+
> No. of Reads: 515 37
> 42
> No. of Writes: 877 343
> 128
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 17 0
> 1
> No. of Writes: 29401 78829
> 1448
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 15 39
> 67031
> No. of Writes: 6233 22903
> 41202
>
> Block Size: 262144b+ 1048576b+
> No. of Reads: 1 105
> No. of Writes: 1 105
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 126136
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 141671
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 117718
> RELEASEDIR
> 0.02 74.00 us 74.00 us 74.00 us 1
> STAT
> 0.15 196.00 us 141.00 us 288.00 us 3
> GETXATTR
> 0.59 105.23 us 44.00 us 146.00 us 22
> STATFS
> 2.05 83.88 us 11.00 us 137.00 us 96
> FLUSH
> 4.79 98.61 us 20.00 us 146.00 us 191
> ENTRYLK
> 5.03 102.93 us 22.00 us 158.00 us 192
> FINODELK
> 5.52 226.07 us 136.00 us 295.00 us 96
> WRITE
> 6.31 261.08 us 150.00 us 345.00 us 95
> CREATE
> 6.72 137.53 us 50.00 us 214.00 us 192
> SETATTR
> 7.68 157.14 us 75.00 us 237.00 us 192
> REMOVEXATTR
> 8.13 166.49 us 81.00 us 282.00 us 192
> XATTROP
> 8.26 169.01 us 76.00 us 275.00 us 192
> FXATTROP
> 17.22 86.46 us 16.00 us 216.00 us 783
> INODELK
> 27.54 138.09 us 43.00 us 266.00 us 784
> LOOKUP
>
> Duration: 644071 seconds
> Data Read: 8902589511 bytes
> Data Written: 9083513756 bytes
>
> Brick: node3(sink):/bricks/brickname
> ------------------------------------------------
> Cumulative Stats:
> Block Size: 1b+ 2b+
> 4b+
> No. of Reads: 0 0
> 0
> No. of Writes: 11 26
> 125
>
> Block Size: 8b+ 16b+
> 32b+
> No. of Reads: 0 0
> 0
> No. of Writes: 829 2341
> 9599
>
> Block Size: 64b+ 128b+
> 256b+
> No. of Reads: 0 0
> 0
> No. of Writes: 12674 9229
> 27346
>
> Block Size: 512b+ 1024b+
> 2048b+
> No. of Reads: 2 10
> 0
> No. of Writes: 23414 28727
> 18372
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 1 0
> 0
> No. of Writes: 48347 92134
> 9675
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 2 11
> 50
> No. of Writes: 11717 24948
> 1022216
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 13805186
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 17674891
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 218068
> RELEASEDIR
> 0.00 24.00 us 24.00 us 24.00 us 1
> OPENDIR
> 0.00 13.06 us 8.00 us 36.00 us 16
> STAT
> 0.00 19.22 us 9.00 us 46.00 us 18
> STATFS
> 0.00 45.54 us 22.00 us 82.00 us 13
> SETXATTR
> 0.00 120.93 us 77.00 us 156.00 us 14
> XATTROP
> 0.00 11.01 us 7.00 us 68.00 us 156
> ENTRYLK
> 0.01 283.15 us 246.00 us 504.00 us 59
> READDIR
> 0.02 899.19 us 39.00 us 17518.00 us 26
> SETATTR
> 0.02 2004.85 us 38.00 us 10406.00 us 13
> WRITE
> 0.02 2022.77 us 24.00 us 21677.00 us 13
> REMOVEXATTR
> 0.03 2965.85 us 34.00 us 37695.00 us 13
> FTRUNCATE
> 0.04 3691.62 us 31.00 us 18386.00 us 13
> FLUSH
> 0.31 2105.65 us 23.00 us 57417.00 us 177
> OPEN
> 0.43 2603.12 us 57.00 us 73929.00 us 202
> FXATTROP
> 0.46 3030.94 us 7.00 us 87892.00 us 186
> FSTAT
> 1.07 33.17 us 18.00 us 17545.00 us 39491
> GETXATTR
> 1.31 123033.46 us 75610.00 us 269227.00 us 13
> FSYNC
> 46.14 704.24 us 6.00 us 268597.00 us 79866
> INODELK
> 50.13 699.93 us 20.00 us 267607.00 us 87307
> LOOKUP
>
> Duration: 112674 seconds
> Data Read: 7441454 bytes
> Data Written: 138577629032 bytes
>
> Interval 0 Stats:
> Block Size: 1b+ 2b+
> 4b+
> No. of Reads: 0 0
> 0
> No. of Writes: 11 26
> 125
>
> Block Size: 8b+ 16b+
> 32b+
> No. of Reads: 0 0
> 0
> No. of Writes: 829 2341
> 9599
>
> Block Size: 64b+ 128b+
> 256b+
> No. of Reads: 0 0
> 0
> No. of Writes: 12674 9229
> 27346
>
> Block Size: 512b+ 1024b+
> 2048b+
> No. of Reads: 2 10
> 0
> No. of Writes: 23414 28727
> 18372
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 1 0
> 0
> No. of Writes: 48347 92134
> 9675
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 2 11
> 50
> No. of Writes: 11717 24948
> 1022216
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 13805186
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 17674862
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 218068
> RELEASEDIR
> 0.00 24.00 us 24.00 us 24.00 us 1
> OPENDIR
> 0.00 13.06 us 8.00 us 36.00 us 16
> STAT
> 0.00 19.22 us 9.00 us 46.00 us 18
> STATFS
> 0.00 45.54 us 22.00 us 82.00 us 13
> SETXATTR
> 0.00 120.93 us 77.00 us 156.00 us 14
> XATTROP
> 0.00 11.01 us 7.00 us 68.00 us 156
> ENTRYLK
> 0.01 283.15 us 246.00 us 504.00 us 59
> READDIR
> 0.02 899.19 us 39.00 us 17518.00 us 26
> SETATTR
> 0.02 2004.85 us 38.00 us 10406.00 us 13
> WRITE
> 0.02 2022.77 us 24.00 us 21677.00 us 13
> REMOVEXATTR
> 0.03 2965.85 us 34.00 us 37695.00 us 13
> FTRUNCATE
> 0.04 3691.62 us 31.00 us 18386.00 us 13
> FLUSH
> 0.31 2105.65 us 23.00 us 57417.00 us 177
> OPEN
> 0.43 2603.12 us 57.00 us 73929.00 us 202
> FXATTROP
> 0.46 3030.94 us 7.00 us 87892.00 us 186
> FSTAT
> 1.07 33.17 us 18.00 us 17545.00 us 39491
> GETXATTR
> 1.31 123033.46 us 75610.00 us 269227.00 us 13
> FSYNC
> 46.14 704.24 us 6.00 us 268597.00 us 79866
> INODELK
> 50.13 699.93 us 20.00 us 267607.00 us 87307
> LOOKUP
>
> Duration: 112674 seconds
> Data Read: 7441454 bytes
> Data Written: 138577629032 bytes
>
> Brick: node4:/bricks/brickname
> -----------------------------------------------
> Cumulative Stats:
> Block Size: 8b+ 32b+
> 64b+
> No. of Reads: 0 9
> 24
> No. of Writes: 62 128
> 335
>
> Block Size: 128b+ 256b+
> 512b+
> No. of Reads: 21 177
> 257
> No. of Writes: 186 779
> 885
>
> Block Size: 1024b+ 2048b+
> 4096b+
> No. of Reads: 30 14
> 7
> No. of Writes: 286 101
> 29410
>
> Block Size: 8192b+ 16384b+
> 32768b+
> No. of Reads: 0 9
> 0
> No. of Writes: 79662 1379
> 6187
>
> Block Size: 65536b+ 131072b+
> 262144b+
> No. of Reads: 29 3924
> 0
> No. of Writes: 22467 32424
> 1
>
> Block Size: 1048576b+
> No. of Reads: 0
> No. of Writes: 105
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 126295
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 141875
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 117220
> RELEASEDIR
> 0.12 119.50 us 102.00 us 147.00 us 4
> GETXATTR
> 0.19 68.18 us 42.00 us 109.00 us 11
> STAT
> 0.44 92.16 us 19.00 us 141.00 us 19
> STATFS
> 1.20 68.03 us 11.00 us 120.00 us 71
> FLUSH
> 2.93 83.02 us 18.00 us 136.00 us 142
> ENTRYLK
> 3.15 89.18 us 16.00 us 160.00 us 142
> FINODELK
> 3.40 192.82 us 76.00 us 271.00 us 71
> WRITE
> 4.04 114.43 us 35.00 us 204.00 us 142
> SETATTR
> 4.87 138.05 us 49.00 us 222.00 us 142
> REMOVEXATTR
> 5.63 159.52 us 56.00 us 262.00 us 142
> FXATTROP
> 10.68 73.81 us 11.00 us 202.00 us 582
> INODELK
> 19.90 1127.35 us 116.00 us 27717.00 us 71
> CREATE
> 21.67 613.82 us 46.00 us 65260.00 us 142
> XATTROP
> 21.80 130.68 us 35.00 us 241.00 us 671
> LOOKUP
>
> Duration: 458509 seconds
> Data Read: 517180943 bytes
> Data Written: 7895152670 bytes
>
> Interval 0 Stats:
> Block Size: 8b+ 32b+
> 64b+
> No. of Reads: 0 9
> 24
> No. of Writes: 62 128
> 335
>
> Block Size: 128b+ 256b+
> 512b+
> No. of Reads: 21 177
> 257
> No. of Writes: 186 779
> 885
>
> Block Size: 1024b+ 2048b+
> 4096b+
> No. of Reads: 30 14
> 7
> No. of Writes: 286 101
> 29410
>
> Block Size: 8192b+ 16384b+
> 32768b+
> No. of Reads: 0 9
> 0
> No. of Writes: 79662 1379
> 6187
>
> Block Size: 65536b+ 131072b+
> 262144b+
> No. of Reads: 29 3924
> 0
> No. of Writes: 22467 32424
> 1
>
> Block Size: 1048576b+
> No. of Reads: 0
> No. of Writes: 105
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 126295
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 141875
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 117220
> RELEASEDIR
> 0.12 119.50 us 102.00 us 147.00 us 4
> GETXATTR
> 0.19 68.18 us 42.00 us 109.00 us 11
> STAT
> 0.44 92.16 us 19.00 us 141.00 us 19
> STATFS
> 1.20 68.03 us 11.00 us 120.00 us 71
> FLUSH
> 2.93 83.02 us 18.00 us 136.00 us 142
> ENTRYLK
> 3.15 89.18 us 16.00 us 160.00 us 142
> FINODELK
> 3.40 192.82 us 76.00 us 271.00 us 71
> WRITE
> 4.04 114.43 us 35.00 us 204.00 us 142
> SETATTR
> 4.87 138.05 us 49.00 us 222.00 us 142
> REMOVEXATTR
> 5.63 159.52 us 56.00 us 262.00 us 142
> FXATTROP
> 10.68 73.81 us 11.00 us 202.00 us 582
> INODELK
> 19.90 1127.35 us 116.00 us 27717.00 us 71
> CREATE
> 21.67 613.82 us 46.00 us 65260.00 us 142
> XATTROP
> 21.80 130.68 us 35.00 us 241.00 us 671
> LOOKUP
>
> Duration: 458509 seconds
> Data Read: 517180943 bytes
> Data Written: 7895152670 bytes
>
> Brick: node5:/bricks/brickname
> ------------------------------------------------
> Cumulative Stats:
> Block Size: 8b+ 16b+
> 32b+
> No. of Reads: 0 14
> 9
> No. of Writes: 62 0
> 128
>
> Block Size: 64b+ 128b+
> 256b+
> No. of Reads: 23 56
> 225
> No. of Writes: 335 186
> 779
>
> Block Size: 512b+ 1024b+
> 2048b+
> No. of Reads: 357 106
> 233
> No. of Writes: 885 286
> 102
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 128 11
> 15
> No. of Writes: 29410 79662
> 1379
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 16 34
> 28965
> No. of Writes: 6191 22467
> 32424
>
> Block Size: 262144b+ 524288b+
> 1048576b+
> No. of Reads: 5 3
> 984
> No. of Writes: 1 0
> 105
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 126301
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 141880
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 117718
> RELEASEDIR
> 0.01 51.75 us 41.00 us 69.00 us 4
> GETXATTR
> 0.02 59.50 us 35.00 us 108.00 us 4
> STAT
> 0.05 44.06 us 28.00 us 118.00 us 16
> STATFS
> 0.10 24.78 us 15.00 us 99.00 us 59
> FLUSH
> 0.34 41.01 us 24.00 us 107.00 us 118
> ENTRYLK
> 0.37 44.43 us 25.00 us 156.00 us 118
> FINODELK
> 0.76 90.88 us 70.00 us 183.00 us 118
> SETATTR
> 0.81 193.88 us 162.00 us 283.00 us 59
> WRITE
> 0.98 116.89 us 85.00 us 212.00 us 118
> REMOVEXATTR
> 1.10 131.18 us 86.00 us 219.00 us 118
> FXATTROP
> 1.52 44.02 us 24.00 us 1004.00 us 487
> INODELK
> 3.35 86.29 us 52.00 us 183.00 us 549
> LOOKUP
> 4.21 504.03 us 83.00 us 43099.00 us 118
> XATTROP
> 86.39 20696.02 us 207.00 us 69802.00 us 59
> CREATE
>
> Duration: 644071 seconds
> Data Read: 4837930222 bytes
> Data Written: 7895351133 bytes
>
> Interval 0 Stats:
> Block Size: 8b+ 16b+
> 32b+
> No. of Reads: 0 14
> 9
> No. of Writes: 62 0
> 128
>
> Block Size: 64b+ 128b+
> 256b+
> No. of Reads: 23 56
> 225
> No. of Writes: 335 186
> 779
>
> Block Size: 512b+ 1024b+
> 2048b+
> No. of Reads: 357 106
> 233
> No. of Writes: 885 286
> 102
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 128 11
> 15
> No. of Writes: 29410 79662
> 1379
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 16 34
> 28965
> No. of Writes: 6191 22467
> 32424
>
> Block Size: 262144b+ 524288b+
> 1048576b+
> No. of Reads: 5 3
> 984
> No. of Writes: 1 0
> 105
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 126301
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 141880
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 117718
> RELEASEDIR
> 0.01 51.75 us 41.00 us 69.00 us 4
> GETXATTR
> 0.02 59.50 us 35.00 us 108.00 us 4
> STAT
> 0.05 44.06 us 28.00 us 118.00 us 16
> STATFS
> 0.10 24.78 us 15.00 us 99.00 us 59
> FLUSH
> 0.34 41.01 us 24.00 us 107.00 us 118
> ENTRYLK
> 0.37 44.43 us 25.00 us 156.00 us 118
> FINODELK
> 0.76 90.88 us 70.00 us 183.00 us 118
> SETATTR
> 0.81 193.88 us 162.00 us 283.00 us 59
> WRITE
> 0.98 116.89 us 85.00 us 212.00 us 118
> REMOVEXATTR
> 1.10 131.18 us 86.00 us 219.00 us 118
> FXATTROP
> 1.52 44.02 us 24.00 us 1004.00 us 487
> INODELK
> 3.35 86.29 us 52.00 us 183.00 us 549
> LOOKUP
> 4.21 504.03 us 83.00 us 43099.00 us 118
> XATTROP
> 86.39 20696.02 us 207.00 us 69802.00 us 59
> CREATE
>
> Duration: 644071 seconds
> Data Read: 4837930222 bytes
> Data Written: 7895351133 bytes
>
> Brick: node6(source):/bricks/brickname
> --------------------------------------------------
> Cumulative Stats:
> Block Size: 1b+ 2b+
> 4b+
> No. of Reads: 7 18
> 89
> No. of Writes: 4 8
> 37
>
> Block Size: 8b+ 16b+
> 32b+
> No. of Reads: 727 2325
> 9459
> No. of Writes: 108 54
> 188
>
> Block Size: 64b+ 128b+
> 256b+
> No. of Reads: 12419 9313
> 27616
> No. of Writes: 360 85
> 772
>
> Block Size: 512b+ 1024b+
> 2048b+
> No. of Reads: 23708 28691
> 18594
> No. of Writes: 847 313
> 138
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 19484 12596
> 8458
> No. of Writes: 29185 79632
> 1431
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 5695 5755
> 1062899
> No. of Writes: 6168 19435
> 32017
>
> Block Size: 262144b+ 1048576b+
> No. of Reads: 0 0
> No. of Writes: 1 105
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 13806534
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 17813646
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 223324
> RELEASEDIR
> 0.00 560.00 us 560.00 us 560.00 us 1
> ENTRYLK
> 0.00 3901.00 us 3901.00 us 3901.00 us 1
> SETXATTR
> 0.00 4010.00 us 4010.00 us 4010.00 us 1
> REMOVEXATTR
> 0.01 62446.08 us 8.00 us 365433.00 us 13
> FLUSH
> 0.01 93887.77 us 52.00 us 588566.00 us 13
> SETATTR
> 0.03 10772.83 us 28.00 us 1121761.00 us 253
> GETXATTR
> 0.04 3190096.00 us 3190096.00 us 3190096.00 us 1
> READDIR
> 0.09 558307.69 us 179931.00 us 3188951.00 us 13
> READ
> 0.11 616756.00 us 74.00 us 7307745.00 us 14
> XATTROP
> 0.12 4754785.50 us 48.00 us 9509523.00 us 2
> OPENDIR
> 0.15 1799185.00 us 2310.00 us 5023537.00 us 7
> STATFS
> 0.16 68757.98 us 10.00 us 872148.00 us 189
> FSTAT
> 0.31 143533.93 us 42.00 us 7002195.00 us 174
> OPEN
> 0.40 160262.95 us 661.00 us 2825083.00 us 202
> READDIRP
> 1.55 624450.87 us 31.00 us 7397432.00 us 203
> FXATTROP
> 22.43 212161.62 us 12.00 us 7397413.00 us 8639
> INODELK
> 74.60 541421.09 us 63.00 us 14463033.00 us 11261
> LOOKUP
>
> Duration: 644071 seconds
> Data Read: 140706386722 bytes
> Data Written: 7549422894 bytes
>
> Interval 0 Stats:
> Block Size: 1b+ 2b+
> 4b+
> No. of Reads: 7 18
> 89
> No. of Writes: 4 8
> 37
>
> Block Size: 8b+ 16b+
> 32b+
> No. of Reads: 727 2325
> 9459
> No. of Writes: 108 54
> 188
>
> Block Size: 64b+ 128b+
> 256b+
> No. of Reads: 12419 9313
> 27616
> No. of Writes: 360 85
> 772
>
> Block Size: 512b+ 1024b+
> 2048b+
> No. of Reads: 23708 28691
> 18594
> No. of Writes: 847 313
> 138
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 19484 12596
> 8458
> No. of Writes: 29185 79632
> 1431
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 5695 5755
> 1062899
> No. of Writes: 6168 19435
> 32017
>
> Block Size: 262144b+ 1048576b+
> No. of Reads: 0 0
> No. of Writes: 1 105
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 13806534
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 17813657
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 223324
> RELEASEDIR
> 0.00 560.00 us 560.00 us 560.00 us 1
> ENTRYLK
> 0.00 3901.00 us 3901.00 us 3901.00 us 1
> SETXATTR
> 0.00 4010.00 us 4010.00 us 4010.00 us 1
> REMOVEXATTR
> 0.01 62446.08 us 8.00 us 365433.00 us 13
> FLUSH
> 0.01 93887.77 us 52.00 us 588566.00 us 13
> SETATTR
> 0.03 10772.83 us 28.00 us 1121761.00 us 253
> GETXATTR
> 0.04 3190096.00 us 3190096.00 us 3190096.00 us 1
> READDIR
> 0.09 558307.69 us 179931.00 us 3188951.00 us 13
> READ
> 0.11 616756.00 us 74.00 us 7307745.00 us 14
> XATTROP
> 0.12 4754785.50 us 48.00 us 9509523.00 us 2
> OPENDIR
> 0.15 1799185.00 us 2310.00 us 5023537.00 us 7
> STATFS
> 0.16 68757.98 us 10.00 us 872148.00 us 189
> FSTAT
> 0.31 143533.93 us 42.00 us 7002195.00 us 174
> OPEN
> 0.40 160262.95 us 661.00 us 2825083.00 us 202
> READDIRP
> 1.55 624450.87 us 31.00 us 7397432.00 us 203
> FXATTROP
> 22.43 212161.62 us 12.00 us 7397413.00 us 8639
> INODELK
> 74.60 541421.09 us 63.00 us 14463033.00 us 11261
> LOOKUP
>
> Duration: 644071 seconds
> Data Read: 140706386722 bytes
> Data Written: 7549422894 bytes
>
>
> On Fri, Aug 7, 2015 at 12:17 AM, Ravishankar N <ravishankar at redhat.com>
> wrote:
>
>>
>>
>> On 08/07/2015 12:11 PM, Prasun Gera wrote:
>>
>> No, no noticeable difference. Still very high, possibly higher than
>> before.
>>
>>
>> I was guessing that the cpu usage could be because of the diff algorithm
>> which computes checksums (which is a cpu intensive task). That doesn't
>> seem to be the case. Could you do a volume profile and see the FOPS that
>> are happening on the bricks and share the result?
>> 1.gluster volume profile <volname> start
>> 2. gluster volume profile <volname> info
>> 3. wait 10-15 seconds
>> 4.gluster volume profile <volname> info
>>
>>
>>
>> The system has come down to a crawl. It's difficult to even ssh or run
>> any commands on the terminal. Do you make anything of the logs ? The brick
>> log is just a giant alternating stream of those two lines I mentioned
>> earlier.
>>
>>
>>
>>
>> On Thu, Aug 6, 2015 at 10:10 PM, Ravishankar N < <ravishankar at redhat.com>
>> ravishankar at redhat.com> wrote:
>>
>>>
>>>
>>> On 08/07/2015 01:33 AM, Prasun Gera wrote:
>>>
>>>> I replaced the brick in a node in my 3x2 dist+repl volume (RHS 3). I'm
>>>> seeing that the heal process, which should essentially be a dump from the
>>>> working replica to the newly added one is taking exceptionally long. It has
>>>> moved ~100 G over a day on a 1Gigabit network. The CPU usage on both the
>>>> nodes of the replica has been pretty high.
>>>>
>>>
>>> Does setting `cluster.data-self-heal-algorithm` to full make a
>>> difference in the cpu usage?
>>>
>>>
>>> I also think that nagios is making it worse. The heal is slow enough as
>>>> it is, and nagios keeps triggering heal info, which I think never
>>>> completes. I also see my logs filling up These are some of the log contents
>>>> which I got by running tail on them:
>>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150807/8b6b3829/attachment.html>
More information about the Gluster-users
mailing list