[Gluster-users] High CPU/load, High latency on FSYNC
Yong Zhang
hiscal at outlook.com
Fri Mar 31 07:30:27 UTC 2017
Hi, All
I have glusterfs 3.10.0 running with two replicated nodes, but one of the node has a bad CPU performance:
Node1/Node2: 4 core, 8GB, using thin LVM for volumes
This is node1:
[cid:image005.png at 01D2AA33.B41E14D0]
And this is node2:
[cid:image006.png at 01D2AA33.B41E14D0]
Currently I have only Prometheus using glusterfs volumes, so I started profile on volume prometheus and got this:
Brick: 10.10.0.100:/glusterfsvolumes/devops/devops-prometheus/brick
-------------------------------------------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 1 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 1 0
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 4 6 32
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 154 281 484
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 993 15290
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245895 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.00 20.50 us 19.00 us 22.00 us 2 FSYNCDIR
0.01 99.00 us 99.00 us 99.00 us 1 GETXATTR
0.01 68.50 us 48.00 us 89.00 us 2 OPENDIR
0.03 84.33 us 40.00 us 164.00 us 6 FSTAT
0.10 64.35 us 34.00 us 177.00 us 23 STATFS
0.16 62.87 us 29.00 us 172.00 us 38 STAT
0.17 54.76 us 22.00 us 194.00 us 46 ENTRYLK
0.18 41.66 us 24.00 us 95.00 us 62 FLUSH
0.22 198.44 us 138.00 us 339.00 us 16 CREATE
0.32 101.65 us 65.00 us 184.00 us 46 OPEN
0.35 726.43 us 350.00 us 1165.00 us 7 UNLINK
0.52 56.22 us 25.00 us 228.00 us 136 FINODELK
0.77 129.15 us 49.00 us 1138.00 us 87 READ
1.57 154.70 us 59.00 us 360.00 us 148 FXATTROP
2.44 159.13 us 41.00 us 5836.00 us 224 WRITE
3.64 188.62 us 54.00 us 848.00 us 282 LOOKUP
89.50 20741.70 us 9457.00 us 84733.00 us 63 FSYNC
Duration: 362318 seconds
Data Read: 2132763575 bytes
Data Written: 83056358475 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 1 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 1 0
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 4 6 32
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 154 281 484
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 993 15290
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245895 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.00 20.50 us 19.00 us 22.00 us 2 FSYNCDIR
0.01 99.00 us 99.00 us 99.00 us 1 GETXATTR
0.01 68.50 us 48.00 us 89.00 us 2 OPENDIR
0.03 84.33 us 40.00 us 164.00 us 6 FSTAT
0.10 64.35 us 34.00 us 177.00 us 23 STATFS
0.16 62.87 us 29.00 us 172.00 us 38 STAT
0.17 54.76 us 22.00 us 194.00 us 46 ENTRYLK
0.18 41.66 us 24.00 us 95.00 us 62 FLUSH
0.22 198.44 us 138.00 us 339.00 us 16 CREATE
0.32 101.65 us 65.00 us 184.00 us 46 OPEN
0.35 726.43 us 350.00 us 1165.00 us 7 UNLINK
0.52 56.22 us 25.00 us 228.00 us 136 FINODELK
0.77 129.15 us 49.00 us 1138.00 us 87 READ
1.57 154.70 us 59.00 us 360.00 us 148 FXATTROP
2.44 159.13 us 41.00 us 5836.00 us 224 WRITE
3.64 188.62 us 54.00 us 848.00 us 282 LOOKUP
89.50 20741.70 us 9457.00 us 84733.00 us 63 FSYNC
Duration: 362318 seconds
Data Read: 2132763575 bytes
Data Written: 83056358475 bytes
Brick: 10.10.0.101:/glusterfsvolumes/devops/devops-prometheus/brick
-------------------------------------------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 1 3 2
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2 14 22
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 166 253 502
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 893 14392
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245861 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.01 89.00 us 89.00 us 89.00 us 1 GETXATTR
0.02 62.00 us 53.00 us 71.00 us 2 OPENDIR
0.02 57.00 us 51.00 us 61.00 us 3 FSTAT
0.04 178.00 us 38.00 us 318.00 us 2 FSYNCDIR
0.12 474.00 us 451.00 us 497.00 us 2 READDIRP
0.18 64.00 us 28.00 us 113.00 us 23 STATFS
0.28 48.30 us 18.00 us 184.00 us 46 ENTRYLK
0.31 41.33 us 18.00 us 87.00 us 60 FLUSH
0.33 66.12 us 28.00 us 246.00 us 40 STAT
0.57 102.39 us 62.00 us 236.00 us 44 OPEN
0.58 664.86 us 311.00 us 855.00 us 7 UNLINK
0.86 51.37 us 22.00 us 101.00 us 134 FINODELK
1.07 118.28 us 46.00 us 437.00 us 72 READ
3.83 137.39 us 41.00 us 2431.00 us 222 WRITE
4.54 250.94 us 57.00 us 15444.00 us 144 FXATTROP
5.89 2932.56 us 134.00 us 17675.00 us 16 CREATE
7.83 224.34 us 61.00 us 12509.00 us 278 LOOKUP
73.50 9592.56 us 6897.00 us 36321.00 us 61 FSYNC
Duration: 362152 seconds
Data Read: 2005452176 bytes
Data Written: 83056358475 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 1 3 2
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2 14 22
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 166 253 502
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 893 14392
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245861 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.01 89.00 us 89.00 us 89.00 us 1 GETXATTR
0.02 62.00 us 53.00 us 71.00 us 2 OPENDIR
0.02 57.00 us 51.00 us 61.00 us 3 FSTAT
0.04 178.00 us 38.00 us 318.00 us 2 FSYNCDIR
0.12 474.00 us 451.00 us 497.00 us 2 READDIRP
0.18 64.00 us 28.00 us 113.00 us 23 STATFS
0.28 48.30 us 18.00 us 184.00 us 46 ENTRYLK
0.31 41.33 us 18.00 us 87.00 us 60 FLUSH
0.33 66.12 us 28.00 us 246.00 us 40 STAT
0.57 102.39 us 62.00 us 236.00 us 44 OPEN
0.58 664.86 us 311.00 us 855.00 us 7 UNLINK
0.86 51.37 us 22.00 us 101.00 us 134 FINODELK
1.07 118.28 us 46.00 us 437.00 us 72 READ
3.83 137.39 us 41.00 us 2431.00 us 222 WRITE
4.54 250.94 us 57.00 us 15444.00 us 144 FXATTROP
5.89 2932.56 us 134.00 us 17675.00 us 16 CREATE
7.83 224.34 us 61.00 us 12509.00 us 278 LOOKUP
73.50 9592.56 us 6897.00 us 36321.00 us 61 FSYNC
Duration: 362152 seconds
Data Read: 2005452176 bytes
Data Written: 83056358475 bytes
Seems latency of FSYNC is very high. Sometimes cpu wait is also very high:
top - 15:27:39 up 4 days, 4:58, 2 users, load average: 0.75, 0.72, 0.70
Tasks: 210 total, 1 running, 209 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 0.2 sy, 0.0 ni, 62.9 id, 36.5 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 8043984 total, 883420 free, 1788912 used, 5371652 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 5819396 avail Mem
Any ideas?
Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170331/455c9152/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 9491E54ADE8B47A088284EAA75AEEC31.png
Type: image/png
Size: 96159 bytes
Desc: 9491E54ADE8B47A088284EAA75AEEC31.png
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170331/455c9152/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1AF5C885C5A5497196A56960EC888E2E.png
Type: image/png
Size: 57197 bytes
Desc: 1AF5C885C5A5497196A56960EC888E2E.png
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170331/455c9152/attachment-0001.png>
More information about the Gluster-users
mailing list