[Gluster-users] High CPU/load, High latency on FSYNC
Yong Zhang
hiscal at outlook.com
Sat Apr 1 00:58:57 UTC 2017
Hi, Serkan
I’m using Glusterfs as my Kubernetes storage backend, and Prometheus is running inside Kubernetes cluster. If this is not recommended I’ll consider running Prometheus outside of Kubernetes. But why performance is totally different on two nodes? Will data be written to both nodes at the same time?
From: Serkan Çoban<mailto:cobanserkan at gmail.com>
Sent: Friday, March 31, 2017 4:08 PM
To: Yong Zhang<mailto:hiscal at outlook.com>
Cc: gluster-users at gluster.org<mailto:gluster-users at gluster.org>
Subject: Re: [Gluster-users] High CPU/load, High latency on FSYNC
Prometheus uses *LOTS* of small files and do small random-io.
I do not recommend to use data volume of prometheus on glusterfs.
Why you need that?
On Fri, Mar 31, 2017 at 10:30 AM, Yong Zhang <hiscal at outlook.com<mailto:hiscal at outlook.com>> wrote:
Hi, All
I have glusterfs 3.10.0 running with two replicated nodes, but one of the node has a bad CPU performance:
Node1/Node2: 4 core, 8GB, using thin LVM for volumes
This is node1:
[cid:image005.png at 01D2AA33.B41E14D0]
And this is node2:
[cid:image006.png at 01D2AA33.B41E14D0]
Currently I have only Prometheus using glusterfs volumes, so I started profile on volume prometheus and got this:
Brick: 10.10.0.100:/glusterfsvolumes/devops/devops-prometheus/brick
-------------------------------------------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 1 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 1 0
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 4 6 32
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 154 281 484
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 993 15290
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245895 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.00 20.50 us 19.00 us 22.00 us 2 FSYNCDIR
0.01 99.00 us 99.00 us 99.00 us 1 GETXATTR
0.01 68.50 us 48.00 us 89.00 us 2 OPENDIR
0.03 84.33 us 40.00 us 164.00 us 6 FSTAT
0.10 64.35 us 34.00 us 177.00 us 23 STATFS
0.16 62.87 us 29.00 us 172.00 us 38 STAT
0.17 54.76 us 22.00 us 194.00 us 46 ENTRYLK
0.18 41.66 us 24.00 us 95.00 us 62 FLUSH
0.22 198.44 us 138.00 us 339.00 us 16 CREATE
0.32 101.65 us 65.00 us 184.00 us 46 OPEN
0.35 726.43 us 350.00 us 1165.00 us 7 UNLINK
0.52 56.22 us 25.00 us 228.00 us 136 FINODELK
0.77 129.15 us 49.00 us 1138.00 us 87 READ
1.57 154.70 us 59.00 us 360.00 us 148 FXATTROP
2.44 159.13 us 41.00 us 5836.00 us 224 WRITE
3.64 188.62 us 54.00 us 848.00 us 282 LOOKUP
89.50 20741.70 us 9457.00 us 84733.00 us 63 FSYNC
Duration: 362318 seconds
Data Read: 2132763575<tel:(213)%20276-3575> bytes
Data Written: 83056358475 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 1 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 1 0
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 4 6 32
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 154 281 484
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 993 15290
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245895 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.00 20.50 us 19.00 us 22.00 us 2 FSYNCDIR
0.01 99.00 us 99.00 us 99.00 us 1 GETXATTR
0.01 68.50 us 48.00 us 89.00 us 2 OPENDIR
0.03 84.33 us 40.00 us 164.00 us 6 FSTAT
0.10 64.35 us 34.00 us 177.00 us 23 STATFS
0.16 62.87 us 29.00 us 172.00 us 38 STAT
0.17 54.76 us 22.00 us 194.00 us 46 ENTRYLK
0.18 41.66 us 24.00 us 95.00 us 62 FLUSH
0.22 198.44 us 138.00 us 339.00 us 16 CREATE
0.32 101.65 us 65.00 us 184.00 us 46 OPEN
0.35 726.43 us 350.00 us 1165.00 us 7 UNLINK
0.52 56.22 us 25.00 us 228.00 us 136 FINODELK
0.77 129.15 us 49.00 us 1138.00 us 87 READ
1.57 154.70 us 59.00 us 360.00 us 148 FXATTROP
2.44 159.13 us 41.00 us 5836.00 us 224 WRITE
3.64 188.62 us 54.00 us 848.00 us 282 LOOKUP
89.50 20741.70 us 9457.00 us 84733.00 us 63 FSYNC
Duration: 362318 seconds
Data Read: 2132763575<tel:(213)%20276-3575> bytes
Data Written: 83056358475 bytes
Brick: 10.10.0.101:/glusterfsvolumes/devops/devops-prometheus/brick
-------------------------------------------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 1 3 2
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2 14 22
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 166 253 502
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 893 14392
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245861 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.01 89.00 us 89.00 us 89.00 us 1 GETXATTR
0.02 62.00 us 53.00 us 71.00 us 2 OPENDIR
0.02 57.00 us 51.00 us 61.00 us 3 FSTAT
0.04 178.00 us 38.00 us 318.00 us 2 FSYNCDIR
0.12 474.00 us 451.00 us 497.00 us 2 READDIRP
0.18 64.00 us 28.00 us 113.00 us 23 STATFS
0.28 48.30 us 18.00 us 184.00 us 46 ENTRYLK
0.31 41.33 us 18.00 us 87.00 us 60 FLUSH
0.33 66.12 us 28.00 us 246.00 us 40 STAT
0.57 102.39 us 62.00 us 236.00 us 44 OPEN
0.58 664.86 us 311.00 us 855.00 us 7 UNLINK
0.86 51.37 us 22.00 us 101.00 us 134 FINODELK
1.07 118.28 us 46.00 us 437.00 us 72 READ
3.83 137.39 us 41.00 us 2431.00 us 222 WRITE
4.54 250.94 us 57.00 us 15444.00 us 144 FXATTROP
5.89 2932.56 us 134.00 us 17675.00 us 16 CREATE
7.83 224.34 us 61.00 us 12509.00 us 278 LOOKUP
73.50 9592.56 us 6897.00 us 36321.00 us 61 FSYNC
Duration: 362152 seconds
Data Read: 2005452176 bytes
Data Written: 83056358475 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 1 1 31
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 20 4283 3085
Block Size: 128b+ 256b+ 512b+
No. of Reads: 1 3 2
No. of Writes: 395 465 555
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2 14 22
No. of Writes: 72453 62462 52882
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 166 253 502
No. of Writes: 43122 21700 17817
Block Size: 65536b+ 131072b+
No. of Reads: 893 14392
No. of Writes: 56237 581071
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 205094 FORGET
0.00 0.00 us 0.00 us 0.00 us 245861 RELEASE
0.00 0.00 us 0.00 us 0.00 us 40492 RELEASEDIR
0.01 89.00 us 89.00 us 89.00 us 1 GETXATTR
0.02 62.00 us 53.00 us 71.00 us 2 OPENDIR
0.02 57.00 us 51.00 us 61.00 us 3 FSTAT
0.04 178.00 us 38.00 us 318.00 us 2 FSYNCDIR
0.12 474.00 us 451.00 us 497.00 us 2 READDIRP
0.18 64.00 us 28.00 us 113.00 us 23 STATFS
0.28 48.30 us 18.00 us 184.00 us 46 ENTRYLK
0.31 41.33 us 18.00 us 87.00 us 60 FLUSH
0.33 66.12 us 28.00 us 246.00 us 40 STAT
0.57 102.39 us 62.00 us 236.00 us 44 OPEN
0.58 664.86 us 311.00 us 855.00 us 7 UNLINK
0.86 51.37 us 22.00 us 101.00 us 134 FINODELK
1.07 118.28 us 46.00 us 437.00 us 72 READ
3.83 137.39 us 41.00 us 2431.00 us 222 WRITE
4.54 250.94 us 57.00 us 15444.00 us 144 FXATTROP
5.89 2932.56 us 134.00 us 17675.00 us 16 CREATE
7.83 224.34 us 61.00 us 12509.00 us 278 LOOKUP
73.50 9592.56 us 6897.00 us 36321.00 us 61 FSYNC
Duration: 362152 seconds
Data Read: 2005452176 bytes
Data Written: 83056358475 bytes
Seems latency of FSYNC is very high. Sometimes cpu wait is also very high:
top - 15:27:39 up 4 days, 4:58, 2 users, load average: 0.75, 0.72, 0.70
Tasks: 210 total, 1 running, 209 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 0.2 sy, 0.0 ni, 62.9 id, 36.5 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 8043984 total, 883420 free, 1788912 used, 5371652 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 5819396 avail Mem
Any ideas?
Thanks.
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170401/00224335/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 9491E54ADE8B47A088284EAA75AEEC31.png
Type: image/png
Size: 96159 bytes
Desc: 9491E54ADE8B47A088284EAA75AEEC31.png
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170401/00224335/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1AF5C885C5A5497196A56960EC888E2E.png
Type: image/png
Size: 57197 bytes
Desc: 1AF5C885C5A5497196A56960EC888E2E.png
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170401/00224335/attachment-0001.png>
More information about the Gluster-users
mailing list