[Gluster-users] High CPU/load, High latency on FSYNC
Serkan Çoban
cobanserkan at gmail.com
Fri Mar 31 08:07:38 UTC 2017
Prometheus uses *LOTS* of small files and do small random-io.
I do not recommend to use data volume of prometheus on glusterfs.
Why you need that?
On Fri, Mar 31, 2017 at 10:30 AM, Yong Zhang <hiscal at outlook.com> wrote:
> Hi, All
>
>
>
> I have glusterfs 3.10.0 running with two replicated nodes, but one of the
> node has a bad CPU performance:
>
>
>
> Node1/Node2: 4 core, 8GB, using thin LVM for volumes
>
>
>
> This is node1:
>
>
>
> And this is node2:
>
>
>
> Currently I have only Prometheus using glusterfs volumes, so I started
> profile on volume prometheus and got this:
>
>
>
> Brick: 10.10.0.100:/glusterfsvolumes/devops/devops-prometheus/brick
>
> -------------------------------------------------------------------
>
> Cumulative Stats:
>
> Block Size: 2b+
> 4b+ 8b+
>
> No. of Reads: 0
> 0 0
>
> No. of Writes: 1
> 1 31
>
>
>
> Block Size: 16b+ 32b+
> 64b+
>
> No. of Reads: 0
> 1 0
>
> No. of Writes: 20 4283
> 3085
>
>
>
> Block Size: 128b+ 256b+
> 512b+
>
> No. of Reads: 0
> 1 0
>
> No. of Writes: 395
> 465 555
>
>
>
> Block Size: 1024b+ 2048b+
> 4096b+
>
> No. of Reads: 4
> 6 32
>
> No. of Writes: 72453 62462
> 52882
>
>
>
> Block Size: 8192b+ 16384b+
> 32768b+
>
> No. of Reads: 154
> 281 484
>
> No. of Writes: 43122 21700
> 17817
>
>
>
> Block Size: 65536b+ 131072b+
>
> No. of Reads: 993 15290
>
> No. of Writes: 56237 581071
>
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
>
> --------- ----------- ----------- ----------- ------------
> ----
>
> 0.00 0.00 us 0.00 us 0.00 us 205094
> FORGET
>
> 0.00 0.00 us 0.00 us 0.00 us 245895
> RELEASE
>
> 0.00 0.00 us 0.00 us 0.00 us 40492
> RELEASEDIR
>
> 0.00 20.50 us 19.00 us 22.00 us 2
> FSYNCDIR
>
> 0.01 99.00 us 99.00 us 99.00 us 1
> GETXATTR
>
> 0.01 68.50 us 48.00 us 89.00 us 2
> OPENDIR
>
> 0.03 84.33 us 40.00 us 164.00 us 6
> FSTAT
>
> 0.10 64.35 us 34.00 us 177.00 us 23
> STATFS
>
> 0.16 62.87 us 29.00 us 172.00 us 38
> STAT
>
> 0.17 54.76 us 22.00 us 194.00 us 46
> ENTRYLK
>
> 0.18 41.66 us 24.00 us 95.00 us 62
> FLUSH
>
> 0.22 198.44 us 138.00 us 339.00 us 16
> CREATE
>
> 0.32 101.65 us 65.00 us 184.00 us 46
> OPEN
>
> 0.35 726.43 us 350.00 us 1165.00 us 7
> UNLINK
>
> 0.52 56.22 us 25.00 us 228.00 us 136
> FINODELK
>
> 0.77 129.15 us 49.00 us 1138.00 us 87
> READ
>
> 1.57 154.70 us 59.00 us 360.00 us 148
> FXATTROP
>
> 2.44 159.13 us 41.00 us 5836.00 us 224
> WRITE
>
> 3.64 188.62 us 54.00 us 848.00 us 282
> LOOKUP
>
> 89.50 20741.70 us 9457.00 us 84733.00 us 63
> FSYNC
>
> Duration: 362318 seconds
>
> Data Read: 2132763575 <(213)%20276-3575> bytes
>
> Data Written: 83056358475 bytes
>
> Interval 0 Stats:
>
> Block Size: 2b+
> 4b+ 8b+
>
> No. of Reads: 0
> 0 0
>
> No. of Writes: 1
> 1 31
>
>
>
> Block Size: 16b+ 32b+
> 64b+
>
> No. of Reads: 0
> 1 0
>
> No. of Writes: 20 4283
> 3085
>
>
>
> Block Size: 128b+ 256b+
> 512b+
>
> No. of Reads: 0
> 1 0
>
> No. of Writes: 395
> 465 555
>
>
>
> Block Size: 1024b+ 2048b+
> 4096b+
>
> No. of Reads: 4
> 6 32
>
> No. of Writes: 72453 62462
> 52882
>
>
>
> Block Size: 8192b+ 16384b+
> 32768b+
>
> No. of Reads: 154
> 281 484
>
> No. of Writes: 43122 21700
> 17817
>
>
>
> Block Size: 65536b+ 131072b+
>
> No. of Reads: 993 15290
>
> No. of Writes: 56237 581071
>
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
>
> --------- ----------- ----------- ----------- ------------
> ----
>
> 0.00 0.00 us 0.00 us 0.00 us 205094
> FORGET
>
> 0.00 0.00 us 0.00 us 0.00 us 245895
> RELEASE
>
> 0.00 0.00 us 0.00 us 0.00 us 40492
> RELEASEDIR
>
> 0.00 20.50 us 19.00 us 22.00 us 2
> FSYNCDIR
>
> 0.01 99.00 us 99.00 us 99.00 us 1
> GETXATTR
>
> 0.01 68.50 us 48.00 us 89.00 us 2
> OPENDIR
>
> 0.03 84.33 us 40.00 us 164.00 us 6
> FSTAT
>
> 0.10 64.35 us 34.00 us 177.00 us 23
> STATFS
>
> 0.16 62.87 us 29.00 us 172.00 us 38
> STAT
>
> 0.17 54.76 us 22.00 us 194.00 us 46
> ENTRYLK
>
> 0.18 41.66 us 24.00 us 95.00 us 62
> FLUSH
>
> 0.22 198.44 us 138.00 us 339.00 us 16
> CREATE
>
> 0.32 101.65 us 65.00 us 184.00 us 46
> OPEN
>
> 0.35 726.43 us 350.00 us 1165.00 us 7
> UNLINK
>
> 0.52 56.22 us 25.00 us 228.00 us 136
> FINODELK
>
> 0.77 129.15 us 49.00 us 1138.00 us 87
> READ
>
> 1.57 154.70 us 59.00 us 360.00 us 148
> FXATTROP
>
> 2.44 159.13 us 41.00 us 5836.00 us 224
> WRITE
>
> 3.64 188.62 us 54.00 us 848.00 us 282
> LOOKUP
>
> 89.50 20741.70 us 9457.00 us 84733.00 us 63
> FSYNC
>
> Duration: 362318 seconds
>
> Data Read: 2132763575 <(213)%20276-3575> bytes
>
> Data Written: 83056358475 bytes
>
> Brick: 10.10.0.101:/glusterfsvolumes/devops/devops-prometheus/brick
>
> -------------------------------------------------------------------
>
> Cumulative Stats:
>
> Block Size: 2b+
> 4b+ 8b+
>
> No. of Reads: 0 0
> 0
>
> No. of Writes: 1
> 1 31
>
>
>
> Block Size: 16b+ 32b+
> 64b+
>
> No. of Reads: 0
> 0 0
>
> No. of Writes: 20 4283
> 3085
>
>
>
> Block Size: 128b+ 256b+
> 512b+
>
> No. of Reads: 1
> 3 2
>
> No. of Writes: 395
> 465 555
>
>
>
> Block Size: 1024b+ 2048b+
> 4096b+
>
> No. of Reads: 2
> 14 22
>
> No. of Writes: 72453 62462
> 52882
>
>
>
> Block Size: 8192b+ 16384b+
> 32768b+
>
> No. of Reads: 166
> 253 502
>
> No. of Writes: 43122 21700
> 17817
>
>
>
> Block Size: 65536b+ 131072b+
>
> No. of Reads: 893 14392
>
> No. of Writes: 56237 581071
>
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
>
> --------- ----------- ----------- ----------- ------------
> ----
>
> 0.00 0.00 us 0.00 us 0.00 us 205094
> FORGET
>
> 0.00 0.00 us 0.00 us 0.00 us 245861
> RELEASE
>
> 0.00 0.00 us 0.00 us 0.00 us 40492
> RELEASEDIR
>
> 0.01 89.00 us 89.00 us 89.00 us 1
> GETXATTR
>
> 0.02 62.00 us 53.00 us 71.00 us 2
> OPENDIR
>
> 0.02 57.00 us 51.00 us 61.00 us 3
> FSTAT
>
> 0.04 178.00 us 38.00 us 318.00 us 2
> FSYNCDIR
>
> 0.12 474.00 us 451.00 us 497.00 us 2
> READDIRP
>
> 0.18 64.00 us 28.00 us 113.00 us 23
> STATFS
>
> 0.28 48.30 us 18.00 us 184.00 us 46
> ENTRYLK
>
> 0.31 41.33 us 18.00 us 87.00 us 60
> FLUSH
>
> 0.33 66.12 us 28.00 us 246.00 us 40
> STAT
>
> 0.57 102.39 us 62.00 us 236.00 us 44
> OPEN
>
> 0.58 664.86 us 311.00 us 855.00 us 7
> UNLINK
>
> 0.86 51.37 us 22.00 us 101.00 us 134
> FINODELK
>
> 1.07 118.28 us 46.00 us 437.00 us 72
> READ
>
> 3.83 137.39 us 41.00 us 2431.00 us 222
> WRITE
>
> 4.54 250.94 us 57.00 us 15444.00 us 144
> FXATTROP
>
> 5.89 2932.56 us 134.00 us 17675.00 us 16
> CREATE
>
> 7.83 224.34 us 61.00 us 12509.00 us 278
> LOOKUP
>
> 73.50 9592.56 us 6897.00 us 36321.00 us 61
> FSYNC
>
> Duration: 362152 seconds
>
> Data Read: 2005452176 bytes
>
> Data Written: 83056358475 bytes
>
> Interval 0 Stats:
>
> Block Size: 2b+
> 4b+ 8b+
>
> No. of Reads: 0
> 0 0
>
> No. of Writes: 1
> 1 31
>
>
>
> Block Size: 16b+ 32b+
> 64b+
>
> No. of Reads: 0
> 0 0
>
> No. of Writes: 20 4283
> 3085
>
>
>
> Block Size: 128b+ 256b+
> 512b+
>
> No. of Reads: 1
> 3 2
>
> No. of Writes: 395
> 465 555
>
>
>
> Block Size: 1024b+ 2048b+
> 4096b+
>
> No. of Reads: 2
> 14 22
>
> No. of Writes: 72453 62462
> 52882
>
>
>
> Block Size: 8192b+ 16384b+
> 32768b+
>
> No. of Reads: 166
> 253 502
>
> No. of Writes: 43122 21700
> 17817
>
>
>
> Block Size: 65536b+ 131072b+
>
> No. of Reads: 893 14392
>
> No. of Writes: 56237 581071
>
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
>
> --------- ----------- ----------- ----------- ------------
> ----
>
> 0.00 0.00 us 0.00 us 0.00 us 205094
> FORGET
>
> 0.00 0.00 us 0.00 us 0.00 us 245861
> RELEASE
>
> 0.00 0.00 us 0.00 us 0.00 us 40492
> RELEASEDIR
>
> 0.01 89.00 us 89.00 us 89.00 us 1
> GETXATTR
>
> 0.02 62.00 us 53.00 us 71.00 us 2
> OPENDIR
>
> 0.02 57.00 us 51.00 us 61.00 us 3
> FSTAT
>
> 0.04 178.00 us 38.00 us 318.00 us 2
> FSYNCDIR
>
> 0.12 474.00 us 451.00 us 497.00 us 2
> READDIRP
>
> 0.18 64.00 us 28.00 us 113.00 us 23
> STATFS
>
> 0.28 48.30 us 18.00 us 184.00 us 46
> ENTRYLK
>
> 0.31 41.33 us 18.00 us 87.00 us 60
> FLUSH
>
> 0.33 66.12 us 28.00 us 246.00 us 40
> STAT
>
> 0.57 102.39 us 62.00 us 236.00 us 44
> OPEN
>
> 0.58 664.86 us 311.00 us 855.00 us 7
> UNLINK
>
> 0.86 51.37 us 22.00 us 101.00 us 134
> FINODELK
>
> 1.07 118.28 us 46.00 us 437.00 us 72
> READ
>
> 3.83 137.39 us 41.00 us 2431.00 us 222
> WRITE
>
> 4.54 250.94 us 57.00 us 15444.00 us 144
> FXATTROP
>
> 5.89 2932.56 us 134.00 us 17675.00 us 16
> CREATE
>
> 7.83 224.34 us 61.00 us 12509.00 us 278
> LOOKUP
>
> 73.50 9592.56 us 6897.00 us 36321.00 us 61
> FSYNC
>
> Duration: 362152 seconds
>
> Data Read: 2005452176 bytes
>
> Data Written: 83056358475 bytes
>
>
>
>
>
> Seems latency of FSYNC is very high. Sometimes cpu wait is also very high:
>
>
>
> top - 15:27:39 up 4 days, 4:58, 2 users, load average: 0.75, 0.72, 0.70
>
> Tasks: 210 total, 1 running, 209 sleeping, 0 stopped, 0 zombie
>
> %Cpu(s): 0.3 us, 0.2 sy, 0.0 ni, 62.9 id, *36.5* wa, 0.0 hi, 0.1
> si, 0.0 st
>
> KiB Mem : 8043984 total, 883420 free, 1788912 used, 5371652 buff/cache
>
> KiB Swap: 0 total, 0 free, 0 used. 5819396 avail Mem
>
> Any ideas?
>
>
>
> Thanks.
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170331/255e1e1d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 9491E54ADE8B47A088284EAA75AEEC31.png
Type: image/png
Size: 96159 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170331/255e1e1d/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1AF5C885C5A5497196A56960EC888E2E.png
Type: image/png
Size: 57197 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170331/255e1e1d/attachment-0003.png>
More information about the Gluster-users
mailing list