[Gluster-users] High I/O And Processor Utilization
Kyle Harris
kyle.harris98 at gmail.com
Fri Jan 8 20:14:22 UTC 2016
Hello,
It’s been a while since I last ran GlusterFS so I thought I might give it
another try here at home in my lab. I am using the 3.7 branch on 2 systems
with a 3rd being an arbiter node. Much like the last time I tried
GlusterFS, I keep running into issues with the glusterfsd process eating up
so many resources that the systems sometimes become all but unusable. A
quick Google search tells me I am not the only one to run into this issue
but I have yet to find a cure. The last time I ran GlusterFS, it was to
host web sites and I just chalked the problem up to a large number of small
files. This time, I am using it to host VM’s and there are only 7 of them
and while they are running, they are not doing anything else.
When I run ‘gluster volume heal gv0 info’, it almost always shows a few
files and sometimes they say “Possibly undergoing heal”. Next time, it
will be different files. So I am wondering if the problems are heal
related and I’m hoping there is a way I can tune for this.
The systems are dual-quad core processors with 72 GB of memory and Gigabit
Nics. Network speed doesn't seem to be the issue as when I check it with
iperf, I can consistently get over 850 Mbits/sec. Processor and/or drive
speed seems to be where the issues are coming from but not sure which comes
first or what to do as I would think I should have ample resources.
When I don’t have Gluster running, I can run “hdparm –Tt /dev/sdb” and
consistently get over 110MB/sec. Granted I am using 7200 rpm SATA3 drives
but for 7 VMs that are all very quiet, I find it hard to believe they are
not fast enough. When I run GlusterFS with the bricks on that drive (and
nothing else running on those drives), I have seen that speed slow down to
20MB/sec. As I type this, one of the systems has 5 VMs up on it and the
other has 2 and both experience the same thing. A quick look at the output
of “top” will show load average of 15-20 and sometimes even higher. Again,
the VMs are running but I have checked and they are not doing much of
anything other than running.
When I originally set this up, I didn't have any extra config settings in
place but as you can see from the output, I do now, although none have
helped thus far. I am connecting to GlusterFS utilizing the built-in NFS
server (v3 but that doesn’t seem to matter either) with options noatime,
nodiratime, rsize=65536, and wsize=65536 but they didn't seem to make much
of a difference either.
So to sum up, the problem seems to be all the heals going on. What can I
do to get this back under control? Luckily it’s in a lab environment so I
can experiment a bit. Here is some additional info that may be of
assistance and thank you in advance:
gluster volume info gv0:
Volume Name: gv0
Type: Replicate
Volume ID: 14e7bb9c-aa5e-4386-8dd2-83a88d93dc54
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: server1:/export/brick1
Brick2: server2:/export/brick1
Brick3: kvm:/export/brick1
Options Reconfigured:
nfs.acl: off
performance.readdir-ahead: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
---
I gathered this with load averages of approximately 12 so it has been much
higher
gluster volume status gv0 info:
Status of volume: gv0
------------------------------------------------------------------------------
Brick : Brick server1:/export/brick1
TCP Port : 49152
RDMA Port : 0
Online : Y
Pid : 4409
File System : ext3
Device : /dev/sdb1
Mount Options : rw
Inode Size : 128
Disk Space Free : 1.7TB
Total Disk Space : 1.8TB
Inode Count : 244203520
Free Inodes : 244203413
------------------------------------------------------------------------------
Brick : Brick server2:/export/brick1
TCP Port : 49152
RDMA Port : 0
Online : Y
Pid : 4535
File System : ext3
Device : /dev/sdb1
Mount Options : rw
Inode Size : 128
Disk Space Free : 1.7TB
Total Disk Space : 1.8TB
Inode Count : 244203520
Free Inodes : 244203405
---
gluster volume provile gv0 as follows:
Brick: server1:/export/brick1
--------------------------------
Cumulative Stats:
Block Size: 16b+ 512b+
1024b+
No. of Reads: 3 1319
39679
No. of Writes: 3 10804
917
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 673 33872
7199
No. of Writes: 3642 219192
133087
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 8808 20790
2275
No. of Writes: 52815 180145
4910
Block Size: 131072b+
No. of Reads: 952554
No. of Writes: 5558597
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 5
FORGET
0.00 0.00 us 0.00 us 0.00 us 755
RELEASE
0.00 0.00 us 0.00 us 0.00 us 214
RELEASEDIR
0.00 4.00 us 4.00 us 4.00 us 2
OPENDIR
0.00 168.67 us 31.00 us 406.00 us 3
READDIR
0.00 535.40 us 84.00 us 1612.00 us 5
OPEN
0.00 975.33 us 916.00 us 1045.00 us 3
FTRUNCATE
0.00 305.36 us 15.00 us 14784.00 us 59
FLUSH
0.00 14988.50 us 9326.00 us 20651.00 us 2
XATTROP
0.00 16257.40 us 61.00 us 58866.00 us 10
LOOKUP
0.01 41136.33 us 37.00 us 219392.00 us 6
GETXATTR
0.01 146916.50 us 51093.00 us 242740.00 us 2
SETATTR
0.01 637.58 us 24.00 us 124014.00 us 651
READ
0.02 241.17 us 6.00 us 75127.00 us 2553
ACCESS
0.02 220.31 us 10.00 us 77807.00 us 3235
STAT
0.10 223016.80 us 72200.00 us 681213.00 us 15
FSYNC
0.79 1466.11 us 29.00 us 236061.00 us 18414
FXATTROP
1.20 1392.82 us 13.00 us 842550.00 us 29201
FINODELK
12.66 3079.72 us 19.00 us 1161311.00 us 139747
INODELK
85.17 34610.43 us 56.00 us 2401944.00 us 83630
WRITE
Duration: 81829 seconds
Data Read: 126295937593 bytes
Data Written: 740051667513 bytes
Interval 7 Stats:
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 0
234 0
No. of Writes: 171
7 16
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 0
0 0
No. of Writes: 1162 472
278
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 1
0 0
No. of Writes: 1593 30
36991
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 1
RELEASE
0.00 4.00 us 4.00 us 4.00 us 1
OPENDIR
0.00 406.00 us 406.00 us 406.00 us 1
READDIR
0.00 979.50 us 347.00 us 1612.00 us 2
OPEN
0.00 1005.00 us 965.00 us 1045.00 us 2
FTRUNCATE
0.00 9326.00 us 9326.00 us 9326.00 us 1
XATTROP
0.00 656.32 us 27.00 us 14784.00 us 25
FLUSH
0.00 51093.00 us 51093.00 us 51093.00 us 1
SETATTR
0.01 16878.60 us 164.00 us 47828.00 us 5
LOOKUP
0.01 363.44 us 24.00 us 47319.00 us 235
READ
0.01 112.50 us 10.00 us 18678.00 us 1162
STAT
0.02 78723.67 us 38.00 us 219392.00 us 3
GETXATTR
0.03 387.51 us 7.00 us 75127.00 us 918
ACCESS
0.16 426656.75 us 217375.00 us 681213.00 us 4
FSYNC
0.51 1440.22 us 37.00 us 175074.00 us 3820
FXATTROP
1.00 1311.97 us 16.00 us 842550.00 us 8218
FINODELK
13.44 1951.10 us 23.00 us 1161311.00 us 73993
INODELK
84.79 22359.52 us 56.00 us 2401944.00 us 40723
WRITE
Duration: 394 seconds
Data Read: 392192 bytes
Data Written: 4934906368 bytes
Brick: kvm:/export/brick1
--------------------------
Cumulative Stats:
Block Size: 16b+ 512b+
1024b+
No. of Reads: 0
0 0
No. of Writes: 5 10288
849
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 0
0 0
No. of Writes: 3399 206329
128093
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 0
0 0
No. of Writes: 50427 175784
4811
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 6260106
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 250
RELEASE
0.00 0.00 us 0.00 us 0.00 us 144
RELEASEDIR
Duration: 72903 seconds
Data Read: 0 bytes
Data Written: 831653482591 bytes
Interval 7 Stats:
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 0
0 0
No. of Writes: 171
7 16
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 0 0
0
No. of Writes: 1162 472
278
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 0
0 0
No. of Writes: 1593 30
36068
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 1
RELEASE
Duration: 394 seconds
Data Read: 0 bytes
Data Written: 4813926912 bytes
Brick: server2:/export/brick1
--------------------------------
Cumulative Stats:
Block Size: 16b+ 512b+
1024b+
No. of Reads: 2 1554
187
No. of Writes: 3 10813
920
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 0 205139
61200
No. of Writes: 3642 219205
133087
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 41437 311428
9278
No. of Writes: 52809 180141
4853
Block Size: 131072b+
No. of Reads: 5558571
No. of Writes: 292354
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 5
FORGET
0.00 0.00 us 0.00 us 0.00 us 795
RELEASE
0.00 0.00 us 0.00 us 0.00 us 196
RELEASEDIR
0.00 94.49 us 10.00 us 2660.00 us 59
FLUSH
0.00 6859.40 us 78.00 us 16605.00 us 5
OPEN
0.00 22617.50 us 11067.00 us 34168.00 us 2
XATTROP
0.01 20768.00 us 112.00 us 81026.00 us 8
LOOKUP
0.04 37279.69 us 19436.00 us 86934.00 us 13
FSYNC
0.85 385.97 us 10.00 us 212031.00 us 28913
FINODELK
1.33 951.03 us 34.00 us 221747.00 us 18400
FXATTROP
1.62 303.60 us 22.00 us 231359.00 us 70024
READ
9.25 868.44 us 16.00 us 407424.00 us 139714
INODELK
86.89 82862.96 us 59.00 us 638518.00 us 13759
WRITE
Duration: 82915 seconds
Data Read: 744922109478 bytes
Data Written: 49788871225 bytes
Interval 7 Stats:
Block Size: 512b+ 1024b+
2048b+
No. of Reads: 30
0 0
No. of Writes: 171
7 16
Block Size: 4096b+ 8192b+
16384b+
No. of Reads: 55 4
25
No. of Writes: 1162 472
278
Block Size: 32768b+ 65536b+
131072b+
No. of Reads: 30 0
37001
No. of Writes: 1593
30 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.00 0.00 us 0.00 us 0.00 us 1
RELEASE
0.00 45.52 us 10.00 us 91.00 us 25
FLUSH
0.00 11067.00 us 11067.00 us 11067.00 us 1
XATTROP
0.00 9621.00 us 2637.00 us 16605.00 us 2
OPEN
0.03 44736.33 us 19436.00 us 86934.00 us 3
FSYNC
0.03 33648.00 us 129.00 us 81026.00 us 4
LOOKUP
1.21 1351.63 us 35.00 us 123188.00 us 3820
FXATTROP
1.21 634.89 us 10.00 us 212031.00 us 8156
FINODELK
2.96 339.80 us 22.00 us 231359.00 us 37145
READ
12.90 743.44 us 17.00 us 407424.00 us 74014
INODELK
81.64 93359.19 us 64.00 us 442704.00 us 3729
WRITE
Duration: 394 seconds
Data Read: 4851727360 bytes
Data Written: 86422016 bytes
Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160108/9e54ebca/attachment.html>
More information about the Gluster-users
mailing list