[Gluster-users] High I/O And Processor Utilization

Kyle Harris kyle.harris98 at gmail.com
Fri Jan 8 20:14:22 UTC 2016


Hello,



It’s been a while since I last ran GlusterFS so I thought I might give it
another try here at home in my lab.  I am using the 3.7 branch on 2 systems
with a 3rd being an arbiter node.  Much like the last time I tried
GlusterFS, I keep running into issues with the glusterfsd process eating up
so many resources that the systems sometimes become all but unusable.  A
quick Google search tells me I am not the only one to run into this issue
but I have yet to find a cure.  The last time I ran GlusterFS, it was to
host web sites and I just chalked the problem up to a large number of small
files.  This time, I am using it to host VM’s and there are only 7 of them
and while they are running, they are not doing anything else.



When I run ‘gluster volume heal gv0 info’, it almost always shows a few
files and sometimes they say “Possibly undergoing heal”.  Next time, it
will be different files.  So I am wondering if the problems are heal
related and I’m hoping there is a way I can tune for this.



The systems are dual-quad core processors with 72 GB of memory and Gigabit
Nics.  Network speed doesn't seem to be the issue as when I check it with
iperf, I can consistently get over 850 Mbits/sec.  Processor and/or drive
speed seems to be where the issues are coming from but not sure which comes
first or what to do as I would think I should have ample resources.



When I don’t have Gluster running, I can run “hdparm –Tt /dev/sdb” and
consistently get over 110MB/sec.  Granted I am using 7200 rpm SATA3 drives
but for 7 VMs that are all very quiet, I find it hard to believe they are
not fast enough.  When I run GlusterFS with the bricks on that drive (and
nothing else running on those drives), I have seen that speed slow down to
20MB/sec.  As I type this, one of the systems has 5 VMs up on it and the
other has 2 and both experience the same thing.  A quick look at the output
of “top” will show load average of 15-20 and sometimes even higher.  Again,
the VMs are running but I have checked and they are not doing much of
anything other than running.



When I originally set this up, I didn't have any extra config settings in
place but as you can see from the output, I do now, although none have
helped thus far.  I am connecting to GlusterFS utilizing the built-in NFS
server (v3 but that doesn’t seem to matter either) with options noatime,
nodiratime, rsize=65536, and wsize=65536 but they didn't seem to make much
of a difference either.



So to sum up, the problem seems to be all the heals going on.  What can I
do to get this back under control?  Luckily it’s in a lab environment so I
can experiment a bit.  Here is some additional info that may be of
assistance and thank you in advance:



gluster volume info gv0:

Volume Name: gv0

Type: Replicate

Volume ID: 14e7bb9c-aa5e-4386-8dd2-83a88d93dc54

Status: Started

Number of Bricks: 1 x 3 = 3

Transport-type: tcp

Bricks:

Brick1: server1:/export/brick1

Brick2: server2:/export/brick1

Brick3: kvm:/export/brick1

Options Reconfigured:

nfs.acl: off

performance.readdir-ahead: on

performance.quick-read: off

performance.read-ahead: off

performance.io-cache: off

performance.stat-prefetch: off

cluster.eager-lock: enable

network.remote-dio: enable



---

I gathered this with load averages of approximately 12 so it has been much
higher

gluster volume status gv0 info:

Status of volume: gv0

------------------------------------------------------------------------------

Brick                : Brick server1:/export/brick1

TCP Port             : 49152

RDMA Port            : 0

Online               : Y

Pid                  : 4409

File System          : ext3

Device               : /dev/sdb1

Mount Options        : rw

Inode Size           : 128

Disk Space Free      : 1.7TB

Total Disk Space     : 1.8TB

Inode Count          : 244203520

Free Inodes          : 244203413

------------------------------------------------------------------------------

Brick                : Brick server2:/export/brick1

TCP Port             : 49152

RDMA Port            : 0

Online               : Y

Pid                  : 4535

File System          : ext3

Device               : /dev/sdb1

Mount Options        : rw

Inode Size           : 128

Disk Space Free      : 1.7TB

Total Disk Space     : 1.8TB

Inode Count          : 244203520

Free Inodes          : 244203405

---

gluster volume provile gv0 as follows:

Brick: server1:/export/brick1

--------------------------------

Cumulative Stats:

   Block Size:                 16b+                 512b+
1024b+

No. of Reads:                    3                  1319
   39679

No. of Writes:                    3                 10804
917



   Block Size:               2048b+                4096b+
8192b+

No. of Reads:                  673                 33872
7199

No. of Writes:                 3642                219192
133087



   Block Size:              16384b+               32768b+
65536b+

No. of Reads:                 8808                 20790
2275

No. of Writes:                52815                180145
4910



   Block Size:             131072b+

No. of Reads:               952554

No. of Writes:              5558597

%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop

---------   -----------   -----------   -----------   ------------
----

      0.00       0.00 us       0.00 us       0.00 us              5
FORGET

      0.00       0.00 us       0.00 us       0.00 us            755
RELEASE

      0.00       0.00 us       0.00 us       0.00 us            214
RELEASEDIR

      0.00       4.00 us       4.00 us       4.00 us              2
OPENDIR

      0.00     168.67 us      31.00 us     406.00 us              3
READDIR

      0.00     535.40 us      84.00 us    1612.00 us              5
OPEN

      0.00     975.33 us     916.00 us    1045.00 us              3
FTRUNCATE

      0.00     305.36 us      15.00 us   14784.00 us             59
FLUSH

      0.00   14988.50 us    9326.00 us   20651.00 us              2
XATTROP

      0.00   16257.40 us      61.00 us   58866.00 us             10
LOOKUP

      0.01   41136.33 us      37.00 us  219392.00 us              6
GETXATTR

      0.01  146916.50 us   51093.00 us  242740.00 us              2
SETATTR

      0.01     637.58 us      24.00 us  124014.00 us            651
READ

      0.02     241.17 us       6.00 us   75127.00 us           2553
ACCESS

      0.02     220.31 us      10.00 us   77807.00 us           3235
STAT

      0.10  223016.80 us   72200.00 us  681213.00 us             15
FSYNC

      0.79    1466.11 us      29.00 us  236061.00 us          18414
FXATTROP

      1.20    1392.82 us      13.00 us  842550.00 us          29201
FINODELK

     12.66    3079.72 us      19.00 us 1161311.00 us         139747
INODELK

     85.17   34610.43 us      56.00 us 2401944.00 us          83630
WRITE



    Duration: 81829 seconds

   Data Read: 126295937593 bytes

Data Written: 740051667513 bytes



Interval 7 Stats:

   Block Size:                512b+                1024b+
2048b+

No. of Reads:                    0
234                     0

No. of Writes:                  171
7                    16



   Block Size:               4096b+                8192b+
16384b+

No. of Reads:                    0
0                     0

No. of Writes:                 1162                   472
278



   Block Size:              32768b+               65536b+
131072b+

No. of Reads:                    1
0                     0

No. of Writes:                 1593                    30
36991



%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop

---------   -----------   -----------   -----------   ------------
----

      0.00       0.00 us       0.00 us       0.00 us              1
RELEASE

      0.00       4.00 us       4.00 us       4.00 us              1
OPENDIR

      0.00     406.00 us     406.00 us     406.00 us              1
READDIR

      0.00     979.50 us     347.00 us    1612.00 us              2
OPEN

      0.00    1005.00 us     965.00 us    1045.00 us              2
FTRUNCATE

      0.00    9326.00 us    9326.00 us    9326.00 us              1
XATTROP

      0.00     656.32 us      27.00 us   14784.00 us             25
FLUSH

      0.00   51093.00 us   51093.00 us   51093.00 us              1
SETATTR

      0.01   16878.60 us     164.00 us   47828.00 us              5
LOOKUP

      0.01     363.44 us      24.00 us   47319.00 us            235
READ

      0.01     112.50 us      10.00 us   18678.00 us           1162
STAT

      0.02   78723.67 us      38.00 us  219392.00 us              3
GETXATTR

      0.03     387.51 us       7.00 us   75127.00 us            918
ACCESS

      0.16  426656.75 us  217375.00 us  681213.00 us              4
FSYNC

      0.51    1440.22 us      37.00 us  175074.00 us           3820
FXATTROP

      1.00    1311.97 us      16.00 us  842550.00 us           8218
FINODELK

     13.44    1951.10 us      23.00 us 1161311.00 us          73993
INODELK

     84.79   22359.52 us      56.00 us 2401944.00 us          40723
WRITE



    Duration: 394 seconds

   Data Read: 392192 bytes

Data Written: 4934906368 bytes



Brick: kvm:/export/brick1

--------------------------

Cumulative Stats:

   Block Size:                 16b+                 512b+
1024b+

No. of Reads:                    0
0                     0

No. of Writes:                    5                 10288
849



   Block Size:               2048b+                4096b+
8192b+

No. of Reads:                    0
0                     0

No. of Writes:                 3399                206329
128093



   Block Size:              16384b+               32768b+
65536b+

No. of Reads:                    0
          0                     0

No. of Writes:                50427                175784
4811



   Block Size:             131072b+

No. of Reads:                    0

No. of Writes:              6260106

%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop

---------   -----------   -----------   -----------   ------------
----

      0.00       0.00 us       0.00 us       0.00 us            250
RELEASE

      0.00       0.00 us       0.00 us       0.00 us            144
RELEASEDIR



    Duration: 72903 seconds

   Data Read: 0 bytes

Data Written: 831653482591 bytes



Interval 7 Stats:

   Block Size:                512b+                1024b+
2048b+

No. of Reads:                    0
          0                     0

No. of Writes:                  171
7                    16



   Block Size:               4096b+                8192b+
16384b+

No. of Reads:                    0                     0
                0

No. of Writes:                 1162                   472
278



   Block Size:              32768b+               65536b+
131072b+

No. of Reads:                    0
0                     0

No. of Writes:                 1593                    30
36068



%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop

---------   -----------   -----------   -----------   ------------
----

      0.00       0.00 us       0.00 us       0.00 us              1
RELEASE



    Duration: 394 seconds

   Data Read: 0 bytes

Data Written: 4813926912 bytes



Brick: server2:/export/brick1

--------------------------------

Cumulative Stats:

   Block Size:                 16b+                 512b+
1024b+

No. of Reads:                    2                  1554
187

No. of Writes:                    3                 10813
920



   Block Size:               2048b+                4096b+
8192b+

No. of Reads:                    0                205139
61200

No. of Writes:                 3642                219205
133087



   Block Size:              16384b+               32768b+
           65536b+

No. of Reads:                41437                311428
9278

No. of Writes:                52809                180141
4853



   Block Size:             131072b+

No. of Reads:              5558571

No. of Writes:               292354

%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop

---------   -----------   -----------   -----------   ------------
----

      0.00       0.00 us       0.00 us       0.00 us              5
     FORGET

      0.00       0.00 us       0.00 us       0.00 us            795
RELEASE

      0.00       0.00 us       0.00 us       0.00 us            196
RELEASEDIR

      0.00      94.49 us      10.00 us    2660.00 us             59
FLUSH

      0.00    6859.40 us      78.00 us   16605.00 us              5
OPEN

      0.00   22617.50 us   11067.00 us   34168.00 us              2
XATTROP

      0.01   20768.00 us     112.00 us   81026.00 us              8
LOOKUP

      0.04   37279.69 us   19436.00 us   86934.00 us             13
FSYNC

      0.85     385.97 us      10.00 us  212031.00 us          28913
FINODELK

      1.33     951.03 us      34.00 us  221747.00 us          18400
FXATTROP

      1.62     303.60 us      22.00 us  231359.00 us          70024
READ

      9.25     868.44 us      16.00 us  407424.00 us         139714
INODELK

     86.89   82862.96 us      59.00 us  638518.00 us          13759
WRITE



    Duration: 82915 seconds

   Data Read: 744922109478 bytes

Data Written: 49788871225 bytes



Interval 7 Stats:

   Block Size:                512b+                1024b+
2048b+

No. of Reads:                   30
0                     0

No. of Writes:                  171
7                    16



   Block Size:               4096b+                8192b+
16384b+

No. of Reads:                   55                     4
25

No. of Writes:                 1162                   472
278



   Block Size:              32768b+               65536b+
131072b+

No. of Reads:                   30                     0
37001

No. of Writes:                 1593
       30                     0



%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop

---------   -----------   -----------   -----------   ------------
----

      0.00       0.00 us       0.00 us       0.00 us              1
RELEASE

      0.00      45.52 us      10.00 us      91.00 us             25
FLUSH

      0.00   11067.00 us   11067.00 us   11067.00 us              1
XATTROP

      0.00    9621.00 us    2637.00 us   16605.00 us              2
OPEN

      0.03   44736.33 us   19436.00 us   86934.00 us              3
FSYNC

      0.03   33648.00 us     129.00 us   81026.00 us              4
LOOKUP

      1.21    1351.63 us      35.00 us  123188.00 us           3820
FXATTROP

      1.21     634.89 us      10.00 us  212031.00 us           8156
FINODELK

      2.96     339.80 us      22.00 us  231359.00 us          37145
READ

     12.90     743.44 us      17.00 us  407424.00 us          74014
INODELK

     81.64   93359.19 us      64.00 us  442704.00 us           3729
WRITE



    Duration: 394 seconds

   Data Read: 4851727360 bytes

Data Written: 86422016 bytes


Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160108/9e54ebca/attachment.html>


More information about the Gluster-users mailing list