[Gluster-users] Client load high (300) using fuse mount
Mitja Mihelič
mitja.mihelic at arnes.si
Tue Jun 2 12:07:28 UTC 2015
On 02. 06. 2015 07:33, Pranith Kumar Karampuri wrote:
> hi Mitja,
> Could you please give output of the following commands:
> 1) gluster volume info
Volume Name: gvol-splet
Type: Replicate
Volume ID: FAKE-ID
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: gluster1.setup.tld:/gluster/gvol-splet/brick0/brick
Brick2: gluster2.setup.tld:/gluster/gvol-splet/brick0/brick
Brick3: gluster3.setup.tld:/gluster/gvol-splet/brick0/brick
Options Reconfigured:
performance.cache-size: 4GB
network.ping-timeout: 15
auth.allow: WEBNODE-IP1,WEBNODE-IP2
cluster.quorum-type: auto
network.remote-dio: on
cluster.eager-lock: on
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
performance.cache-refresh-timeout: 4
performance.io-thread-count: 32
nfs.disable: on
> 2) gluster volume profile <volname> start
> 3) Wait while the CPU is high for 5-10 minutes
> 4) gluster volume profile <volname> info >
> output-you-need-to-attach-to-this-mail.txt
I cannot give you the results from the production system, because the
web server was unresponsive and I switched back to local storage.
The attached file contains results from the setup that was briefly in
production and will be again when this is solved. The load is sythetic,
generated by jmeter. During the test iotop on GlusterFS peers showed
practically zero disk activity. Pretty much the same as under a real
world load. Average load on the web node was a bit above 50 constantly.
I will try to get the results from the production setup.
Rerads, Mitja
>
> 4th command tells what are the operations that are issued a lot.
>
> Pranith
>
> On 06/01/2015 04:41 PM, Mitja Mihelič wrote:
>> Hi!
>>
>> I am trying to set up a Wordpress cluster using GlusterFS used for
>> storage. Web nodes will access the same Wordpress install on a volume
>> mounted via FUSE from a 3 peer GlusterFS TSP.
>>
>> I started with one web node and Wordpress on local storage. The load
>> average was constantly about 5. iotop showed about 300kB/s disk reads
>> or less. The load average was below 6.
>>
>> When I mounted the GlusterFS volume to the web node the 1min load
>> average went over 300. Each of the 3 peers is transmitting about
>> 10MB/s to my web node regardless of the load.
>> TSP peers are on 10Gbit NICs and the web node is on a 1Gbit NIC.
>>
>> I'm out of ideas here... Could it be the network?
>> What should I look at for optimizing the network stack on the client?
>>
>> Options set on TSP:
>> Options Reconfigured:
>> performance.cache-size: 4GB
>> network.ping-timeout: 15
>> cluster.quorum-type: auto
>> network.remote-dio: on
>> cluster.eager-lock: on
>> performance.stat-prefetch: off
>> performance.io-cache: off
>> performance.read-ahead: off
>> performance.quick-read: off
>> performance.cache-refresh-timeout: 4
>> performance.io-thread-count: 32
>> nfs.disable: on
>>
>> Regards, Mitja
>>
-------------- next part --------------
Brick: gluster1.setup.tld:/gluster/gvol-splet/brick0/brick
---------------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 1
No. of Writes: 28 66 180
Block Size: 8b+ 16b+ 32b+
No. of Reads: 2 94354 131
No. of Writes: 212 42 51
Block Size: 64b+ 128b+ 256b+
No. of Reads: 71 2160815 208832
No. of Writes: 631 372 386
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 2481316 1414880 1377502
No. of Writes: 147 111 261
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 2753313 2770744 3389212
No. of Writes: 17604 2566 996
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1284591 803165 390224
No. of Writes: 721 1035 11387
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 3569 FORGET
0.00 0.00 us 0.00 us 0.00 us 6622013 RELEASE
0.00 0.00 us 0.00 us 0.00 us 5019505 RELEASEDIR
0.00 35.00 us 35.00 us 35.00 us 1 SETXATTR
0.00 67.00 us 67.00 us 67.00 us 1 SETATTR
0.00 62.50 us 56.00 us 69.00 us 2 ENTRYLK
0.00 32.00 us 20.00 us 39.00 us 6 INODELK
0.00 291.00 us 291.00 us 291.00 us 1 MKDIR
0.00 71.62 us 38.00 us 173.00 us 24 READLINK
0.00 48.32 us 12.00 us 276.00 us 118 READDIR
0.00 63.85 us 15.00 us 743.00 us 143 GETXATTR
0.01 52.48 us 16.00 us 586.00 us 746 STATFS
0.06 42.60 us 8.00 us 1196.00 us 5632 FSTAT
0.09 60.38 us 1.00 us 9462.00 us 5895 OPENDIR
0.52 390.60 us 16.00 us 3806.00 us 5351 READDIRP
1.36 49.93 us 2.00 us 7780.00 us 109232 STAT
1.48 63.59 us 13.00 us 13598.00 us 93432 READ
21.32 38.31 us 1.00 us 18114.00 us 2227255 FLUSH
36.36 64.60 us 15.00 us 18155.00 us 2252801 LOOKUP
38.79 62.69 us 17.00 us 13115.00 us 2476771 OPEN
Duration: 174991 seconds
Data Read: 300709628988 bytes
Data Written: 1739805813 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 1
No. of Writes: 28 66 180
Block Size: 8b+ 16b+ 32b+
No. of Reads: 2 94354 131
No. of Writes: 212 42 51
Block Size: 64b+ 128b+ 256b+
No. of Reads: 71 2160815 208832
No. of Writes: 631 372 386
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 2481316 1414880 1377502
No. of Writes: 147 111 261
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 2753313 2770744 3389212
No. of Writes: 17604 2566 996
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1284591 803165 390224
No. of Writes: 721 1035 11387
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 3569 FORGET
0.00 0.00 us 0.00 us 0.00 us 6622013 RELEASE
0.00 0.00 us 0.00 us 0.00 us 5019505 RELEASEDIR
0.00 35.00 us 35.00 us 35.00 us 1 SETXATTR
0.00 67.00 us 67.00 us 67.00 us 1 SETATTR
0.00 62.50 us 56.00 us 69.00 us 2 ENTRYLK
0.00 32.00 us 20.00 us 39.00 us 6 INODELK
0.00 291.00 us 291.00 us 291.00 us 1 MKDIR
0.00 71.62 us 38.00 us 173.00 us 24 READLINK
0.00 48.32 us 12.00 us 276.00 us 118 READDIR
0.00 63.85 us 15.00 us 743.00 us 143 GETXATTR
0.01 52.48 us 16.00 us 586.00 us 746 STATFS
0.06 42.60 us 8.00 us 1196.00 us 5632 FSTAT
0.09 60.38 us 1.00 us 9462.00 us 5895 OPENDIR
0.52 390.60 us 16.00 us 3806.00 us 5351 READDIRP
1.36 49.93 us 2.00 us 7780.00 us 109232 STAT
1.48 63.59 us 13.00 us 13598.00 us 93432 READ
21.32 38.31 us 1.00 us 18114.00 us 2227255 FLUSH
36.36 64.60 us 15.00 us 18155.00 us 2252801 LOOKUP
38.79 62.69 us 17.00 us 13115.00 us 2476771 OPEN
Duration: 174991 seconds
Data Read: 300709628988 bytes
Data Written: 1739805813 bytes
Brick: gluster3.setup.tld:/gluster/gvol-splet/brick0/brick
---------------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 79
No. of Writes: 28 66 180
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 573 249
No. of Writes: 212 42 51
Block Size: 64b+ 128b+ 256b+
No. of Reads: 223 2003541 1685421
No. of Writes: 631 372 386
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 338053 989804 1454919
No. of Writes: 147 111 261
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 1963655 5331377 3046634
No. of Writes: 17603 2566 996
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 845440 1915103 299323
No. of Writes: 720 1033 11386
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 3563 FORGET
0.00 0.00 us 0.00 us 0.00 us 6620572 RELEASE
0.00 0.00 us 0.00 us 0.00 us 5011703 RELEASEDIR
0.00 32.00 us 32.00 us 32.00 us 1 SETXATTR
0.00 68.00 us 68.00 us 68.00 us 1 SETATTR
0.00 37.00 us 33.00 us 41.00 us 2 ENTRYLK
0.00 36.50 us 21.00 us 74.00 us 6 INODELK
0.00 238.00 us 238.00 us 238.00 us 1 MKDIR
0.00 48.90 us 12.00 us 234.00 us 116 READDIR
0.00 67.88 us 15.00 us 783.00 us 141 GETXATTR
0.01 58.59 us 19.00 us 1732.00 us 746 STATFS
0.08 55.79 us 1.00 us 1190.00 us 5894 OPENDIR
0.10 45.55 us 8.00 us 1305.00 us 9328 FSTAT
0.21 326.85 us 15.00 us 1681.00 us 2728 READDIRP
1.49 54.34 us 3.00 us 18958.00 us 115090 STAT
1.98 71.15 us 8.00 us 20654.00 us 117248 READ
21.31 40.22 us 2.00 us 10795.00 us 2227256 FLUSH
36.28 67.71 us 11.00 us 19040.00 us 2252805 LOOKUP
38.53 65.39 us 19.00 us 17907.00 us 2476772 OPEN
Duration: 174244 seconds
Data Read: 369768437139 bytes
Data Written: 1739421118 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 79
No. of Writes: 28 66 180
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 573 249
No. of Writes: 212 42 51
Block Size: 64b+ 128b+ 256b+
No. of Reads: 223 2003541 1685421
No. of Writes: 631 372 386
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 338053 989804 1454919
No. of Writes: 147 111 261
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 1963655 5331377 3046634
No. of Writes: 17603 2566 996
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 845440 1915103 299323
No. of Writes: 720 1033 11386
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 3563 FORGET
0.00 0.00 us 0.00 us 0.00 us 6620572 RELEASE
0.00 0.00 us 0.00 us 0.00 us 5011703 RELEASEDIR
0.00 32.00 us 32.00 us 32.00 us 1 SETXATTR
0.00 68.00 us 68.00 us 68.00 us 1 SETATTR
0.00 37.00 us 33.00 us 41.00 us 2 ENTRYLK
0.00 36.50 us 21.00 us 74.00 us 6 INODELK
0.00 238.00 us 238.00 us 238.00 us 1 MKDIR
0.00 48.90 us 12.00 us 234.00 us 116 READDIR
0.00 67.88 us 15.00 us 783.00 us 141 GETXATTR
0.01 58.59 us 19.00 us 1732.00 us 746 STATFS
0.08 55.79 us 1.00 us 1190.00 us 5894 OPENDIR
0.10 45.55 us 8.00 us 1305.00 us 9328 FSTAT
0.21 326.85 us 15.00 us 1681.00 us 2728 READDIRP
1.49 54.34 us 3.00 us 18958.00 us 115090 STAT
1.98 71.15 us 8.00 us 20654.00 us 117248 READ
21.31 40.22 us 2.00 us 10795.00 us 2227256 FLUSH
36.28 67.71 us 11.00 us 19040.00 us 2252805 LOOKUP
38.53 65.39 us 19.00 us 17907.00 us 2476772 OPEN
Duration: 174244 seconds
Data Read: 369768437139 bytes
Data Written: 1739421118 bytes
Brick: gluster2.setup.tld:/gluster/gvol-splet/brick0/brick
---------------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 2
No. of Writes: 28 66 180
Block Size: 8b+ 16b+ 32b+
No. of Reads: 1 188384 349
No. of Writes: 212 42 51
Block Size: 64b+ 128b+ 256b+
No. of Reads: 634 210098 284036
No. of Writes: 631 372 386
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 1782229 2067791 1914912
No. of Writes: 147 111 261
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 3797675 2091925 3298736
No. of Writes: 17603 2566 996
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1287459 1486233 381139
No. of Writes: 720 1033 11386
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 3571 FORGET
0.00 0.00 us 0.00 us 0.00 us 6621022 RELEASE
0.00 0.00 us 0.00 us 0.00 us 5012368 RELEASEDIR
0.00 55.00 us 55.00 us 55.00 us 1 SETXATTR
0.00 66.00 us 66.00 us 66.00 us 1 SETATTR
0.00 58.00 us 45.00 us 71.00 us 2 ENTRYLK
0.00 206.00 us 206.00 us 206.00 us 1 MKDIR
0.00 74.50 us 38.00 us 109.00 us 6 INODELK
0.00 65.58 us 13.00 us 562.00 us 118 READDIR
0.00 61.71 us 17.00 us 360.00 us 143 GETXATTR
0.01 54.48 us 20.00 us 471.00 us 746 STATFS
0.08 59.82 us 1.00 us 2510.00 us 5895 OPENDIR
0.09 51.75 us 11.00 us 5992.00 us 7837 FSTAT
0.33 59.02 us 11.00 us 4863.00 us 25459 STAT
0.41 76.79 us 15.00 us 13114.00 us 23838 READ
0.42 420.23 us 17.00 us 3692.00 us 4466 READDIRP
21.71 43.76 us 2.00 us 20153.00 us 2227256 FLUSH
36.89 73.52 us 16.00 us 37861.00 us 2252806 LOOKUP
40.06 72.61 us 22.00 us 17262.00 us 2476772 OPEN
Duration: 174493 seconds
Data Read: 347482097249 bytes
Data Written: 1739421118 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 2
No. of Writes: 28 66 180
Block Size: 8b+ 16b+ 32b+
No. of Reads: 1 188384 349
No. of Writes: 212 42 51
Block Size: 64b+ 128b+ 256b+
No. of Reads: 634 210098 284036
No. of Writes: 631 372 386
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 1782229 2067791 1914912
No. of Writes: 147 111 261
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 3797675 2091925 3298736
No. of Writes: 17603 2566 996
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1287459 1486233 381139
No. of Writes: 720 1033 11386
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 3571 FORGET
0.00 0.00 us 0.00 us 0.00 us 6621022 RELEASE
0.00 0.00 us 0.00 us 0.00 us 5012368 RELEASEDIR
0.00 55.00 us 55.00 us 55.00 us 1 SETXATTR
0.00 66.00 us 66.00 us 66.00 us 1 SETATTR
0.00 58.00 us 45.00 us 71.00 us 2 ENTRYLK
0.00 206.00 us 206.00 us 206.00 us 1 MKDIR
0.00 74.50 us 38.00 us 109.00 us 6 INODELK
0.00 65.58 us 13.00 us 562.00 us 118 READDIR
0.00 61.71 us 17.00 us 360.00 us 143 GETXATTR
0.01 54.48 us 20.00 us 471.00 us 746 STATFS
0.08 59.82 us 1.00 us 2510.00 us 5895 OPENDIR
0.09 51.75 us 11.00 us 5992.00 us 7837 FSTAT
0.33 59.02 us 11.00 us 4863.00 us 25459 STAT
0.41 76.79 us 15.00 us 13114.00 us 23838 READ
0.42 420.23 us 17.00 us 3692.00 us 4466 READDIRP
21.71 43.76 us 2.00 us 20153.00 us 2227256 FLUSH
36.89 73.52 us 16.00 us 37861.00 us 2252806 LOOKUP
40.06 72.61 us 22.00 us 17262.00 us 2476772 OPEN
Duration: 174493 seconds
Data Read: 347482097249 bytes
Data Written: 1739421118 bytes
More information about the Gluster-users
mailing list