[Gluster-devel] successive bonnie++ tests taking longer and longer to run (system load steadily increasing)
Raghavendra G
raghavendra at gluster.com
Mon Mar 22 09:44:37 UTC 2010
Hi Daniel,
We ran some performance tests in our lab, with reasonable cache-size for
io-cache (so that it won't consume all the memory present in the system).
setup: A simple client/server configuration with/without io-cache on client
side.
====
Here are the results.
without io-cache:
=============
[root at client05 test]# time bonnie++ -u anush
Using uid:1013, gid:1013.
Version 1.03c ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
client05 4G 24972 36 28187 1 13149 1 26898 40 30860 1
234.5 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
16 1226 2 4956 2 1806 2 1201 3 5522 3
1874 3
with io-cache:
===========
Version 1.03c ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
client05 4G 24920 37 28529 1 4576 0 36690 52 61428 3
186.2 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
16 1230 2 4945 3 1816 3 1134 2 5576 5
1906 3
As you can see the test which is most affected by the presence of io-cache
is Rewrite. During rewrite, bonnie++ reads an 8kb block, dirties it and
writes back to same location in file. This test consists of 3 system calls -
read, lseek and write. Since cache is flushed at each write, effectively
data is always read from backend filesystem. However the difference is that,
since io-cache stores data in pages of 128KB each, it also does reads in
terms of 128KB block sizes. This it does in the hope that the overhead
involved in reading 128KB blocks (instead of in block-sizes what fuse has
asked) is acceptable, since successive reads can be fetched from cache.
Hence every read of 8KB block size from bonnie++ is transformed into a read
of 128KB block-size is io-cache is present. This is the cause of lower
performance when io-cache is present during rewrites.
To confirm what we had suspected, we changed the page-size in io-cache to
8KB and following are the results.
without io-cache:
=============
[root at client05 test]# bonnie++ -u anush
Version 1.03c ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
client05 4G 24917 35 28243 1 13130 1 27295 39 30836 1
239.8 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
16 1224 2 4949 2 1798 2 1186 2 5509 3
1876 2
with io-cache:
==========
[root at client05 test]# bonnie++ -u
anush
Version 1.03c ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
client05 4G 24925 36 28019 2 12786 1 31224 44 28096 1
220.8 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
16 1192 3 4897 2 1784 3 1215 3 5477 4
1842 3
As we can see, there is not much difference in Rewrite performance
with/without io-cache. Hence if there are writes to the same offsets from
which reads have been done, data will always be read from back-end and
overhead of io-cache can be significant. In these scenarios io-cache is not
recommended. However for other cases, such as mostly read-only operations,
performance boost from io-cache can be significant.
regards,
On Tue, Mar 2, 2010 at 1:30 PM, Daniel Maher
<dma+gluster at witbe.net<dma%2Bgluster at witbe.net>
> wrote:
> Daniel Maher wrote:
>
>> Raghavendra G wrote:
>>
>>>
>>> However, at two points during the multi-day test run, something
>>> strange happened. The time to completion dropped _dramatically_,
>>> and stayed there for numerous iterations, before jumping back up again
>>> :
>>>
>>>
>>> Mostly reads are being served from io-cache?
>>>
>>
>> Perhaps ; it is worth noting that even though the operations are
>> consistent, the data are being generated randomly. I concede that,
>> statistically speaking, some of those 0's and 1's would be cached
>> effectively, but this shouldn't account for a sudden ~ 50% increase in
>> efficiency that, just as suddenly as it appears, disappears again.
>>
>> While it is irresponsible to extrapolate based on three points, my newest
>> test run with io-cache disabled has yielded 10m30s, 10m36s, and 10m34s so
>> far...
>>
>
> After hundreds of iterations the average « real » time per run was
> 10m25.522s . This was with io-cache totally disabled.
>
> Thus, it has been shown that given a series of systematic read and write
> operations on progressively larger files filled with random data, the usage
> of io-cache is not appropriate (and will cause severe performance problems).
>
> Of course, one could have postulated this intuitively - but there's nothing
> like some hard data to back up a hypothesis. :)
>
> The real mystery is why the test with a small io-cache yielded two groups
> of highly varient TTCs...
>
>
>
> --
> Daniel Maher <dma+gluster AT witbe DOT net>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
--
Raghavendra G
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20100322/ec4e8220/attachment-0003.html>
More information about the Gluster-devel
mailing list