[Gluster-devel] successive bonnie++ tests taking longer and longer to run (system load steadily increasing)

[Daniel Maher]

Raghavendra G wrote:
> Hi Daniel,
> 
> Can you decrease the cache size in io-cache (say to 128MB)? If the files 
> being served are bigger than this size, you can as well remove io-cache 
> from configuration. Do let us know if this solves your issue.

Some of the files that are generated during the test are under 128M, and 
some are much larger - effectively, the test deals with larger and 
larger sets of data as it runs its course.

At your suggestion, i changed the io-cache size from the default setting 
(as generated by the configurator) to 128M, and re-ran the test.  The 
results are now much more consistent with what would be expected :

real    21m31.019s
real    21m30.932s
real    21m28.965s
real    21m26.095s
real    21m26.388s
real    21m29.377s
real    21m32.180s
real    21m26.600s
real    21m28.354s
(cut for brevity)

However, at two points during the multi-day test run, something strange 
happened.  The time to completion dropped _dramatically_, and stayed 
there for numerous iterations, before jumping back up again :

real	21m27.806s
real	21m45.434s
real	14m41.468s
real	12m41.172s
real	12m39.779s
real	12m34.595s
real	12m42.223s
real	12m41.111s
(repeat 30 iterations)
real	12m40.185s
real	12m40.517s
real	13m25.394s
real	21m26.375s
real	21m28.476s

I am forced to assume that something else on either the client or the 
server nodes caused this behaviour, but i cannot imagine what.  The 
nodes, frankly, aren't doing anything else.  Very strange.

I am going to remove the io-cache entirely and see what, if any, effect 
that has.

-- 
Daniel Maher <dma+gluster AT witbe DOT net>