[Gluster-users] 3.10.5 vs 3.12.0 huge performance loss

[Shyam Ranganathan]

On 09/06/2017 05:48 AM, Serkan Çoban wrote:
> Hi,
> 
> Just do some ingestion tests to 40 node 16+4EC 19PB single volume.
> 100 clients are writing each has 5 threads total 500 threads.
> With 3.10.5 each server has 800MB/s network traffic, cluster total is 32GB/s
> With 3.12.0 each server has 200MB/s network traffic, cluster total is 8GB/s
> I did not change any volume options in both configs.

I just performed some *basic* IOZone tests on a 6 x (4+2) disperse 
volume and compared this against 3.10.5 and 3.12.0. The tests are no 
where near your capacity, but I do not see anything alarming in the 
results. (4 server, 4 clients, 4 worker thread per client)

I do notice a 6% drop in Sequential and random write performance, and 
gains in the sequential and random reads.

I need to improve the test to do larger files and for a longer duration, 
hence not reporting any numbers as yet.

Tests were against 3.10.5 and then a down server upgrade to 3.12.0 and 
remounting on the clients (after the versions were upgraded there as well).

I guess your test can be characterized as a sequential write workload 
(ingestion of data). What is the average file size being ingested? I can 
mimic something equivalent to that to look at this further.

I would like to ensure there are no evident performance regressions as 
you report.

Shyam