[Gluster-users] Performance

[Joe Landman]

On 04/20/2011 03:12 PM, Mohit Anchlia wrote:
> I guess I am not clear enough :) I am running multiple concurrent
> threads (in parallel) writing actual files and OVERALL combined
> throughput (from all the threads) never go above 20MB/s. It's not just
> running one single dd at a time.
>
>   gluster volume info all
>
> Volume Name: stress-volume
> Type: Distributed-Replicate
> Status: Started
> Number of Bricks: 3 x 2 = 6
> Transport-type: tcp
> Bricks:
> Brick1: dsdb1:/data/gluster
> Brick2: dsdb2:/data/gluster
> Brick3: dsdb3:/data/gluster
> Brick4: dsdb4:/data/gluster
> Brick5: dsdb5:/data/gluster
> Brick6: dsdb6:/data/gluster

Ok ... your write bandwidth is an example of something called a zero sum 
game.  No matter how many threads you throw at it, you will never exceed 
the maximum bandwidth (sum of the bandwidths per thread).

In the simplest case, you have 1 write thread which will get 20 MB/s. 
With two simultaneous writers, each may get ~10 MB/s.  4 simultaneous 
would be about 5 MB/s.

Now add in the mirroring operation while doing the write.  You have 1 
write and one mirror operation going.  Thats 10MB/s per thread.  10 for 
the write, and 10 for the mirror.

What I am getting at is that mirroring is not cheap on the bandwidth 
level of the system.  Given the low performance of this unit to begin 
with, mirroring is going to be problematic in terms of the amount of 
bandwidth consumed and thus available for other processes.

Take the fio input decks I indicated in the previous post, and vary the 
number of jobs from 1 to say 24.  Please report what happens to the 
aggregate write bandwidth (it will report it).  You only need a few 
points, say 1, 4, 8, 12, 16, 20, 24 to see the effect.

Assuming my guess is correct, the bandwidth won't be constant, but will 
drop rapidly with the oversubscription of write threads.  You will get 2 
writes for every fs write.  So at some point the bandwidth will drop 
rapidly.



-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615