landman at scalableinformatics.com
Wed Apr 20 19:27:39 UTC 2011
On 04/20/2011 03:12 PM, Mohit Anchlia wrote:
> I guess I am not clear enough :) I am running multiple concurrent
> threads (in parallel) writing actual files and OVERALL combined
> throughput (from all the threads) never go above 20MB/s. It's not just
> running one single dd at a time.
> gluster volume info all
> Volume Name: stress-volume
> Type: Distributed-Replicate
> Status: Started
> Number of Bricks: 3 x 2 = 6
> Transport-type: tcp
> Brick1: dsdb1:/data/gluster
> Brick2: dsdb2:/data/gluster
> Brick3: dsdb3:/data/gluster
> Brick4: dsdb4:/data/gluster
> Brick5: dsdb5:/data/gluster
> Brick6: dsdb6:/data/gluster
Ok ... your write bandwidth is an example of something called a zero sum
game. No matter how many threads you throw at it, you will never exceed
the maximum bandwidth (sum of the bandwidths per thread).
In the simplest case, you have 1 write thread which will get 20 MB/s.
With two simultaneous writers, each may get ~10 MB/s. 4 simultaneous
would be about 5 MB/s.
Now add in the mirroring operation while doing the write. You have 1
write and one mirror operation going. Thats 10MB/s per thread. 10 for
the write, and 10 for the mirror.
What I am getting at is that mirroring is not cheap on the bandwidth
level of the system. Given the low performance of this unit to begin
with, mirroring is going to be problematic in terms of the amount of
bandwidth consumed and thus available for other processes.
Take the fio input decks I indicated in the previous post, and vary the
number of jobs from 1 to say 24. Please report what happens to the
aggregate write bandwidth (it will report it). You only need a few
points, say 1, 4, 8, 12, 16, 20, 24 to see the effect.
Assuming my guess is correct, the bandwidth won't be constant, but will
drop rapidly with the oversubscription of write threads. You will get 2
writes for every fs write. So at some point the bandwidth will drop
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web : http://scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
More information about the Gluster-users