[Gluster-users] 3.2.2 Performance Issue

[Joe Landman]

On 08/11/2011 08:28 AM, Stephan von Krawczynski wrote:
> On Wed, 10 Aug 2011 12:08:39 -0700
> Mohit Anchlia<mohitanchlia at gmail.com>  wrote:
>
>> Did you run dd tests on all your servers? Could it be one of the disk is slower?
>>
>> On Wed, Aug 10, 2011 at 10:51 AM, Joey McDonald<joey at scare.org>  wrote:
>>> Hi Joe, thanks for your response!
>>>
>>>>>
>>>>> An order of magnitude slower with replication. What's going on I wonder?
>>>>> Thanks for any suggestions.
>>>>
>>>> You are dealing with contention for Gigabit bandwidth.  Replication will
>>>> do that, and will be pronounced over 1GbE.  Much less of an issue over 10GbE
>>>> or Infiniband.
>
> If that was a GBit contention you can check out by spreading your boxes
> over different switches. That should prevent a contention problem.
> Unfortunately I can tell you it did not help on our side, so we doubt the
> explanation.

Contention over a single port won't be helped by increasing the number 
of switches.

This isn't that hard to work out for yourself.  If you have 1 constant 
stream over a 100MB/s link, you'll get close to 100MB/s.

Now have 2 streams operating over this link.  Assuming good balance, and 
a 100% duty cycle (50% per stream), you'll get, again, 100MB/s used, or 
50 MB/s per client.

Now have 6 streams over this link. Assuming good balance, and a 100% 
duty cycle (16.7% per stream), you'll get, again, 100MB/s used, or 16.7 
MB/s per client.  Which, is curiously close to what was observed with a 
replica 6.  For replica 2, it should be closer to 1/2 the bandwidth.

Note that this analysis assumes full duplex gigabit.  Half duplex would 
  divide these results in half.

Note also that these assume reasonably good gigabit NICs.  Some of the 
lower end NICs, like the Broadcom's shipped in Dell and HP units, might 
not behave as well under load.

The point of this analysis is that it is very easy to run out of network 
bandwidth pretty quickly on gigabit networks, so you shouldn't be 
surprised when you run out of network bandwidth on gigabit networks. 
You are contending for a fixed (relatively small) sized resource from N 
requestors.  On average, you should expect 1/N bandwidth per requester. 
  Inclusive of clients.

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615