[Gluster-users] 3.2.2 Performance Issue

[Burnash, James]

Cogently put and helpful, Joe. Thanks. I'm filing this under "good answers to frequently asked technical questions". You have a number of spots in that archive already :-)

James Burnash
Unix Engineer
Knight Capital Group


-----Original Message-----
From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Joe Landman
Sent: Thursday, August 11, 2011 8:51 AM
To: gluster-users at gluster.org
Subject: Re: [Gluster-users] 3.2.2 Performance Issue

On 08/11/2011 08:28 AM, Stephan von Krawczynski wrote:
> On Wed, 10 Aug 2011 12:08:39 -0700
> Mohit Anchlia<mohitanchlia at gmail.com>  wrote:
>
>> Did you run dd tests on all your servers? Could it be one of the disk is slower?
>>
>> On Wed, Aug 10, 2011 at 10:51 AM, Joey McDonald<joey at scare.org>  wrote:
>>> Hi Joe, thanks for your response!
>>>
>>>>>
>>>>> An order of magnitude slower with replication. What's going on I wonder?
>>>>> Thanks for any suggestions.
>>>>
>>>> You are dealing with contention for Gigabit bandwidth.  Replication
>>>> will do that, and will be pronounced over 1GbE.  Much less of an
>>>> issue over 10GbE or Infiniband.
>
> If that was a GBit contention you can check out by spreading your
> boxes over different switches. That should prevent a contention problem.
> Unfortunately I can tell you it did not help on our side, so we doubt
> the explanation.

Contention over a single port won't be helped by increasing the number of switches.

This isn't that hard to work out for yourself.  If you have 1 constant stream over a 100MB/s link, you'll get close to 100MB/s.

Now have 2 streams operating over this link.  Assuming good balance, and a 100% duty cycle (50% per stream), you'll get, again, 100MB/s used, or
50 MB/s per client.

Now have 6 streams over this link. Assuming good balance, and a 100% duty cycle (16.7% per stream), you'll get, again, 100MB/s used, or 16.7 MB/s per client.  Which, is curiously close to what was observed with a replica 6.  For replica 2, it should be closer to 1/2 the bandwidth.

Note that this analysis assumes full duplex gigabit.  Half duplex would
  divide these results in half.

Note also that these assume reasonably good gigabit NICs.  Some of the lower end NICs, like the Broadcom's shipped in Dell and HP units, might not behave as well under load.

The point of this analysis is that it is very easy to run out of network bandwidth pretty quickly on gigabit networks, so you shouldn't be surprised when you run out of network bandwidth on gigabit networks.
You are contending for a fixed (relatively small) sized resource from N requestors.  On average, you should expect 1/N bandwidth per requester.
  Inclusive of clients.

--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


DISCLAIMER:
This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission.
NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com