[Gluster-users] Throughout over infiniband

Mon Sep 10 15:03:21 UTC 2012

<html><body>Everyone,

  This is just a response to the issue of nfs vs glusterfs and the performance for glsuter as I think some of the information may be useful here and has not been discussed.

  For the sake of clarity, I do not run infiniband..but I am running 10GB. My normal production speeds sit around 600MB/s to 700MB/s with the native gluster client.

My setup has 12 servers each with a single 24 disk sata raid 5 10TB brick.  Gluster setup of 6x2.

  Before I settled on this setup I run extensive tests over about 6 weeks to confirm my setup...  in my case glusterfs native out performed NFS considerably in aggretgate data xfers....I also found that the peak performance of the glusterfs client in my setup was at about 12 servers.  This distributed the write and read loads very well....after 12 servers adding more produces diminishing returns.

  I bring this up as no one has been talking about how their brick setup may be effecting the performance as well as the number of servers hosting bricks....Putting multiple bricks on a single server does not increase the load capacity anywhere near like adding another server with the additional brick.

The point I wanted to make was you need to look at all sides of your setup in order to get the best performance...In my case this involved evaluating the raid setup (tuning the size of blocks), the file system used for the brick on the raid (and tuning it, specifically for the size and types of files be manipulated), the memory and cpu in the servers, the network BW, and the client access method.   I had to look at all of these (and each of them had an impact on the final performance numbers) before I found my best setup.....I do not think you can just unilaterally dismiss the gluster setup until you have done a COMPLETE analysis for how to best setup your environment.

Just sharing my thoughts as when I first was setting up gluster I thought I could just install it and tweak the options and be good to go but once I understood everything it is dependent on and addressed all of those options and tuning as well....I significantly improved my overall performance to well over what I was able to achieve with nfs.

Feel free to shoot me with comments or questions.

Bryan Washer

-----Original Message-----
From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Fernando Frediani (Qube)
Sent: Monday, September 10, 2012 8:14 AM
To: 'Stephan von Krawczynski'; 'Whit Blauvelt'
Cc: 'gluster-users at gluster.org'; 'Brian Candler'
Subject: Re: [Gluster-users] Throughout over infiniband

Well, I would say there is a reason, if the Gluster client performed as expected.
Using the Gluster client it should in theory access the file(s) directly from the nodes where they reside and not having to go though a single node exporting the NFS folder which would then have to gather the file.
Yes the NFS has all the caching stuff but if the Gluster client behaviour was  similar it should be able to get similar performance which doesn't seem to be what has been resported.
I did tests myself using Gluster client and NFS and NFS got better performance also and I believe this is due the caching.

Fernando

-----Original Message-----
From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Stephan von Krawczynski
Sent: 10 September 2012 13:57
To: Whit Blauvelt
Cc: gluster-users at gluster.org; Brian Candler
Subject: Re: [Gluster-users] Throughout over infiniband

On Mon, 10 Sep 2012 08:06:51 -0400
Whit Blauvelt <whit.gluster at transpect.com> wrote:

> On Mon, Sep 10, 2012 at 11:13:11AM +0200, Stephan von Krawczynski wrote:
> > [...]
> > If you're lucky you reach something like 1/3 of the NFS performance.
> [Gluster NFS Client]
> Whit

There is a reason why one would switch from NFS to GlusterFS, and mostly it is redundancy. If you start using a NFS-client type you cut yourself off the "complete solution". As said elsewhere you can as well export GlusterFS via kernel-nfs-server. But honestly, it is a patch. It would be better by far if things are done right, native glusterfs client in kernel-space.
And remember, generally there should be no big difference between NFS and GlusterFS with bricks spread over several networks - if it is done how it should be, without userspace.

--
MfG,
Stephan
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

NOTICE: This email and any attachments may contain confidential and proprietary information of NetSuite Inc. and is for the sole use of the intended recipient for the stated purpose. Any improper use or distribution is prohibited. If you are not the intended recipient, please notify the sender; do not review, copy or distribute; and promptly delete or destroy all transmitted information. Please note that all communications and information transmitted through this email system may be monitored by NetSuite or its agents and that all incoming email is automatically scanned by a third party spam and filtering service

</body></html>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120910/048fbb5e/attachment.html>