[Gluster-users] Performance gluster 3.2.5 + QLogic Infiniband

[Michael Mayer]

Hi all,

thanks for your suggestions,

i think I have "solved" the performance issue now. I had a few too many 
kernel patches included. I am back to the stock RHEL 5.8 kernel with 
stock QLogic OFED and everything works ok (IPoIB). My original intent 
was to explore cachefs on RHEL5 by building a 2.6.32 kernel but while 
cachefs worked like a treat performance for gluster was as bad as 
reported previously - so will go without cachefs for now and reintroduce 
cachefs in an OS upgrade later on.

I even have a nicely working rdma setup now and - using that - 
performance is 900 MB/s + and that consistently so.

Since I have two volumes exported by the same bricks it seems I only can 
get one of them to use RDMA, the other will then refuse to mount and 
only mount if not using rdma on that one - but that is not a real 
problem for now as the second one is only used for backup purposes.

Michael,

On 04/12/2012 01:13 AM, Fabricio Cannini wrote:
> Hi there
>
> The only time i setup a gluster "distributed scratch" like Michael is 
> doing,
> ( 3.0.5 Debian packages ) i too choose IPoIB simply because i could 
> not get rdma working at all.
> Time was short and IPoIB "Just worked" well enough for our demand at 
> the time, so i didn't looked into this issue. Plus, pinging and 
> ssh'ing into a node through the IB interface comes handy when 
> diagnosing and fixing networking issues.
>
> Em quarta-feira, 11 de abril de 2012, Sabuj 
> Pattanayek<sabujp at gmail.com <mailto:sabujp at gmail.com>> escreveu:
> > I wonder if it's possible to have both rdma and ipoib served by a
> > single glusterfsd so I can test this? I guess so, since it's just a
> > tcp mount?
> >
> > On Wed, Apr 11, 2012 at 1:43 PM, Harry Mangalam 
> <harry.mangalam at uci.edu <mailto:harry.mangalam at uci.edu>> wrote:
> >> On Tuesday 10 April 2012 15:47:08 Bryan Whitehead wrote:
> >>
> >>> with my infiniband setup I found my performance was much better by
> >>
> >>> setting up a TCP network over infiniband and then using pure tcp as
> >>
> >>> the transport with my gluster volume. For the life of me I couldn't
> >>
> >>> get rdma to beat tcp.
> >>
> >> Thanks for that data point, Brian.
> >>
> >> Very interesting. Is this a common experience? The RDMA experience 
> has not
> >> been a very smooth one for me and doing everything with IPoIB would 
> save a
> >> lot of headaches, especially if it's also higher performance.
> >>
> >> hjm
> >>
> >> --
> >>
> >> Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
> >>
> >> [ZOT 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
> >>
> >> 415 South Circle View Dr, Irvine, CA, 92697 [shipping]
> >>
> >> MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
> >>
> >> --
> >>
> >>
> >> _______________________________________________
> >> Gluster-users mailing list
> >> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> >>
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> >
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120421/b40503a3/attachment.html>