[Gluster-users] Performance issues with striped volume over Infiniband

[Michael Mayer]

Adrian,

I think you should really try to restart from scratch, i.e.
- setup ipoib with corresponding entries in DNS or /etc/hosts (make sure 
the IB hosts are pingable)
- gluster peer probe on the ipoib hostnames
- recreate volume using ipoib hostnames for the bricks
For transport you can still use tcp (I usually set tcp,rdma to be 
flexible later on)

I started naively myself a couple of weeks ago and did exactly the same 
than you assuming that IB transport would be autodetected. In fact, the 
100 MB/s you are seeing indicated very strongly that any communication 
is still over Gigabit Ethernet (Which has a theoretical peak performance 
of 125 MB/s but given switch quality and protocol overhead you will 
reach more like 100-110 MB/s in reality)

Michael.

On 04/20/2012 07:42 PM, Bryan Whitehead wrote:
> Max out the number of IO threads and apply a patch to make gluster
> more agressive about spawning threads as in this thread:
>
> http://gluster.org/pipermail/gluster-users/2012-February/009590.html
>
> (the above thread is actually pretty good for getting performance out
> of gluster with infiniband (I use IP over IB myself - could never get
> the native rDMA to be fast enough).
>
> On Fri, Apr 20, 2012 at 3:27 AM, Ionescu, A.<a.ionescu at student.vu.nl>  wrote:
>> I tried this, but the performance is still the same (about ~100MB/s).
>>
>> A strange thing I noticed is that the node on which I run the benchmarking tool reports ~350 MB/s traffic on the loopback interface.
>>
>> Does anyone have any other suggestions?
>>
>> Thanks,
>> Adrian
>> ________________________________________
>> From: gluster-users-bounces at gluster.org [gluster-users-bounces at gluster.org] on behalf of Sabuj Pattanayek [sabujp at gmail.com]
>> Sent: 19 April 2012 22:32
>> To: gluster-users at gluster.org
>> Subject: Re: [Gluster-users] Performance issues with striped volume over        Infiniband
>>
>> On Thu, Apr 19, 2012 at 2:00 PM, Ionescu, A.<a.ionescu at student.vu.nl>  wrote:
>>> Thanks for your answer, Sabuj. However, I am not sure I understand what you mean by trying with ipoib.
>>> Do you mean specifying "transport tcp" and using the ipoib ips/hostnames for the bricks?
>>> If you manage to find a way to overcome this performance limitation, please share with us your solution.
>> Yes.
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users