[Gluster-users] Performance issues with striped volume over Infiniband

[Ionescu, A.]

Michael,

Thanks for your suggestion. I had the same intuition as you, but then I used "iptraf" and saw no eth0 traffic associated with I/O on the Gluster volume. (the tool doesn't show the ib0 interface, unfortunately). node01 and node02 are entered into /etc/hosts, they resolve to the ipoib addresses and are pingable.

I will try increasing the number of threads and applying the patch Bryan suggested.

Thanks,
Adrian
________________________________________
From: Michael Mayer [michael at mayer.cx]
Sent: 21 April 2012 10:21
To: Bryan Whitehead
Cc: Ionescu, A.; gluster-users at gluster.org
Subject: Re: [Gluster-users] Performance issues with striped volume over Infiniband

Adrian,

I think you should really try to restart from scratch, i.e.
- setup ipoib with corresponding entries in DNS or /etc/hosts (make sure
the IB hosts are pingable)
- gluster peer probe on the ipoib hostnames
- recreate volume using ipoib hostnames for the bricks
For transport you can still use tcp (I usually set tcp,rdma to be
flexible later on)

I started naively myself a couple of weeks ago and did exactly the same
than you assuming that IB transport would be autodetected. In fact, the
100 MB/s you are seeing indicated very strongly that any communication
is still over Gigabit Ethernet (Which has a theoretical peak performance
of 125 MB/s but given switch quality and protocol overhead you will
reach more like 100-110 MB/s in reality)

Michael.

On 04/20/2012 07:42 PM, Bryan Whitehead wrote:
> Max out the number of IO threads and apply a patch to make gluster
> more agressive about spawning threads as in this thread:
>
> http://gluster.org/pipermail/gluster-users/2012-February/009590.html
>
> (the above thread is actually pretty good for getting performance out
> of gluster with infiniband (I use IP over IB myself - could never get
> the native rDMA to be fast enough).
>
> On Fri, Apr 20, 2012 at 3:27 AM, Ionescu, A.<a.ionescu at student.vu.nl>  wrote:
>> I tried this, but the performance is still the same (about ~100MB/s).
>>
>> A strange thing I noticed is that the node on which I run the benchmarking tool reports ~350 MB/s traffic on the loopback interface.
>>
>> Does anyone have any other suggestions?
>>
>> Thanks,
>> Adrian
>> ________________________________________
>> From: gluster-users-bounces at gluster.org [gluster-users-bounces at gluster.org] on behalf of Sabuj Pattanayek [sabujp at gmail.com]
>> Sent: 19 April 2012 22:32
>> To: gluster-users at gluster.org
>> Subject: Re: [Gluster-users] Performance issues with striped volume over        Infiniband
>>
>> On Thu, Apr 19, 2012 at 2:00 PM, Ionescu, A.<a.ionescu at student.vu.nl>  wrote:
>>> Thanks for your answer, Sabuj. However, I am not sure I understand what you mean by trying with ipoib.
>>> Do you mean specifying "transport tcp" and using the ipoib ips/hostnames for the bricks?
>>> If you manage to find a way to overcome this performance limitation, please share with us your solution.
>> Yes.
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users