[Gluster-users] Gluster 3.3.0 on CentOS 6 - GigabitEthernet vs InfiniBand

[Bartek Krawczyk]

On 18 October 2012 08:44, Ling Ho <ling at slac.stanford.edu> wrote:
> When you mount using rdma, try running some network tools like iftop to see
> if traffics is going through your Ge interface.
>
> If your volume is created with both tcp and rdma, my experience is rdma does
> not work under 3.3.0 and it will always fall back to tcp.
>
> However ipoib works fine for us. Again you should check where the traffics
> go.
I used tcpdump and iftop and confirmed the traffic using IPoIB goes
through ib0 interface, not eth.
When I use rdma transport the traffic doesn't show on neither eth nor
ib0 interface - so I guess it's correctly using RDMA.
I re-ran the iozone -a tests and they're the same. In addition I did a
"dd" test fo read and write on IPoIB and RDMA mounted volume.

IPoIB:
[root at master gluster3]# dd if=/dev/zero of=test bs=100M count=50
50+0 przeczytanych recordów
50+0 zapisanych recordów
skopiowane 5242880000 bajtów (5,2 GB), 16,997 s, 308 MB/s

[root at master gluster3]# dd if=test of=/dev/null
10240000+0 przeczytanych recordów
10240000+0 zapisanych recordów
skopiowane 5242880000 bajtów (5,2 GB), 28,4185 s, 184 MB/s


RDMA:
[root at master gluster]# dd if=/dev/zero of=test bs=100M count=50
50+0 przeczytanych recordów
50+0 zapisanych recordów
skopiowane 5242880000 bajtów (5,2 GB), 70,3636 s, 74,5 MB/s

[root at master gluster]# dd if=test of=/dev/null
10240000+0 przeczytanych recordów
10240000+0 zapisanych recordów
skopiowane 5242880000 bajtów (5,2 GB), 10,8389 s, 484 MB/s

I did a "sync" between those tests. Funny isn't it? RDMA is much
slower than IPoIB on writing and faster on reading.

I think we'll stick to the IPoIB until something's fixed in glusterfs.
And still - why the results are so similar to Gigabit Ethernet?

Regards

-- 
Bartek Krawczyk
network and system administrator