[Gluster-devel] Native RDMA in libgfapi

[Piotr Rybicki]

W dniu 2015-11-25 o 08:16, Raghavendra Talur pisze:
>
>
> On Fri, Nov 20, 2015 at 9:00 PM, Piotr Rybicki
> <piotr.rybicki at innervision.pl <mailto:piotr.rybicki at innervision.pl>> wrote:
>
>     Hi All.
>
>     Are there any plans for this feature?
>
>
> Yes, it is in the plans. All that is needed to another function in the
> api which lets you define the IO transport to use.
> The functionality is already there, if you use gfapi with a volume which
> was created with only rdma as transport, it would use gfapi as of today.
> For a tcp,rdma volume it would always use tcp and gfapi does not have a
> option of specifying to use rdma transport.
>
> We should have it in 3.8 version of Gluster.

Well, i mean something else.

Actually, when i mount gluster volume via fuse i can choose between RDMA 
and TCP, and it works fine (in either case).

I'm testing libvirtd, qemu and gfapi scenario.
In libvirt, I can also define protocol (TCP/RDMA). When in libvirt i set 
RDMA transport, and gluster volume uses tcp,rdma - then it somehow fails 
over to TCP. When libvirt has RDMA defined, and volume is RDMA only - it 
doesn't work (although fuse mount and gluster bricks interconnect are 
working fine).

That is expected I believe, based on documentation:
https://gluster.readthedocs.org/en/latest/Administrator%20Guide/RDMA%20Transport/

My question is: is native RDMA transport in libgfapi (used in libvirtd 
and qemu) is not yet supported, or there is something wrong with my 
softoware (ofed version 3.12) or hardware (mellanox qdr).

If native RDMA transport in libgfapi is not yet supported, then I'll be 
more than happy to help testing patches/sending debug info.

Unfortunatelly I'm not so fluent in C, especially when programming using 
IB environment.

Best regards
Piotr Rybicki

>
>
>     Just tested latest glusterfs (3.7.6), and it still doesn't work (as
>     expected, since there was no info in changelog about it).
>
>     Native RDMA transport should give a significant boost in
>     performance, based on my observations in fuse mount.
>
>     If that is any of help, I'm more than happy to test patches ;-)
>
>     I'm using Mellanox QDR cards and ofed 3.12.
>
>     Best regards
>     Piotr Rybicki