[Gluster-devel] RDMA transport problems in GLUSTER on host with MIC

[Fedele Stabile]

Hi,
it happens that RDMA gluster transport does not works anymore
after I have configured ibscif virtual connector for Infiniband in a
server with a XeonPHI coprocessor.
I suppose it is possible to change the RDMA device from ibscif that
supports only WARP to qib that is the physical device and has RDMA
support.

I have CentOS 6.6 and GLUSTER 3.8.5, OFED 3.12-1 MPSS 3.5.2 and I have
followed the installation instructions of MPSS_Users_Guide (Revision
3.5) that suggested to remove
compat-rdma-devel and compat-rdma packages.

I have noticed that running the command:
ib_send_bw
gives the following error:

# ib_send_bw

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------
------------------
                    Send BW Test
 Dual-port       : OFF        Device         : scif0
 Number of qps   : 1        Transport type : IW
 Connection type : RC        Using SRQ      : OFF
 RX depth        : 512
 CQ Moderation   : 100
 Mtu             : 2048[B]
 Link type       : Ethernet
 Gid index       : 0
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------
------------------
 local address: LID 0x3e8 QPN 0x0003 PSN 0x123123
 GID: 76:121:186:102:03:119:00:00:00:00:00:00:00:00:00:00
ethernet_read_keys: Couldn't read remote address
 Unable to read to socket/rdam_cm
Failed to exchange data between server and clients

Instead using the output of the command

ib_send_bw -d qib0

gives correct results:

# ib_send_bw -d qib0

************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------
------------------
                    Send BW Test
 Dual-port       : OFF        Device         : qib0
 Number of qps   : 1        Transport type : IB
 Connection type : RC        Using SRQ      : OFF
 RX depth        : 512
 CQ Moderation   : 100
 Mtu             : 2048[B]
 Link type       : IB
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------
------------------
 local address: LID 0x0a QPN 0x0169 PSN 0xe0b768
 remote address: LID 0x20 QPN 0x28b280 PSN 0xc3008c
---------------------------------------------------------------------
------------------
 #bytes     #iterations    BW peak[MB/sec]    BW
average[MB/sec]   MsgRate[Mpps]
 65536      1000             0.00               2160.87           0.034
574
---------------------------------------------------------------------
------------------