[Gluster-users] RDMA transport problems in GLUSTER on host with MIC

[Anoop C S]

On Wed, 2017-01-18 at 12:56 +0100, Fedele Stabile wrote:
> Hi,
> it happens that RDMA gluster transport does not works anymore
> after I have configured ibscif virtual connector for Infiniband in a
> server with a XeonPHI coprocessor.
> 
> I have CentOS 6.6 and GLUSTER 3.8.5, OFED 3.12-1 MPSS 3.5.2 and I have
> followed the installation instructions of MPSS_Users_Guide (Revision
> 3.5) that suggested to remove
> compat-rdma-devel and compat-rdma packages.
> 

It would help if you could somehow clearly understand the reason for removing those packages. May be
they are critical and not intended to be removed. Please ask for help from OFED.

> I have noticed that running the command:
> ib_send_bw
> gives the following error:
> 
> # ib_send_bw
> 
> ************************************
> * Waiting for client to connect... *
> ************************************
> ---------------------------------------------------------------------
> ------------------
>                     Send BW Test
>  Dual-port       : OFF        Device         : scif0
>  Number of qps   : 1        Transport type : IW
>  Connection type : RC        Using SRQ      : OFF
>  RX depth        : 512
>  CQ Moderation   : 100
>  Mtu             : 2048[B]
>  Link type       : Ethernet
>  Gid index       : 0
>  Max inline data : 0[B]
>  rdma_cm QPs     : OFF
>  Data ex. method : Ethernet
> ---------------------------------------------------------------------
> ------------------
>  local address: LID 0x3e8 QPN 0x0003 PSN 0x123123
>  GID: 76:121:186:102:03:119:00:00:00:00:00:00:00:00:00:00
> ethernet_read_keys: Couldn't read remote address
>  Unable to read to socket/rdam_cm
> Failed to exchange data between server and clients
> 

The above error have nothing to do with GlusterFS. Can you please give more context on what failed
for you while trying out GlusterFS with RDMA transport?

> Instead using the output of the command
> 
> ib_send_bw -d qib0
> 
> gives correct results:
> 
> # ib_send_bw -d qib0
> 
> ************************************
> * Waiting for client to connect... *
> ************************************
> ---------------------------------------------------------------------
> ------------------
>                     Send BW Test
>  Dual-port       : OFF        Device         : qib0
>  Number of qps   : 1        Transport type : IB
>  Connection type : RC        Using SRQ      : OFF
>  RX depth        : 512
>  CQ Moderation   : 100
>  Mtu             : 2048[B]
>  Link type       : IB
>  Max inline data : 0[B]
>  rdma_cm QPs     : OFF
>  Data ex. method : Ethernet
> ---------------------------------------------------------------------
> ------------------
>  local address: LID 0x0a QPN 0x0169 PSN 0xe0b768
>  remote address: LID 0x20 QPN 0x28b280 PSN 0xc3008c
> ---------------------------------------------------------------------
> ------------------
>  #bytes     #iterations    BW peak[MB/sec]    BW
> average[MB/sec]   MsgRate[Mpps]
>  65536      1000             0.00               2160.87           0.034
> 574
> ---------------------------------------------------------------------
> ------------------
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users