[Gluster-users] RDMA transport problems in GLUSTER on host with MIC

[Fedele Stabile]

Thank you for your help, 
I will answer to your questions:

Il giorno ven, 20/01/2017 alle 12.58 +0530, Anoop C S ha scritto:
> On Wed, 2017-01-18 at 12:56 +0100, Fedele Stabile wrote:
> > Hi,
> > it happens that RDMA gluster transport does not works anymore
> > after I have configured ibscif virtual connector for Infiniband in
> > a
> > server with a XeonPHI coprocessor.
> > 
> > I have CentOS 6.6 and GLUSTER 3.8.5, OFED 3.12-1 MPSS 3.5.2 and I
> > have
> > followed the installation instructions of MPSS_Users_Guide
> > (Revision
> > 3.5) that suggested to remove
> > compat-rdma-devel and compat-rdma packages.
> > 
> 
> It would help if you could somehow clearly understand the reason for
> removing those packages. May be
> they are critical and not intended to be removed. Please ask for help
> from OFED.
> 
Files of ackages compat-rdma-devel and compat-rdma are substituted by
others from MPSS package that contains all the Software Stack for
server and MIC card including ofed drivers.

> > I have noticed that running the command:
> > ib_send_bw
> > gives the following error:
> > 
> > # ib_send_bw
> > 
> > ************************************
> > * Waiting for client to connect... *
> > ************************************
> > -----------------------------------------------------------------
> > ----
> > ------------------
> >                     Send BW Test
> >  Dual-port       : OFF        Device         : scif0
> >  Number of qps   : 1        Transport type : IW
> >  Connection type : RC        Using SRQ      : OFF
> >  RX depth        : 512
> >  CQ Moderation   : 100
> >  Mtu             : 2048[B]
> >  Link type       : Ethernet
> >  Gid index       : 0
> >  Max inline data : 0[B]
> >  rdma_cm QPs     : OFF
> >  Data ex. method : Ethernet
> > -----------------------------------------------------------------
> > ----
> > ------------------
> >  local address: LID 0x3e8 QPN 0x0003 PSN 0x123123
> >  GID: 76:121:186:102:03:119:00:00:00:00:00:00:00:00:00:00
> > ethernet_read_keys: Couldn't read remote address
> >  Unable to read to socket/rdam_cm
> > Failed to exchange data between server and clients
> > 
> 
> The above error have nothing to do with GlusterFS. Can you please
> give more context on what failed
> for you while trying out GlusterFS with RDMA transport?
In glusterd.vol.log when I start glusterd I see:
[rdma.c:4837:gf_rdma_listen] 0-rdma.management: rdma option set failed
[Funzione non implementata]
But RDMA is correctly working on qib0 device as you can see below:
> 
> > Instead using the output of the command
> > 
> > ib_send_bw -d qib0
> > 
> > gives correct results:
> > 
> > # ib_send_bw -d qib0
> > 
> > ************************************
> > * Waiting for client to connect... *
> > ************************************
> > -----------------------------------------------------------------
> > ----
> > ------------------
> >                     Send BW Test
> >  Dual-port       : OFF        Device         : qib0
> >  Number of qps   : 1        Transport type : IB
> >  Connection type : RC        Using SRQ      : OFF
> >  RX depth        : 512
> >  CQ Moderation   : 100
> >  Mtu             : 2048[B]
> >  Link type       : IB
> >  Max inline data : 0[B]
> >  rdma_cm QPs     : OFF
> >  Data ex. method : Ethernet
> > -----------------------------------------------------------------
> > ----
> > ------------------
> >  local address: LID 0x0a QPN 0x0169 PSN 0xe0b768
> >  remote address: LID 0x20 QPN 0x28b280 PSN 0xc3008c
> > -----------------------------------------------------------------
> > ----
> > ------------------
> >  #bytes     #iterations    BW peak[MB/sec]    BW
> > average[MB/sec]   MsgRate[Mpps]
> >  65536      1000             0.00               2160.87           0
> > .034
> > 574
> > -----------------------------------------------------------------
> > ----
> > ------------------
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users
> 
>