[Gluster-users] rdma.management: could not create QP [Permission denied]

[Thomas Simmons]

Hello All,

I'm having trouble mounting gluster volumes over RDMA. Everything seems to
work fine until I try to mount the gluster volume. At this point it seems
to crash glusterd and give me an error "could not create QP [Permission
Denied]. If I change change the transport to tcp, everything works as
expected. I am pretty new to Infiniband and RDMA, so this may very well be
something I'm doing wrong. That said, I was able to get NFSoRDMA (and
qperf) working on these same hosts, so I'm confident my hardware is
working. This is on CentOS 7.5 using gluster 3.12.15 from the CentOS
repository. I can't find anything regarding this specific error and any
help is much appreciated.

# rdma works...
[root at gluster1 ~]# qperf --use_bits_per_sec gluster2-ib rc_bi_bw
rc_bi_bw:
    bw  =  35.3 Gb/sec

# NFSoRDMA works....
[root at gluster1 data]# mount -t nfs gluster2-ib:/test /mnt/test -o
rdma,port=20049
[root at gluster1 ~]#
[root at gluster1 data]# df -h|grep root
/dev/mapper/centos_gluster1-root   33G  7.8G   26G  24% /
gluster2-ib:/test            33G   29G  4.1G  88% /mnt/test

[root at gluster1 ~]# gluster volume create data replica 3 transport rdma
gluster1-ib:/zp1/data vmh2-ib:/zp1/data vmh3-ib:/zp1/data force
volume create: data: success: please start the volume to access data
[root at gluster1 ~]#
[root at gluster1 ~]# gluster volume start data
volume start: data: success
[root at gluster1 ~]#
[root at gluster1 ~]# gluster volume info data

Volume Name: data
Type: Replicate
Volume ID: 6f32999c-2d20-4fc2-838b-82972171df7e
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: rdma
Bricks:
Brick1: gluster1-ib:/zp1/data
Brick2: gluster2-ib:/zp1/data
Brick3: gluster3-ib:/zp1/data
Options Reconfigured:
nfs.disable: on
performance.client-io-threads: off
[root at gluster1 ~]#
[root at gluster1 ~]# mount -t glusterfs -o transport=rdma gluster1:/data
/mnt/data
Mount failed. Please check the log file for more details.
[root at gluster1 ~]#
[root at gluster1 ~]# gluster volume info data
Connection failed. Please check if gluster daemon is operational.
[root at gluster1 ~]#
[root at gluster1 ~]# tail -n 30 /var/log/messages|grep glusterd
Nov 10 11:18:07 gluster1 glusterd[1881]: [2018-11-10 16:18:07.457818] C
[MSGID: 103021] [rdma.c:3263:gf_rdma_create_qp] 0-rdma.management:
rdma.management: could not create QP [Permission denied]
Nov 10 11:18:07 gluster1 glusterd[1881]: pending frames:
Nov 10 11:18:07 gluster1 glusterd[1881]: patchset: git://
git.gluster.org/glusterfs.git
Nov 10 11:18:07 gluster1 glusterd[1881]: signal received: 11
Nov 10 11:18:07 gluster1 glusterd[1881]: time of crash:
Nov 10 11:18:07 gluster1 glusterd[1881]: 2018-11-10 16:18:07
Nov 10 11:18:07 gluster1 glusterd[1881]: configuration details:
Nov 10 11:18:07 gluster1 glusterd[1881]: argp 1
Nov 10 11:18:07 gluster1 glusterd[1881]: backtrace 1
Nov 10 11:18:07 gluster1 glusterd[1881]: dlfcn 1
Nov 10 11:18:07 gluster1 glusterd[1881]: libpthread 1
Nov 10 11:18:07 gluster1 glusterd[1881]: llistxattr 1
Nov 10 11:18:07 gluster1 glusterd[1881]: setfsid 1
Nov 10 11:18:07 gluster1 glusterd[1881]: spinlock 1
Nov 10 11:18:07 gluster1 glusterd[1881]: epoll.h 1
Nov 10 11:18:07 gluster1 glusterd[1881]: xattr.h 1
Nov 10 11:18:07 gluster1 glusterd[1881]: st_atim.tv_nsec 1
Nov 10 11:18:07 gluster1 glusterd[1881]: package-string: glusterfs 3.12.15
Nov 10 11:18:07 gluster1 glusterd[1881]: ---------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181110/745e58e2/attachment.html>