[Gluster-users] Starting RDMA volume fails

[Jochen Becker]

Hi folks,

I have some problems starting a replica volume on a two node infiniband 
setup.
Both systems are running the same hardware and Infiniband (ipoib, 
ibverbs) seems to work well.

OS is Centos 7.1 fresh install and updated, Mellanox Ofed is in use, 
openibd is running both gluster peers are in connected state with each 
other. Creating the volume was no problem at all, but starting always 
fails. Using the force option the volume seems to be started but cannot 
be mounted.

Here are the entries of mnt-bricks-instances.log that happen during the 
command gluster volume start instances:

[2015-11-09 22:01:00.153360] I [MSGID: 100030] [glusterfsd.c:2318:main] 
0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 
3.7.5 (args: /usr/sbin/glusterfsd -s compute02 --volfile-id 
instances.compute02.mnt-bricks-instances -p 
/var/lib/glusterd/vols/instances/run/compute02-mnt-bricks-instances.pid 
-S /var/run/gluster/8f5e59a0b8d5949b51b4c276192b0725.socket --brick-name 
/mnt/bricks/instances -l 
/var/log/glusterfs/bricks/mnt-bricks-instances.log --xlator-option 
*-posix.glusterd-uuid=52109ce5-6173-4d22-bffc-a03c71d24791 --brick-port 
49152 --xlator-option instances-server.listen-port=49152 
--volfile-server-transport=rdma)
[2015-11-09 22:01:00.169326] I [MSGID: 101190] 
[event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread 
with index 1
[2015-11-09 22:01:00.177472] I [graph.c:269:gf_add_cmdline_options] 
0-instances-server: adding option 'listen-port' for volume 
'instances-server' with value '49152'
[2015-11-09 22:01:00.177519] I [graph.c:269:gf_add_cmdline_options] 
0-instances-posix: adding option 'glusterd-uuid' for volume 
'instances-posix' with value '52109ce5-6173-4d22-bffc-a03c71d24791'
[2015-11-09 22:01:00.177826] I [MSGID: 115034] 
[server.c:403:_check_for_auth_option] 0-/mnt/bricks/instances: skip 
format check for non-addr auth option auth.login./mnt/bricks/instances.allow
[2015-11-09 22:01:00.177913] I [MSGID: 115034] 
[server.c:403:_check_for_auth_option] 0-/mnt/bricks/instances: skip 
format check for non-addr auth option 
auth.login.9d64b3ec-9d24-41ac-ba84-cf58c67c9b21.password
[2015-11-09 22:01:00.177916] I [MSGID: 101190] 
[event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread 
with index 2
[2015-11-09 22:01:00.179299] I 
[rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: 
Configured rpc.outstanding-rpc-limit with value 64
[2015-11-09 22:01:00.181636] W [MSGID: 101002] 
[options.c:957:xl_opt_validate] 0-instances-server: option 'listen-port' 
is deprecated, preferred is 'transport.rdma.listen-port', continuing 
with correction
[2015-11-09 22:01:00.183742] W [MSGID: 103071] 
[rdma.c:4592:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event 
channel creation failed [Keine Berechtigung]
[2015-11-09 22:01:00.183782] W [MSGID: 103055] [rdma.c:4899:init] 
0-rdma.instances-server: Failed to initialize IB Device
[2015-11-09 22:01:00.183796] W [rpc-transport.c:359:rpc_transport_load] 
0-rpc-transport: 'rdma' initialization failed
[2015-11-09 22:01:00.183866] W [rpcsvc.c:1597:rpcsvc_transport_create] 
0-rpc-service: cannot create listener, initing the transport failed
[2015-11-09 22:01:00.183884] W [MSGID: 115045] [server.c:1019:init] 
0-instances-server: creation of listener failed
[2015-11-09 22:01:00.183898] E [MSGID: 101019] 
[xlator.c:428:xlator_init] 0-instances-server: Initialization of volume 
'instances-server' failed, review your volfile again
[2015-11-09 22:01:00.183912] E [graph.c:322:glusterfs_graph_init] 
0-instances-server: initializing translator failed
[2015-11-09 22:01:00.183921] E [graph.c:661:glusterfs_graph_activate] 
0-graph: init failed
[2015-11-09 22:01:00.184429] W [glusterfsd.c:1236:cleanup_and_exit] 
(-->/usr/sbin/glusterfsd(mgmt_getspec_cbk+0x331) [0x7f7b6aee02f1] 
-->/usr/sbin/glusterfsd(glusterfs_process_volfp+0x126) [0x7f7b6aedb0f6] 
-->/usr/sbin/glusterfsd(cleanup_and_exit+0x69) [0x7f7b6aeda6d9] ) 0-: 
received signum (0), shutting down

I think the problem starts at 2015-11-09 22:01:00.183742 where the 
channel creation with rdma_cm fails. "Keine Berechtigung" means 
something like missing permissions/rights. Module rdma_cm is loaded and 
I can't find any other problem with the Infiniband or rdma. I have no 
clue what is going wrong here, so any hints on how to proceed are 
appreciated.

Cheers,

Jochen