[Gluster-users] RDMA Problems with GlusterFS 3.1.1

[Jeremy Stout]

Whenever I try to start or mount a GlusterFS 3.1.1 volume that uses
RDMA, I'm seeing the following error messages in the log file on the
server:
[2010-11-30 18:37:53.51270] I [nfs.c:652:init] nfs: NFS service started
[2010-11-30 18:37:53.51362] W [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-11-30 18:37:53.51375] W [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-11-30 18:37:53.59628] E [rdma.c:2066:rdma_create_cq]
rpc-transport/rdma: testdir-client-0: creation of send_cq failed
[2010-11-30 18:37:53.59851] E [rdma.c:3771:rdma_get_device]
rpc-transport/rdma: testdir-client-0: could not create CQ
[2010-11-30 18:37:53.59925] E [rdma.c:3957:rdma_init]
rpc-transport/rdma: could not create rdma device for mthca0
[2010-11-30 18:37:53.60009] E [rdma.c:4789:init] testdir-client-0:
Failed to initialize IB Device
[2010-11-30 18:37:53.60030] E [rpc-transport.c:971:rpc_transport_load]
rpc-transport: 'rdma' initialization failed

On the client, I see:
[2010-11-30 18:43:49.653469] W [io-stats.c:1644:init] testdir:
dangling volume. check volfile
[2010-11-30 18:43:49.653573] W [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-11-30 18:43:49.653607] W [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-11-30 18:43:49.736275] E [rdma.c:2066:rdma_create_cq]
rpc-transport/rdma: testdir-client-0: creation of send_cq failed
[2010-11-30 18:43:49.736651] E [rdma.c:3771:rdma_get_device]
rpc-transport/rdma: testdir-client-0: could not create CQ
[2010-11-30 18:43:49.736689] E [rdma.c:3957:rdma_init]
rpc-transport/rdma: could not create rdma device for mthca0
[2010-11-30 18:43:49.736805] E [rdma.c:4789:init] testdir-client-0:
Failed to initialize IB Device
[2010-11-30 18:43:49.736841] E
[rpc-transport.c:971:rpc_transport_load] rpc-transport: 'rdma'
initialization failed

This results in an unsuccessful mount.

I created the mount using the following commands:
/usr/local/glusterfs/3.1.1/sbin/gluster volume create testdir
transport rdma submit-1:/exports
/usr/local/glusterfs/3.1.1/sbin/gluster volume start testdir

To mount the directory, I use:
mount -t glusterfs submit-1:/testdir /mnt/glusterfs

I don't think it is an Infiniband problem since GlusterFS 3.0.6 and
GlusterFS 3.1.0 worked on the same systems. For GlusterFS 3.1.0, the
commands listed above produced no error messages.

If anyone can provide help with debugging these error messages, it
would be appreciated.