[Gluster-users] RDMA Problems with GlusterFS 3.1.1

[Anand Avati]

Can you verify that ibv_srq_pingpong works from the server where this log
file is from?

Thanks,
Avati

On Wed, Dec 1, 2010 at 7:44 PM, Jeremy Stout <stout.jeremy at gmail.com> wrote:

> Whenever I try to start or mount a GlusterFS 3.1.1 volume that uses
> RDMA, I'm seeing the following error messages in the log file on the
> server:
> [2010-11-30 18:37:53.51270] I [nfs.c:652:init] nfs: NFS service started
> [2010-11-30 18:37:53.51362] W [dict.c:1204:data_to_str] dict: @data=(nil)
> [2010-11-30 18:37:53.51375] W [dict.c:1204:data_to_str] dict: @data=(nil)
> [2010-11-30 18:37:53.59628] E [rdma.c:2066:rdma_create_cq]
> rpc-transport/rdma: testdir-client-0: creation of send_cq failed
> [2010-11-30 18:37:53.59851] E [rdma.c:3771:rdma_get_device]
> rpc-transport/rdma: testdir-client-0: could not create CQ
> [2010-11-30 18:37:53.59925] E [rdma.c:3957:rdma_init]
> rpc-transport/rdma: could not create rdma device for mthca0
> [2010-11-30 18:37:53.60009] E [rdma.c:4789:init] testdir-client-0:
> Failed to initialize IB Device
> [2010-11-30 18:37:53.60030] E [rpc-transport.c:971:rpc_transport_load]
> rpc-transport: 'rdma' initialization failed
>
> On the client, I see:
> [2010-11-30 18:43:49.653469] W [io-stats.c:1644:init] testdir:
> dangling volume. check volfile
> [2010-11-30 18:43:49.653573] W [dict.c:1204:data_to_str] dict: @data=(nil)
> [2010-11-30 18:43:49.653607] W [dict.c:1204:data_to_str] dict: @data=(nil)
> [2010-11-30 18:43:49.736275] E [rdma.c:2066:rdma_create_cq]
> rpc-transport/rdma: testdir-client-0: creation of send_cq failed
> [2010-11-30 18:43:49.736651] E [rdma.c:3771:rdma_get_device]
> rpc-transport/rdma: testdir-client-0: could not create CQ
> [2010-11-30 18:43:49.736689] E [rdma.c:3957:rdma_init]
> rpc-transport/rdma: could not create rdma device for mthca0
> [2010-11-30 18:43:49.736805] E [rdma.c:4789:init] testdir-client-0:
> Failed to initialize IB Device
> [2010-11-30 18:43:49.736841] E
> [rpc-transport.c:971:rpc_transport_load] rpc-transport: 'rdma'
> initialization failed
>
> This results in an unsuccessful mount.
>
> I created the mount using the following commands:
> /usr/local/glusterfs/3.1.1/sbin/gluster volume create testdir
> transport rdma submit-1:/exports
> /usr/local/glusterfs/3.1.1/sbin/gluster volume start testdir
>
> To mount the directory, I use:
> mount -t glusterfs submit-1:/testdir /mnt/glusterfs
>
> I don't think it is an Infiniband problem since GlusterFS 3.0.6 and
> GlusterFS 3.1.0 worked on the same systems. For GlusterFS 3.1.0, the
> commands listed above produced no error messages.
>
> If anyone can provide help with debugging these error messages, it
> would be appreciated.
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>