[Gluster-users] glusterfs 3.1.1 rdma module crashing when mounting volume

[Joerg Blank]

Hi all,

We have a small HPC cluster and I tried to harness the spare disk space
of our compute nodes to take of some load from the cluster's nfs server.

I started of using glusterfs 3.0.x packaged with Debian Squeeze (all
nodes use this version)

I tried updating to glusterfs 3.1.x using the prepackaged files [1]
from gluster.org, but found out I was no longer able to use the
Infiniband interconnect, because the packages seem to be compiled
without rdma support.

To get the faster interconnect back I repackaged glusterfs 3.1.1 from
the source tarball and installed it on all nodes. However rdma crashes
when mounting a volume on the head node [2], it works fine from the
compute nodes. The only significant in respect to infiniband is, that
the head node uses another nic:

Work Nodes:
02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0
5GT/s - IB QDR / 10GigE] (rev b0)

Head Node:
06:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx
HCA] (rev 20)


If anyone has an idea how to get this working, please let me know.

Regards,

Jörg Blank


[1] http://download.gluster.com/pub/gluster/glusterfs/3.1/LATEST/Debian/

[2] Backtrace from logs:

[2010-12-24 22:45:11.516902] W [io-stats.c:1644:init] test-volume:
dangling volume. check volfile
[2010-12-24 22:45:11.516943] W [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-12-24 22:45:11.516955] W [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-12-24 22:45:11.527333] E [rdma.c:2066:rdma_create_cq]
rpc-transport/rdma: test-volume-client-1: creation of send_cq failed
[2010-12-24 22:45:11.527529] E [rdma.c:3771:rdma_get_device]
rpc-transport/rdma: test-volume-client-1: could not create CQ
[2010-12-24 22:45:11.527541] E [rdma.c:3957:rdma_init]
rpc-transport/rdma: could not create rdma device for mthca0
[2010-12-24 22:45:11.527611] E [rdma.c:4789:init] test-volume-client-1:
Failed to initialize IB Device
[2010-12-24 22:45:11.527623] E [rpc-transport.c:971:rpc_transport_load]
rpc-transport: 'rdma' initialization failed
pending frames:

patchset: v3.1.1
signal received: 11
time of crash: 2010-12-24 22:45:11
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.1.1
/lib/libc.so.6(+0x321e0)[0x7f8e2c3ea1e0]
/lib/libc.so.6(+0x7a126)[0x7f8e2c432126]
/usr/lib/glusterfs/3.1.1/rpc-transport/rdma.so(init+0x37c)[0x7f8e28956e7c]
/usr/lib/libgfrpc.so.0(rpc_transport_load+0x365)[0x7f8e2cd5a035]
/usr/lib/libgfrpc.so.0(rpc_clnt_new+0xf9)[0x7f8e2cd5de59]
/usr/lib/glusterfs/3.1.1/xlator/protocol/client.so(client_init_rpc+0xa9)[0x7f8e29c32b09]
/usr/lib/glusterfs/3.1.1/xlator/protocol/client.so(init+0xf1)[0x7f8e29c32cb1]
/usr/lib/libglusterfs.so.0(xlator_init+0x58)[0x7f8e2cf7c978]
/usr/lib/libglusterfs.so.0(glusterfs_graph_init+0x35)[0x7f8e2cfa5b05]
/usr/lib/libglusterfs.so.0(glusterfs_graph_activate+0x38)[0x7f8e2cfa5c48]
/usr/sbin/glusterfs(glusterfs_process_volfp+0xba)[0x40447a]
/usr/sbin/glusterfs(mgmt_getspec_cbk+0xc7)[0x405cc7]
/usr/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x7f8e2cd5cb75]
/usr/lib/libgfrpc.so.0(rpc_clnt_notify+0xc9)[0x7f8e2cd5cdc9]
/usr/lib/libgfrpc.so.0(rpc_transport_notify+0x2d)[0x7f8e2cd57d7d]
/usr/lib/glusterfs/3.1.1/rpc-transport/socket.so(socket_event_poll_in+0x34)[0x7f8e2a870c94]
/usr/lib/glusterfs/3.1.1/rpc-transport/socket.so(socket_event_handler+0xb3)[0x7f8e2a870d63]
/usr/lib/libglusterfs.so.0(+0x3a272)[0x7f8e2cf9d272]
/usr/sbin/glusterfs(main+0x247)[0x4054c7]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7f8e2c3d6c4d]
/usr/sbin/glusterfs[0x403179]


------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDirig Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------