[Gluster-users] gluster rdma

[Derek Yarnell]

Hi,

So I wanted to test a gluster install w/ RDMA only support.  RDMA is
working with a successful running of ib_write_bw test between both
nodes.  After I start the gluster daemons I can no longer run the
ib_write_bw tests and also gluster is showing errors on startup,

[2012-01-16 18:27:58.199062] I [graph.c:268:gf_add_cmdline_options]
0-management: adding option 'upgrade' for volume 'management' with value
'on'
[2012-01-16 18:27:58.199105] I [glusterd.c:574:init] 0-management: Using
/etc/glusterd as working directory
[2012-01-16 18:27:58.199707] E [rpc-transport.c:261:rpc_transport_load]
0-rpc-transport:
/opt/glusterfs/3.3beta2/lib64/glusterfs/3.3beta2/rpc-transport/rdma.so:
cannot open shared object file: No such file or directory
[2012-01-16 18:27:58.199729] E [rpc-transport.c:265:rpc_transport_load]
0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not
valid or not found on this machine
[2012-01-16 18:27:58.199736] W [rpcsvc.c:1320:rpcsvc_transport_create]
0-rpc-service: cannot create listener, initing the transport failed
[2012-01-16 18:27:58.199788] I [glusterd.c:89:glusterd_uuid_init]
0-glusterd: retrieved UUID: e76627a1-4d1b-4d96-beef-ad5811970faf
[2012-01-16 18:27:58.200437] I
[glusterd.c:294:glusterd_check_gsync_present] 0-: geo-replication module
not installed in the system
Given volfile:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option working-directory /etc/glusterd
  4:     option transport-type socket,rdma
  5:     option transport.socket.keepalive-time 10
  6:     option transport.socket.keepalive-interval 2
  7:     option transport.socket.read-fail-log off
  8: end-volume

+------------------------------------------------------------------------------+
[2012-01-16 18:28:08.202896] W [glusterfsd.c:750:cleanup_and_exit]
(-->/lib64/libc.so.6(clone+0x6d) [0x3020ad44bd]
(-->/lib64/libpthread.so.0 [0x302160673d]
(-->glusterd(glusterfs_sigwaiter+0x17c) [0x404a0c]))) 0-: received
signum (15), shutting down
[2012-01-16 18:28:18.248888] I [glusterd.c:574:init] 0-management: Using
/etc/glusterd as working directory
[2012-01-16 18:28:18.271281] E [rdma.c:198:rdma_new_post]
0-rpc-transport/rdma: memory registration failed
[2012-01-16 18:28:18.271329] E [rdma.c:2341:__rdma_create_posts]
0-rpc-transport/rdma: rdma.management: post creation failed
[2012-01-16 18:28:18.273491] E [rdma.c:3861:rdma_get_device]
0-rpc-transport/rdma: rdma.management: could not allocate posts
[2012-01-16 18:28:18.273512] E [rdma.c:3984:rdma_init]
0-rpc-transport/rdma: could not create rdma device for ipath0
[2012-01-16 18:28:18.273521] E [rdma.c:4806:init] 0-rdma.management:
Failed to initialize IB Device
[2012-01-16 18:28:18.273531] E [rpc-transport.c:325:rpc_transport_load]
0-rpc-transport: 'rdma' initialization failed
[2012-01-16 18:28:18.273542] W [rpcsvc.c:1320:rpcsvc_transport_create]
0-rpc-service: cannot create listener, initing the transport failed
[2012-01-16 18:28:18.273621] I [glusterd.c:89:glusterd_uuid_init]
0-glusterd: retrieved UUID: e76627a1-4d1b-4d96-beef-ad5811970faf
[2012-01-16 18:28:18.275417] I
[glusterd.c:294:glusterd_check_gsync_present] 0-: geo-replication module
not installed in the system
Given volfile:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option working-directory /etc/glusterd
  4:     option transport-type socket,rdma
  5:     option transport.socket.keepalive-time 10
  6:     option transport.socket.keepalive-interval 2
  7:     option transport.socket.read-fail-log off
  8: end-volume

+------------------------------------------------------------------------------+

Any pointers? I have tried both 3.2.5 and 3.3b2.

Thanks,
derek

-- 
---
Derek T. Yarnell
University of Maryland
Institute for Advanced Computer Studies