[Gluster-users] gluster rdma

[Derek Yarnell]

On 1/16/12 6:33 PM, Joe Landman wrote:
> On 01/16/2012 06:29 PM, Derek Yarnell wrote:
>> Hi,
>>
>> So I wanted to test a gluster install w/ RDMA only support.  RDMA is
>> working with a successful running of ib_write_bw test between both
>> nodes.  After I start the gluster daemons I can no longer run the
>> ib_write_bw tests and also gluster is showing errors on startup,
>>
> 
> 1st question:  Did you upgrade from earlier 3.x?
> 
> 2nd question:  what does your mount command look like?
> 
> For 3.2.x (x>2 or so), you need to mount
> 
>     mount -o intr -t glusterfs server:/volume.rdma /mnt
> 
> for a volume. Note the ".rdma".  Once you have this, make sure you are
> correctly mounted.  Do a df -h and see if it gives you a transport
> disconnected.
> 
> 
> 

Hi,

1) No this is just a new testbed.

2) I am not even to the mount stage yet.  The errors in the log I am
seeing are when I start the management daemon.  I went through the
process of creating the volume but still no dice.

  gluster peer probe 192.168.1.101
  gluster volume create gtest replica 2 transport rdma
192.168.1.100:/scratch0/gluster 192.168.1.101:/scratch0/gluster
  gluster volume start gtest

But the volume is not starting correctly,

# gluster volume status gtest
Brick status for volume: gtest
Brick                                                   Port    Online  PID
---------------------------------------------------------------------------
192.168.1.100:/scratch0/gluster                         24009   N       0
---------------------------------------------------------------------------
192.168.1.101:/scratch0/gluster                         24009   N       0

# cat bricks/scratch0-gluster.log
[2012-01-16 19:17:41.928419] W [socket.c:420:__socket_keepalive]
0-socket: failed to set keep idle on socket 8
[2012-01-16 19:17:41.928485] W
[socket.c:1875:socket_server_event_handler] 0-socket.glusterfsd: Failed
to set keep-alive: Operation not supported
[2012-01-16 19:17:41.942922] I [graph.c:268:gf_add_cmdline_options]
0-gtest-server: adding option 'listen-port' for volume 'gtest-server'
with value '24009'
[2012-01-16 19:17:41.943651] W [options.c:647:xl_opt_validate]
0-gtest-server: option 'listen-port' is deprecated, preferred is
'transport.rdma.listen-port', continuing with correction
[2012-01-16 19:17:41.946893] E [rdma.c:198:rdma_new_post]
0-rpc-transport/rdma: memory registration failed
[2012-01-16 19:17:41.946932] E [rdma.c:2341:__rdma_create_posts]
0-rpc-transport/rdma: rdma.gtest-server: post creation failed
[2012-01-16 19:17:41.948827] E [rdma.c:3861:rdma_get_device]
0-rpc-transport/rdma: rdma.gtest-server: could not allocate posts
[2012-01-16 19:17:41.948848] E [rdma.c:3984:rdma_init]
0-rpc-transport/rdma: could not create rdma device for ipath0
[2012-01-16 19:17:41.948857] E [rdma.c:4806:init] 0-rdma.gtest-server:
Failed to initialize IB Device
[2012-01-16 19:17:41.948867] E [rpc-transport.c:325:rpc_transport_load]
0-rpc-transport: 'rdma' initialization failed
[2012-01-16 19:17:41.948874] W [rpcsvc.c:1320:rpcsvc_transport_create]
0-rpc-service: cannot create listener, initing the transport failed
[2012-01-16 19:17:41.948882] W [server.c:659:init] 0-gtest-server:
creation of listener failed
[2012-01-16 19:17:41.948889] E [xlator.c:393:xlator_init]
0-gtest-server: Initialization of volume 'gtest-server' failed, review
your volfile again
[2012-01-16 19:17:41.948895] E [graph.c:325:glusterfs_graph_init]
0-gtest-server: initializing translator failed
[2012-01-16 19:17:41.948901] E [graph.c:521:glusterfs_graph_activate]
0-graph: init failed
[2012-01-16 19:17:41.949209] W [glusterfsd.c:750:cleanup_and_exit]
(-->/opt/glusterfs/3.3beta2/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2)
[0x2b31e8bd3f02]
(-->/opt/glusterfs/3.3beta2/sbin/glusterfsd(mgmt_getspec_cbk+0xc5)
[0x408365]
(-->/opt/glusterfs/3.3beta2/sbin/glusterfsd(glusterfs_process_volfp+0x17a)
[0x40469a]))) 0-: received signum (0), shutting down


-- 
---
Derek T. Yarnell
University of Maryland
Institute for Advanced Computer Studies