[Gluster-users] Trouble getting NUFA/IB-VERBS running

[Scandic]

Hi people,
I'm getting in trouble with my first IB setup. The glusterfsd.log  
contains the following information. Any hint where to start?
It's a 2-Node System connected to a 8-Port IB Switch. Linux is CentOS  
5.3 x86_64. Mellanox Drivers are installed.

= 
= 
= 
= 
= 
= 
= 
= 
========================================================================
Version      : glusterfs 2.0.0 built on Apr 24 2009 19:27:41
TLA Revision : 7b2e459db65edd302aa12476bc73b3b7a17b1410
Starting Time: 2009-05-13 01:22:03
Command line : /usr/sbin/glusterfsd -f /etc/glusterfs/glusterfsd.vol
PID          : 15237
System name  : Linux
Nodename     : Node1
Kernel Release : 2.6.18-128.el5
Hardware Identifier: x86_64

Given volfile:
+ 
------------------------------------------------------------------------------+
   1: volume posix
   2:  type storage/posix
   3:  option directory /home/export
   4: end-volume
   5:
   6: volume locks
   7:  type features/locks
   8:  subvolumes posix
   9: end-volume
  10:
  11: volume brick
  12:  type performance/io-threads
  13:  option autoscaling yes
  14:  subvolumes locks
  15: end-volume
  16:
  17: volume server
  18:  type protocol/server
  19:  option transport-type ib-verbs
  20:  #option transport.ib-verbs.listen-port 6990
  21:  option auth.addr.brick.allow *
  22:  subvolumes brick
  23: end-volume
  24:
  25: volume Node1
  26:  type protocol/client
  27:  option transport-type ib-verbs
  28:  #option remote-port 6990
  29:  option remote-host Node1
  30:  option remote-subvolume brick
  31: end-volume
  32:
  33: volume Node2
  34:  type protocol/client
  35:  option transport-type ib-verbs
  36:  #option remote-port 6990
  37:  option remote-host Node2
  38:  option remote-subvolume brick
  39: end-volume
  40:
  41: volume nufa
  42:  type cluster/nufa
  43:  option local-volume-name Node1
  44:  subvolumes Node1 Node2
  45: end-volume
  46:
  47: volume writebehind
  48:   type performance/write-behind
  49:   option page-size 128KB
  50:   option cache-size 1MB
  51:   subvolumes nufa
  52: end-volume
  53:
  54: volume cache
  55:   type performance/io-cache
  56:   option cache-size 512MB
  57:   subvolumes writebehind
  58: end-volume
  59:

+ 
------------------------------------------------------------------------------+

2009-05-13 01:22:03 W [io-cache.c:1342:init] cache: dangling volume.  
check volfile
2009-05-13 01:22:03 N [glusterfsd.c:1152:main] glusterfs: Successfully  
started
2009-05-13 01:22:04 E [ib-verbs.c:1226:ib_verbs_send_completion_proc]  
transport/ib-verbs: send work request on `mthca0' returned error  
wc.status = 12, wc.vendor_err =
  129, post->buf = 0x2aaaab6f1000, wc.byte_len = 0, post->reused = 1
2009-05-13 01:22:04 E [ib-verbs.c:1996:ib_verbs_event_handler]  
transport/ib-verbs: Node2: pollin received on tcp socket (peer:  
192.168.0.192:6997) after handshake is
complete
2009-05-13 01:22:04 E [ib-verbs.c:1226:ib_verbs_send_completion_proc]  
transport/ib-verbs: send work request on `mthca0' returned error  
wc.status = 12, wc.vendor_err =
  129, post->buf = 0x2aaaab6cf000, wc.byte_len = 0, post->reused = 1
2009-05-13 01:22:04 E [saved-frames.c:165:saved_frames_unwind] Node2:  
forced unwinding frame type(2) op(SETVOLUME)
2009-05-13 01:22:04 E [client-protocol.c:6267:client_setvolume_cbk]  
Node2: setvolume failed (Transport endpoint is not connected)
2009-05-13 01:22:04 N [client-protocol.c:7014:notify] Node2:  
disconnected