[Gluster-users] Gluster 2.0.1 and ib-verbs

[Jason Williams]

So this may seem like a very newb question, but I don't quite understand 
why this isn't working.  The configuration I'm using is very simple, and 
I've included both vol files below.  The problem seems to be that the 
client is sending something over the IPoIB interface AFTER it's already 
started working over the ib-verbs interface... atleast, that's what I 
gather from the following errors when run in debug mode...

[2009-06-08 16:58:20] D [ib-verbs.c:507:ib_verbs_receive] 
transport/ib-verbs: server: corrupt header received
[2009-06-08 16:58:20] D [ib-verbs.c:1156:ib_verbs_recv_completion_proc] 
transport/ib-verbs: pollin notification to server failed, disconnecting 
transport
[2009-06-08 16:58:20] E [ib-verbs.c:1996:ib_verbs_event_handler] 
transport/ib-verbs: server: pollin received on tcp socket (peer: 
172.21.1.154:1023) after handshake is complete
[2009-06-08 16:58:20] D [ib-verbs.c:1865:ib_verbs_handshake_pollerr] 
transport/ib-verbs: server: peer disconnected, cleaning up
[2009-06-08 16:58:20] N [server-protocol.c:7796:notify] server: 
172.21.1.154:1023 disconnected

If I'm missing something in my configuration, please feel free to point 
it out to me. Any help is greatly appreciated as I plan on going much 
larger than just NFS type filesystem setups, if the performance can be 
tweaked good enough.

-- 
Jason Williams
Systems Administrator
Johns Hopkins University
Physics and Astronomy Department


---- Configuration Follows ----

# file: /etc/glusterfs/glusterfs-server.vol
volume posix
  type storage/posix
  option directory /export/scratch
end-volume

volume locks
  type features/locks
  option mandatory-locks on
  subvolumes posix
end-volume

volume brick
  type performance/io-threads
  option thread-count 8
  subvolumes locks
end-volume

volume server
  type protocol/server
  option client-volume-filename /etc/glusterfs/glusterfs-client.vol
  option transport-type ib-verbs/server
#  option transport-type tcp/server
  option transport.ib-verbs.listen-path 172.21.0.2
  option auth.addr.brick.allow * # Edit and add list of allowed clients 
comma separated IP addrs(names) here
  subvolumes brick
end-volume


# file: /etc/glusterfs/glusterfs-client.vol
volume remote
  type protocol/client
  option transport-type ib-verbs/client
#  option transport-type tcp/client
  option remote-host 172.21.0.2 # can be IP or hostname
  option remote-subvolume brick
end-volume

volume writebehind
  type performance/write-behind
  option block-size 1MB
  option window-size 4MB
  subvolumes remote
end-volume

volume cache
  type performance/io-cache
  option cache-size 512MB
  subvolumes writebehind
end-volume