[Gluster-users] Gluster 2.0.1 and ib-verbs

Jason Williams jasonw at jhu.edu
Mon Jun 8 21:08:37 UTC 2009

So this may seem like a very newb question, but I don't quite understand 
why this isn't working.  The configuration I'm using is very simple, and 
I've included both vol files below.  The problem seems to be that the 
client is sending something over the IPoIB interface AFTER it's already 
started working over the ib-verbs interface... atleast, that's what I 
gather from the following errors when run in debug mode...

[2009-06-08 16:58:20] D [ib-verbs.c:507:ib_verbs_receive] 
transport/ib-verbs: server: corrupt header received
[2009-06-08 16:58:20] D [ib-verbs.c:1156:ib_verbs_recv_completion_proc] 
transport/ib-verbs: pollin notification to server failed, disconnecting 
[2009-06-08 16:58:20] E [ib-verbs.c:1996:ib_verbs_event_handler] 
transport/ib-verbs: server: pollin received on tcp socket (peer: after handshake is complete
[2009-06-08 16:58:20] D [ib-verbs.c:1865:ib_verbs_handshake_pollerr] 
transport/ib-verbs: server: peer disconnected, cleaning up
[2009-06-08 16:58:20] N [server-protocol.c:7796:notify] server: disconnected

If I'm missing something in my configuration, please feel free to point 
it out to me. Any help is greatly appreciated as I plan on going much 
larger than just NFS type filesystem setups, if the performance can be 
tweaked good enough.

Jason Williams
Systems Administrator
Johns Hopkins University
Physics and Astronomy Department

---- Configuration Follows ----

# file: /etc/glusterfs/glusterfs-server.vol
volume posix
  type storage/posix
  option directory /export/scratch

volume locks
  type features/locks
  option mandatory-locks on
  subvolumes posix

volume brick
  type performance/io-threads
  option thread-count 8
  subvolumes locks

volume server
  type protocol/server
  option client-volume-filename /etc/glusterfs/glusterfs-client.vol
  option transport-type ib-verbs/server
#  option transport-type tcp/server
  option transport.ib-verbs.listen-path
  option auth.addr.brick.allow * # Edit and add list of allowed clients 
comma separated IP addrs(names) here
  subvolumes brick

# file: /etc/glusterfs/glusterfs-client.vol
volume remote
  type protocol/client
  option transport-type ib-verbs/client
#  option transport-type tcp/client
  option remote-host # can be IP or hostname
  option remote-subvolume brick

volume writebehind
  type performance/write-behind
  option block-size 1MB
  option window-size 4MB
  subvolumes remote

volume cache
  type performance/io-cache
  option cache-size 512MB
  subvolumes writebehind

More information about the Gluster-users mailing list