[Gluster-users] Problems getting Glusterfs running on IB

[Peter Pelekis]

I'm having some problems getting Glusterfs to run on my system. This is 
the first time using it so you I'm not to knowledgeable yet. The system 
I'm using has infiniband installed so I would like to use the IB for the 
transport  for better performance. Heres what I have done so far.  I 
first set up glusterfs to run over tcp on the Gig-E network and that 
worked fine so I decided to move to the IB network and thats when my 
problems started. Heres some information about the system.

Glusterfs: glusterfs-1.3.9
Fuse: fuse-2.7.3glfs10
Cards: Qlogic QLE7240
Drivers: InfiniPath-2.2
OS: Centos5.2
Kernel: 2.6.18-53
4 Storage nodes
46 Compute nodes

When I first ran configure on glusterfs I got the following message 
after it ran configure.

GlusterFS configure summary
===========================
Fuse client        : yes
Infiniband verbs   : no
epoll IO multiplex : yes

If I look at lsmod I do have a ib_uverbs module installed and working 
but the configure script doesn't pick it up. I decided move on and use 
ib-sdp instead. I modified all my client and server volume files to 
reflect ib-sdp for the transport then I started the server on my 4 
storage nodes it started fine.  I then went to one of my compute nodes 
to mount the volume.
glusterfsd --transport=ib-sdp -s storage-ib1 /glusterfs

It returned with the following error.
glusterfs: could not open specfile

Then I tried to specify the volume file to see if that would work.
glusterfsd --spec-file=/usr/local/etc/glusterfs/glusterfs-client.vol 
/glusterfs

It returned fine so I thought everything was working until I ran df I 
got the following error.
df: `/glusterfs': Transport endpoint is not connected

I had it log the error that its getting when I try and mount the filesystem.
2008-07-24 17:28:01 E [ib-sdp-client.c:141:ib_sdp_connect] trans: error: 
not in progress - trace: Network is unreachable
2008-07-24 17:28:01 W [client-protocol.c:332:client_protocol_xfer] 
trans: not connected at the moment to submit frame type(2) op(4)
2008-07-24 17:28:01 E [client-protocol.c:4538:client_getspec_cbk] trans: 
no proper reply from server, returning ENOTCONN

This where I'm now I looked around the devel list and didn't find 
anything on using the qlogic drivers just the ofed IB drivers. Heres 
what my configuration files. I would like to us ib-verbs if thats 
possible the performance should be much better then ib-sdp.

 volume posix-stripe
           type storage/posix
           option directory /export/glusterfs
   end-volume

   volume server
           type protocol/server
           option transport-type ib-sdp/server
           option auth.ip.posix-stripe.allow 192.168.2.*
           subvolumes posix-stripe
   end-volume

   volume client-stripe-1
     type protocol/client
     option transport-type ib-sdp/client
     option remote-host storage-ib1.cluster
     option remote-subvolume posix-stripe
   end-volume

   volume client-stripe-2
     type protocol/client
     option transport-type ib-sdp/client
     option remote-host storage-ib2.cluster
     option remote-subvolume posix-stripe
   end-volume

   volume client-stripe-3
     type protocol/client
     option transport-type ib-sdp/client
     option remote-host storage-ib3.cluster
     option remote-subvolume posix-stripe
   end-volume

   volume client-stripe-4
     type protocol/client
     option transport-type ib-sdp/client
     option remote-host storage-ib4.cluster
     option remote-subvolume posix-stripe
   end-volume

   volume stripe
     type cluster/stripe
     option block-size *:2MB # All files ending with .img are striped 
with 2MB stripe block size.
     subvolumes client-stripe-1 client-stripe-2 client-stripe-3 
client-stripe-4
   end-volume

-Peter

-- 
Peter Pelekis  		5444 Napa Street
Western Scientific 	San Diego CA, 92110
Senior Systems Engineer		www.wsm.com
pete.pelekis at wsm.com		Fax 619-220-6590
Phone 619-220-6580 x212 Toll Free 800-443-6699
GSA# GS-35F-5009H

Visit us at:

"I'm a great believer in luck, and I find the harder I work the more I have of it."
-Thomas Jefferson

"If everything seems under control, you're not going fast enough."
-Mario Andretti