[Gluster-users] Problems getting Glusterfs running on IB
Mickey Mazarick
mic at digitaltadpole.com
Fri Jul 25 16:05:27 UTC 2008
All the stuff the you need for infiniband is in the latest OFED package:
http://www.openfabrics.org/downloads/OFED/ofed-1.3.1/OFED-1.3.1.tgz
We are using Mellanox cards and also had to flash them up to the newest
version for ibverbs to work.
-Mickey Mazarick
Peter Pelekis wrote:
> The Infinipath drivers don't have a libibverbs they call it
> libinfinipath. I'll test out the ib-sdp interface to see if it works.
>
> -Peter
>
> Raghavendra G wrote:
>
>> Hi Peter,
>> Glusterfs ib-verbs transport needs libibverbs to be installed. Is it
>> installed on your system. Also, in case of ib-sdp are you able to ping
>> the other node using the interface which has ib-sdp support?
>>
>> regards,
>> On Fri, Jul 25, 2008 at 2:26 AM, Peter Pelekis <pete.pelekis at wsm.com
>> <mailto:pete.pelekis at wsm.com>> wrote:
>>
>> I'm having some problems getting Glusterfs to run on my system.
>> This is
>> the first time using it so you I'm not to knowledgeable yet. The
>> system
>> I'm using has infiniband installed so I would like to use the IB
>> for the
>> transport for better performance. Heres what I have done so far. I
>> first set up glusterfs to run over tcp on the Gig-E network and that
>> worked fine so I decided to move to the IB network and thats when my
>> problems started. Heres some information about the system.
>>
>> Glusterfs: glusterfs-1.3.9
>> Fuse: fuse-2.7.3glfs10
>> Cards: Qlogic QLE7240
>> Drivers: InfiniPath-2.2
>> OS: Centos5.2
>> Kernel: 2.6.18-53
>> 4 Storage nodes
>> 46 Compute nodes
>>
>> When I first ran configure on glusterfs I got the following message
>> after it ran configure.
>>
>> GlusterFS configure summary
>> ===========================
>> Fuse client : yes
>> Infiniband verbs : no
>> epoll IO multiplex : yes
>>
>> If I look at lsmod I do have a ib_uverbs module installed and working
>> but the configure script doesn't pick it up. I decided move on and use
>> ib-sdp instead. I modified all my client and server volume files to
>> reflect ib-sdp for the transport then I started the server on my 4
>> storage nodes it started fine. I then went to one of my compute nodes
>> to mount the volume.
>> glusterfsd --transport=ib-sdp -s storage-ib1 /glusterfs
>>
>> It returned with the following error.
>> glusterfs: could not open specfile
>>
>> Then I tried to specify the volume file to see if that would work.
>> glusterfsd --spec-file=/usr/local/etc/glusterfs/glusterfs-client.vol
>> /glusterfs
>>
>> It returned fine so I thought everything was working until I ran df I
>> got the following error.
>> df: `/glusterfs': Transport endpoint is not connected
>>
>> I had it log the error that its getting when I try and mount the
>> filesystem.
>> 2008-07-24 17:28:01 E [ib-sdp-client.c:141:ib_sdp_connect] trans:
>> error:
>> not in progress - trace: Network is unreachable
>> 2008-07-24 17:28:01 W [client-protocol.c:332:client_protocol_xfer]
>> trans: not connected at the moment to submit frame type(2) op(4)
>> 2008-07-24 17:28:01 E [client-protocol.c:4538:client_getspec_cbk]
>> trans:
>> no proper reply from server, returning ENOTCONN
>>
>> This where I'm now I looked around the devel list and didn't find
>> anything on using the qlogic drivers just the ofed IB drivers. Heres
>> what my configuration files. I would like to us ib-verbs if thats
>> possible the performance should be much better then ib-sdp.
>>
>> volume posix-stripe
>> type storage/posix
>> option directory /export/glusterfs
>> end-volume
>>
>> volume server
>> type protocol/server
>> option transport-type ib-sdp/server
>> option auth.ip.posix-stripe.allow 192.168.2.
>> <http://192.168.2.>*
>> subvolumes posix-stripe
>> end-volume
>>
>> volume client-stripe-1
>> type protocol/client
>> option transport-type ib-sdp/client
>> option remote-host storage-ib1.cluster
>> option remote-subvolume posix-stripe
>> end-volume
>>
>> volume client-stripe-2
>> type protocol/client
>> option transport-type ib-sdp/client
>> option remote-host storage-ib2.cluster
>> option remote-subvolume posix-stripe
>> end-volume
>>
>> volume client-stripe-3
>> type protocol/client
>> option transport-type ib-sdp/client
>> option remote-host storage-ib3.cluster
>> option remote-subvolume posix-stripe
>> end-volume
>>
>> volume client-stripe-4
>> type protocol/client
>> option transport-type ib-sdp/client
>> option remote-host storage-ib4.cluster
>> option remote-subvolume posix-stripe
>> end-volume
>>
>> volume stripe
>> type cluster/stripe
>> option block-size *:2MB # All files ending with .img are striped
>> with 2MB stripe block size.
>> subvolumes client-stripe-1 client-stripe-2 client-stripe-3
>> client-stripe-4
>> end-volume
>>
>> -Peter
>>
>> --
>> Peter Pelekis 5444 Napa Street
>> Western Scientific San Diego CA, 92110
>> Senior Systems Engineer www.wsm.com <http://www.wsm.com>
>> pete.pelekis at wsm.com <mailto:pete.pelekis at wsm.com> Fax
>> 619-220-6590
>> Phone 619-220-6580 x212 Toll Free 800-443-6699
>> GSA# GS-35F-5009H
>>
>> Visit us at:
>>
>> "I'm a great believer in luck, and I find the harder I work the
>> more I have of it."
>> -Thomas Jefferson
>>
>> "If everything seems under control, you're not going fast enough."
>> -Mario Andretti
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>>
>>
>>
>>
>> --
>> Raghavendra G
>>
>> A centipede was happy quite, until a toad in fun,
>> Said, "Prey, which leg comes after which?",
>> This raised his doubts to such a pitch,
>> He fell flat into the ditch,
>> Not knowing how to run.
>> -Anonymous
>>
>
>
>
--
More information about the Gluster-users
mailing list