[Gluster-users] Problems getting Glusterfs running on IB
Peter Pelekis
pete.pelekis at wsm.com
Fri Jul 25 15:50:05 UTC 2008
The Infinipath drivers don't have a libibverbs they call it
libinfinipath. I'll test out the ib-sdp interface to see if it works.
-Peter
Raghavendra G wrote:
> Hi Peter,
> Glusterfs ib-verbs transport needs libibverbs to be installed. Is it
> installed on your system. Also, in case of ib-sdp are you able to ping
> the other node using the interface which has ib-sdp support?
>
> regards,
> On Fri, Jul 25, 2008 at 2:26 AM, Peter Pelekis <pete.pelekis at wsm.com
> <mailto:pete.pelekis at wsm.com>> wrote:
>
> I'm having some problems getting Glusterfs to run on my system.
> This is
> the first time using it so you I'm not to knowledgeable yet. The
> system
> I'm using has infiniband installed so I would like to use the IB
> for the
> transport for better performance. Heres what I have done so far. I
> first set up glusterfs to run over tcp on the Gig-E network and that
> worked fine so I decided to move to the IB network and thats when my
> problems started. Heres some information about the system.
>
> Glusterfs: glusterfs-1.3.9
> Fuse: fuse-2.7.3glfs10
> Cards: Qlogic QLE7240
> Drivers: InfiniPath-2.2
> OS: Centos5.2
> Kernel: 2.6.18-53
> 4 Storage nodes
> 46 Compute nodes
>
> When I first ran configure on glusterfs I got the following message
> after it ran configure.
>
> GlusterFS configure summary
> ===========================
> Fuse client : yes
> Infiniband verbs : no
> epoll IO multiplex : yes
>
> If I look at lsmod I do have a ib_uverbs module installed and working
> but the configure script doesn't pick it up. I decided move on and use
> ib-sdp instead. I modified all my client and server volume files to
> reflect ib-sdp for the transport then I started the server on my 4
> storage nodes it started fine. I then went to one of my compute nodes
> to mount the volume.
> glusterfsd --transport=ib-sdp -s storage-ib1 /glusterfs
>
> It returned with the following error.
> glusterfs: could not open specfile
>
> Then I tried to specify the volume file to see if that would work.
> glusterfsd --spec-file=/usr/local/etc/glusterfs/glusterfs-client.vol
> /glusterfs
>
> It returned fine so I thought everything was working until I ran df I
> got the following error.
> df: `/glusterfs': Transport endpoint is not connected
>
> I had it log the error that its getting when I try and mount the
> filesystem.
> 2008-07-24 17:28:01 E [ib-sdp-client.c:141:ib_sdp_connect] trans:
> error:
> not in progress - trace: Network is unreachable
> 2008-07-24 17:28:01 W [client-protocol.c:332:client_protocol_xfer]
> trans: not connected at the moment to submit frame type(2) op(4)
> 2008-07-24 17:28:01 E [client-protocol.c:4538:client_getspec_cbk]
> trans:
> no proper reply from server, returning ENOTCONN
>
> This where I'm now I looked around the devel list and didn't find
> anything on using the qlogic drivers just the ofed IB drivers. Heres
> what my configuration files. I would like to us ib-verbs if thats
> possible the performance should be much better then ib-sdp.
>
> volume posix-stripe
> type storage/posix
> option directory /export/glusterfs
> end-volume
>
> volume server
> type protocol/server
> option transport-type ib-sdp/server
> option auth.ip.posix-stripe.allow 192.168.2.
> <http://192.168.2.>*
> subvolumes posix-stripe
> end-volume
>
> volume client-stripe-1
> type protocol/client
> option transport-type ib-sdp/client
> option remote-host storage-ib1.cluster
> option remote-subvolume posix-stripe
> end-volume
>
> volume client-stripe-2
> type protocol/client
> option transport-type ib-sdp/client
> option remote-host storage-ib2.cluster
> option remote-subvolume posix-stripe
> end-volume
>
> volume client-stripe-3
> type protocol/client
> option transport-type ib-sdp/client
> option remote-host storage-ib3.cluster
> option remote-subvolume posix-stripe
> end-volume
>
> volume client-stripe-4
> type protocol/client
> option transport-type ib-sdp/client
> option remote-host storage-ib4.cluster
> option remote-subvolume posix-stripe
> end-volume
>
> volume stripe
> type cluster/stripe
> option block-size *:2MB # All files ending with .img are striped
> with 2MB stripe block size.
> subvolumes client-stripe-1 client-stripe-2 client-stripe-3
> client-stripe-4
> end-volume
>
> -Peter
>
> --
> Peter Pelekis 5444 Napa Street
> Western Scientific San Diego CA, 92110
> Senior Systems Engineer www.wsm.com <http://www.wsm.com>
> pete.pelekis at wsm.com <mailto:pete.pelekis at wsm.com> Fax
> 619-220-6590
> Phone 619-220-6580 x212 Toll Free 800-443-6699
> GSA# GS-35F-5009H
>
> Visit us at:
>
> "I'm a great believer in luck, and I find the harder I work the
> more I have of it."
> -Thomas Jefferson
>
> "If everything seems under control, you're not going fast enough."
> -Mario Andretti
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>
>
>
>
> --
> Raghavendra G
>
> A centipede was happy quite, until a toad in fun,
> Said, "Prey, which leg comes after which?",
> This raised his doubts to such a pitch,
> He fell flat into the ditch,
> Not knowing how to run.
> -Anonymous
--
Peter Pelekis 5444 Napa Street
Western Scientific San Diego CA, 92110
Senior Systems Engineer www.wsm.com
pete.pelekis at wsm.com Fax 619-220-6590
Phone 619-220-6580 x212 Toll Free 800-443-6699
GSA# GS-35F-5009H
Visit us at:
"I'm a great believer in luck, and I find the harder I work the more I have of it."
-Thomas Jefferson
"If everything seems under control, you're not going fast enough."
-Mario Andretti
More information about the Gluster-users
mailing list