[Gluster-users] Fedora 11 - 2.6.31 Kernel - Fuse 2.8.0 - Infiniband
Nathan Stratton
nathan at robotics.net
Tue Sep 22 17:35:08 UTC 2009
Hate to post again, but anyone have any ideas on this?
-Nathan
On Fri, 18 Sep 2009, Nathan Stratton wrote:
>
> Has anyone been able to get Infiniband working with 2.6.31 kernel and fuse
> 2.8.0? My config works fine on my Centos 2.6.18 box, so I know that is ok.
>
> Infiniband looks good:
>
> [root at xen1 src]# lsmod |grep ib
> ib_ucm 13752 0
> ib_uverbs 32256 2 rdma_ucm,ib_ucm
> ib_ipoib 68880 0
> ib_mthca 123700 0
>
> [root at xen1 src]# ibv_devices
> device node GUID
> ------ ----------------
> mthca0 0005ad00000327e8
>
> Gluster looks like it starts OK, but I can't touch the mount and after a
> while it times out. Debug logs:
>
>
> [2009-09-18 19:36:17] D [glusterfsd.c:354:_get_specfp] glusterfs: loading
> volume file /usr/local/etc/glusterfs/glusterfs.vol
> ================================================================================
> Version : glusterfs 2.0.6 built on Sep 18 2009 09:54:43
> TLA Revision : v2.0.6
> Starting Time: 2009-09-18 19:36:17
> Command line : glusterfs -L DEBUG -l /var/log/glusterfs.log
> --disable-direct-io-mode /share
> PID : 8303
> System name : Linux
> Nodename : xen1.hou.blinkmind.com
> Kernel Release : 2.6.31
> Hardware Identifier: x86_64
>
> Given volfile:
> +------------------------------------------------------------------------------+
> 1: volume brick0
> 2: type protocol/client
> 3: option transport-type ib-verbs/client
> 4: option remote-host 172.16.0.200
> 5: option remote-port 6997
> 6: option transport.address-family inet/inet6
> 7: option remote-subvolume brick
> 8: end-volume
> 9:
> 10: volume mirror0
> 11: type protocol/client
> 12: option transport-type ib-verbs/client
> 13: option remote-host 172.16.0.201
> 14: option remote-port 6997
> 15: option transport.address-family inet/inet6
> 16: option remote-subvolume brick
> 17: end-volume
> 18:
> 19: volume brick1
> 20: type protocol/client
> 21: option transport-type ib-verbs/client
> 22: option remote-host 172.16.0.202
> 23: option remote-port 6997
> 24: option transport.address-family inet/inet6
> 25: option remote-subvolume brick
> 26: end-volume
> 27:
> 28: volume mirror1
> 29: type protocol/client
> 30: option transport-type ib-verbs/client
> 31: option remote-host 172.16.0.203
> 32: option remote-port 6997
> 33: option transport.address-family inet/inet6
> 34: option remote-subvolume brick
> 35: end-volume
> 36:
> 37: volume brick2
> 38: type protocol/client
> 39: option transport-type ib-verbs/client
> 40: option remote-host 172.16.0.204
> 41: option remote-port 6997
> 42: option transport.address-family inet/inet6
> 43: option remote-subvolume brick
> 44: end-volume
> 45:
> 46: volume mirror2
> 47: type protocol/client
> 48: option transport-type ib-verbs/client
> 49: option remote-host 172.16.0.205
> 50: option remote-port 6997
> 51: option transport.address-family inet/inet6
> 52: option remote-subvolume brick
> 53: end-volume
> 54:
> 55: volume block0
> 56: type cluster/replicate
> 57: subvolumes brick0 mirror0
> 58: end-volume
> 59:
> 60: volume block1
> 61: type cluster/replicate
> 62: subvolumes brick1 mirror1
> 63: end-volume
> 64:
> 65: volume block2
> 66: type cluster/replicate
> 67: subvolumes brick2 mirror2
> 68: end-volume
> 69:
> 70: volume unify
> 71: type cluster/distribute
> 72: subvolumes block0 block1 block2
> 73: end-volume
> 74:
>
> +------------------------------------------------------------------------------+
> [2009-09-18 19:36:17] D [glusterfsd.c:1205:main] glusterfs: running in pid
> 8303
> [2009-09-18 19:36:17] D [client-protocol.c:5952:init] brick0: defaulting
> frame-timeout to 30mins
> [2009-09-18 19:36:17] D [client-protocol.c:5963:init] brick0: defaulting
> ping-timeout to 10
> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] brick0:
> no range check required for 'option remote-port 6997'
> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] brick0:
> no range check required for 'option remote-port 6997'
> [2009-09-18 19:36:17] D [client-protocol.c:5952:init] mirror0: defaulting
> frame-timeout to 30mins
> [2009-09-18 19:36:17] D [client-protocol.c:5963:init] mirror0: defaulting
> ping-timeout to 10
> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] mirror0:
> no range check required for 'option remote-port 6997'
> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] mirror0:
> no range check required for 'option remote-port 6997'
> [2009-09-18 19:36:17] D [client-protocol.c:5952:init] brick1: defaulting
> frame-timeout to 30mins
> [2009-09-18 19:36:17] D [client-protocol.c:5963:init] brick1: defaulting
> ping-timeout to 10
> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] brick1:
> no range check required for 'option remote-port 6997'
> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] brick1:
> no range check required for 'option remote-port 6997'
> [2009-09-18 19:36:17] D [client-protocol.c:5952:init] mirror1: defaulting
> frame-timeout to 30mins
> [2009-09-18 19:36:17] D [client-protocol.c:5963:init] mirror1: defaulting
> ping-timeout to 10
> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] mirror1:
> no range check required for 'option remote-port 6997'
> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] mirror1:
> no range check required for 'option remote-port 6997'
> [2009-09-18 19:36:17] D [client-protocol.c:5952:init] brick2: defaulting
> frame-timeout to 30mins
> [2009-09-18 19:36:17] D [client-protocol.c:5963:init] brick2: defaulting
> ping-timeout to 10
> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] brick2:
> no range check required for 'option remote-port 6997'
> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] brick2:
> no range check required for 'option remote-port 6997'
> [2009-09-18 19:36:17] D [client-protocol.c:5952:init] mirror2: defaulting
> frame-timeout to 30mins
> [2009-09-18 19:36:17] D [client-protocol.c:5963:init] mirror2: defaulting
> ping-timeout to 10
> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] mirror2:
> no range check required for 'option remote-port 6997'
> [2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: attempt
> to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
> [2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] mirror2:
> no range check required for 'option remote-port 6997'
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick0: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick0: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror0: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror0: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick1: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick1: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror1: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror1: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick2: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick2: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror2: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror2: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick0: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick0: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror0: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror0: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick1: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick1: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror1: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror1: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick2: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick2: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror2: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror2: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> [2009-09-18 19:36:17] N [glusterfsd.c:1224:main] glusterfs: Successfully
> started
> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick0: got
> GF_EVENT_CHILD_UP
> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick0: got
> GF_EVENT_CHILD_UP
> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror0: got
> GF_EVENT_CHILD_UP
> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror0: got
> GF_EVENT_CHILD_UP
> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick1: got
> GF_EVENT_CHILD_UP
> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick1: got
> GF_EVENT_CHILD_UP
> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror1: got
> GF_EVENT_CHILD_UP
> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror1: got
> GF_EVENT_CHILD_UP
> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick2: got
> GF_EVENT_CHILD_UP
> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick2: got
> GF_EVENT_CHILD_UP
> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror2: got
> GF_EVENT_CHILD_UP
> [2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror2: got
> GF_EVENT_CHILD_UP
> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick0: bailing out
> frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 1800
> [2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk] brick0:
> setvolume failed (Transport endpoint is not connected)
> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick0: bailing out
> frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 1800
> [2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk] brick0:
> setvolume failed (Transport endpoint is not connected)
> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror0: bailing
> out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 1800
> [2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk]
> mirror0: setvolume failed (Transport endpoint is not connected)
> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror0: bailing
> out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 1800
> [2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk]
> mirror0: setvolume failed (Transport endpoint is not connected)
> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick1: bailing out
> frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 1800
> [2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk] brick1:
> setvolume failed (Transport endpoint is not connected)
> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick1: bailing out
> frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 1800
> [2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk] brick1:
> setvolume failed (Transport endpoint is not connected)
> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror1: bailing
> out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 1800
> [2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk]
> mirror1: setvolume failed (Transport endpoint is not connected)
> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror1: bailing
> out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 1800
> [2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk]
> mirror1: setvolume failed (Transport endpoint is not connected)
> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick2: bailing out
> frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 1800
> [2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk] brick2:
> setvolume failed (Transport endpoint is not connected)
> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick2: bailing out
> frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 1800
> [2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk] brick2:
> setvolume failed (Transport endpoint is not connected)
> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror2: bailing
> out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 1800
> [2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk]
> mirror2: setvolume failed (Transport endpoint is not connected)
> [2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror2: bailing
> out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 1800
> [2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk]
> mirror2: setvolume failed (Transport endpoint is not connected)
> [2009-09-18 20:06:18] D [dht-common.c:820:dht_lookup] unify: no subvolume in
> layout for path=/, checking on all the subvols to see if it is a directory
> [2009-09-18 20:06:18] D [dht-common.c:113:dht_lookup_dir_cbk] unify: lookup
> of / on block0 returned error (Transport endpoint is not connected)
> [2009-09-18 20:06:18] D [dht-common.c:113:dht_lookup_dir_cbk] unify: lookup
> of / on block1 returned error (Transport endpoint is not connected)
> [2009-09-18 20:06:18] D [dht-common.c:113:dht_lookup_dir_cbk] unify: lookup
> of / on block2 returned error (Transport endpoint is not connected)
> [2009-09-18 20:06:18] D [fuse-bridge.c:2385:fuse_root_lookup_cbk] fuse: first
> lookup on root failed.
> [2009-09-18 20:06:18] W [fuse-bridge.c:1841:fuse_statfs_cbk] glusterfs-fuse:
> 2: ERR => -1 (Transport endpoint is not connected)
>
>
>
>> <>
> Nathan Stratton CTO, BlinkMind, Inc.
> nathan at robotics.net nathan at blinkmind.com
> http://www.robotics.net http://www.blinkmind.com
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
More information about the Gluster-users
mailing list