[Gluster-users] Fedora 11 - 2.6.31 Kernel - Fuse 2.8.0 - Infiniband

Nathan Stratton nathan at robotics.net
Sat Sep 19 01:22:31 UTC 2009


Has anyone been able to get Infiniband working with 2.6.31 kernel and fuse 
2.8.0? My config works fine on my Centos 2.6.18 box, so I know that is ok.

Infiniband looks good:

[root at xen1 src]# lsmod |grep ib
ib_ucm                 13752  0
ib_uverbs              32256  2 rdma_ucm,ib_ucm
ib_ipoib               68880  0
ib_mthca              123700  0

[root at xen1 src]# ibv_devices
     device          	   node GUID
     ------          	----------------
     mthca0          	0005ad00000327e8

Gluster looks like it starts OK, but I can't touch the mount and after a 
while it times out. Debug logs:


[2009-09-18 19:36:17] D [glusterfsd.c:354:_get_specfp] glusterfs: loading 
volume file /usr/local/etc/glusterfs/glusterfs.vol
================================================================================
Version      : glusterfs 2.0.6 built on Sep 18 2009 09:54:43
TLA Revision : v2.0.6
Starting Time: 2009-09-18 19:36:17
Command line : glusterfs -L DEBUG -l /var/log/glusterfs.log 
--disable-direct-io-mode /share
PID          : 8303
System name  : Linux
Nodename     : xen1.hou.blinkmind.com
Kernel Release : 2.6.31
Hardware Identifier: x86_64

Given volfile:
+------------------------------------------------------------------------------+
   1: volume brick0
   2:  type protocol/client
   3:  option transport-type ib-verbs/client
   4:  option remote-host 172.16.0.200
   5:  option remote-port 6997
   6:  option transport.address-family inet/inet6
   7:  option remote-subvolume brick
   8: end-volume
   9:
  10: volume mirror0
  11:  type protocol/client
  12:  option transport-type ib-verbs/client
  13:  option remote-host 172.16.0.201
  14:  option remote-port 6997
  15:  option transport.address-family inet/inet6
  16:  option remote-subvolume brick
  17: end-volume
  18:
  19: volume brick1
  20:  type protocol/client
  21:  option transport-type ib-verbs/client
  22:  option remote-host 172.16.0.202
  23:  option remote-port 6997
  24:  option transport.address-family inet/inet6
  25:  option remote-subvolume brick
  26: end-volume
  27:
  28: volume mirror1
  29:  type protocol/client
  30:  option transport-type ib-verbs/client
  31:  option remote-host 172.16.0.203
  32:  option remote-port 6997
  33:  option transport.address-family inet/inet6
  34:  option remote-subvolume brick
  35: end-volume
  36:
  37: volume brick2
  38:  type protocol/client
  39:  option transport-type ib-verbs/client
  40:  option remote-host 172.16.0.204
  41:  option remote-port 6997
  42:  option transport.address-family inet/inet6
  43:  option remote-subvolume brick
  44: end-volume
  45:
  46: volume mirror2
  47:  type protocol/client
  48:  option transport-type ib-verbs/client
  49:  option remote-host 172.16.0.205
  50:  option remote-port 6997
  51:  option transport.address-family inet/inet6
  52:  option remote-subvolume brick
  53: end-volume
  54:
  55: volume block0
  56:  type cluster/replicate
  57:  subvolumes brick0 mirror0
  58: end-volume
  59:
  60: volume block1
  61:  type cluster/replicate
  62:  subvolumes brick1 mirror1
  63: end-volume
  64:
  65: volume block2
  66:  type cluster/replicate
  67:  subvolumes brick2 mirror2
  68: end-volume
  69:
  70: volume unify
  71:  type cluster/distribute
  72:  subvolumes block0 block1 block2
  73: end-volume
  74:

+------------------------------------------------------------------------------+
[2009-09-18 19:36:17] D [glusterfsd.c:1205:main] glusterfs: running in pid 
8303
[2009-09-18 19:36:17] D [client-protocol.c:5952:init] brick0: defaulting 
frame-timeout to 30mins
[2009-09-18 19:36:17] D [client-protocol.c:5963:init] brick0: defaulting 
ping-timeout to 10
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: 
attempt to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] 
brick0: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: 
attempt to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] 
brick0: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [client-protocol.c:5952:init] mirror0: defaulting 
frame-timeout to 30mins
[2009-09-18 19:36:17] D [client-protocol.c:5963:init] mirror0: defaulting 
ping-timeout to 10
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: 
attempt to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] 
mirror0: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: 
attempt to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] 
mirror0: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [client-protocol.c:5952:init] brick1: defaulting 
frame-timeout to 30mins
[2009-09-18 19:36:17] D [client-protocol.c:5963:init] brick1: defaulting 
ping-timeout to 10
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: 
attempt to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] 
brick1: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: 
attempt to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] 
brick1: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [client-protocol.c:5952:init] mirror1: defaulting 
frame-timeout to 30mins
[2009-09-18 19:36:17] D [client-protocol.c:5963:init] mirror1: defaulting 
ping-timeout to 10
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: 
attempt to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] 
mirror1: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: 
attempt to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] 
mirror1: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [client-protocol.c:5952:init] brick2: defaulting 
frame-timeout to 30mins
[2009-09-18 19:36:17] D [client-protocol.c:5963:init] brick2: defaulting 
ping-timeout to 10
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: 
attempt to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] 
brick2: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: 
attempt to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] 
brick2: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [client-protocol.c:5952:init] mirror2: defaulting 
frame-timeout to 30mins
[2009-09-18 19:36:17] D [client-protocol.c:5963:init] mirror2: defaulting 
ping-timeout to 10
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: 
attempt to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] 
mirror2: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [transport.c:141:transport_load] transport: 
attempt to load file /usr/local/lib/glusterfs/2.0.6/transport/ib-verbs.so
[2009-09-18 19:36:17] D [xlator.c:276:_volume_option_value_validate] 
mirror2: no range check required for 'option remote-port 6997'
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick0: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick0: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror0: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror0: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick1: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick1: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror1: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror1: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick2: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick2: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror2: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror2: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick0: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick0: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror0: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror0: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick1: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick1: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror1: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror1: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick2: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] brick2: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror2: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] D [client-protocol.c:6280:notify] mirror2: got 
GF_EVENT_PARENT_UP, attempting connect on transport
[2009-09-18 19:36:17] N [glusterfsd.c:1224:main] glusterfs: Successfully 
started
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick0: got 
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick0: got 
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror0: got 
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror0: got 
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick1: got 
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick1: got 
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror1: got 
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror1: got 
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick2: got 
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] brick2: got 
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror2: got 
GF_EVENT_CHILD_UP
[2009-09-18 19:36:17] D [client-protocol.c:6294:notify] mirror2: got 
GF_EVENT_CHILD_UP
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick0: bailing 
out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 
1800
[2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk] 
brick0: setvolume failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick0: bailing 
out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 
1800
[2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk] 
brick0: setvolume failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror0: bailing 
out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 
1800
[2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk] 
mirror0: setvolume failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror0: bailing 
out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 
1800
[2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk] 
mirror0: setvolume failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick1: bailing 
out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 
1800
[2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk] 
brick1: setvolume failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick1: bailing 
out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 
1800
[2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk] 
brick1: setvolume failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror1: bailing 
out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 
1800
[2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk] 
mirror1: setvolume failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror1: bailing 
out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 
1800
[2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk] 
mirror1: setvolume failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick2: bailing 
out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 
1800
[2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk] 
brick2: setvolume failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] brick2: bailing 
out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 
1800
[2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk] 
brick2: setvolume failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror2: bailing 
out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 
1800
[2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk] 
mirror2: setvolume failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] E [client-protocol.c:289:call_bail] mirror2: bailing 
out frame SETVOLUME(0) frame sent = 2009-09-18 19:36:17. frame-timeout = 
1800
[2009-09-18 20:06:18] D [client-protocol.c:5491:client_setvolume_cbk] 
mirror2: setvolume failed (Transport endpoint is not connected)
[2009-09-18 20:06:18] D [dht-common.c:820:dht_lookup] unify: no subvolume 
in layout for path=/, checking on all the subvols to see if it is a 
directory
[2009-09-18 20:06:18] D [dht-common.c:113:dht_lookup_dir_cbk] unify: 
lookup of / on block0 returned error (Transport endpoint is not connected)
[2009-09-18 20:06:18] D [dht-common.c:113:dht_lookup_dir_cbk] unify: 
lookup of / on block1 returned error (Transport endpoint is not connected)
[2009-09-18 20:06:18] D [dht-common.c:113:dht_lookup_dir_cbk] unify: 
lookup of / on block2 returned error (Transport endpoint is not connected)
[2009-09-18 20:06:18] D [fuse-bridge.c:2385:fuse_root_lookup_cbk] fuse: 
first lookup on root failed.
[2009-09-18 20:06:18] W [fuse-bridge.c:1841:fuse_statfs_cbk] 
glusterfs-fuse: 2: ERR => -1 (Transport endpoint is not connected)



><>
Nathan Stratton                                CTO, BlinkMind, Inc.
nathan at robotics.net                         nathan at blinkmind.com
http://www.robotics.net                        http://www.blinkmind.com



More information about the Gluster-users mailing list