[Gluster-devel] Infiniband help

Sat Oct 20 01:45:26 UTC 2007

On Fri, 19 Oct 2007, Anand Avati wrote:

> Nathan,
>   if you have IPoIB working, using ib-verbs should be straight forward. Just
> use the IPoIB's IP addresses and transport-type as ib-verbs/{client,server}
> and things should just work.
>
> ib-sdp was a 'stopgap' solution when the ib-verbs driver was not yet
> implemented. You need ib_sdp kernel module for it to work. Also make sure
> you have 'ib_uverbs' kernel module for ib-verbs to work. If still things
> dont work, run both the server and client with -LDEBUG and attach the logs.

Server:

2007-10-19 20:36:42 D [client-protocol.c:4294:client_protocol_reconnect]
brick-a-ns: attempting reconnect
2007-10-19 20:36:42 D [ib-verbs-client.c:188:ib_verbs_client_connect]
brick-a-ns: connection on 5 success, attempting to handshake
2007-10-19 20:36:42 E [ib-verbs.c:762:ib_verbs_handshake]
transport/ib-verbs: brick-a-ns: could not send IB handshake data
2007-10-19 20:36:42 E [ib-verbs-client.c:197:ib_verbs_client_connect]
brick-a-ns: ib_verbs_handshake failed
2007-10-19 20:36:42 D [client-protocol.c:4294:client_protocol_reconnect]
brick-b-ns: attempting reconnect
2007-10-19 20:36:42 D [ib-verbs-client.c:188:ib_verbs_client_connect]
brick-b-ns: connection on 5 success, attempting to handshake
2007-10-19 20:36:42 E [ib-verbs.c:762:ib_verbs_handshake]
transport/ib-verbs: brick-b-ns: could not send IB handshake data
2007-10-19 20:36:42 E [ib-verbs-client.c:197:ib_verbs_client_connect]
brick-b-ns: ib_verbs_handshake failed
2007-10-19 20:36:42 D [client-protocol.c:4294:client_protocol_reconnect]
brick-a: attempting reconnect
2007-10-19 20:36:42 D [ib-verbs-client.c:188:ib_verbs_client_connect]
brick-a: connection on 5 success, attempting to handshake
2007-10-19 20:36:42 E [ib-verbs.c:762:ib_verbs_handshake]
transport/ib-verbs: brick-a: could not send IB handshake data
2007-10-19 20:36:42 E [ib-verbs-client.c:197:ib_verbs_client_connect]
brick-a: ib_verbs_handshake failed
2007-10-19 20:36:42 D [client-protocol.c:4294:client_protocol_reconnect]
mirror-a: attempting reconnect
2007-10-19 20:36:42 D [ib-verbs-client.c:188:ib_verbs_client_connect]
mirror-a: connection on 5 success, attempting to handshake
2007-10-19 20:36:42 E [ib-verbs.c:762:ib_verbs_handshake]
transport/ib-verbs: mirror-a: could not send IB handshake data
2007-10-19 20:36:42 E [ib-verbs-client.c:197:ib_verbs_client_connect]
mirror-a: ib_verbs_handshake failed
2007-10-19 20:36:42 D [client-protocol.c:4294:client_protocol_reconnect]
brick-b: attempting reconnect
2007-10-19 20:36:42 D [ib-verbs-client.c:188:ib_verbs_client_connect]
brick-b: connection on 5 success, attempting to handshake
2007-10-19 20:36:42 E [ib-verbs.c:762:ib_verbs_handshake]
transport/ib-verbs: brick-b: could not send IB handshake data
2007-10-19 20:36:42 E [ib-verbs-client.c:197:ib_verbs_client_connect]
brick-b: ib_verbs_handshake failed
2007-10-19 20:36:42 D [client-protocol.c:4294:client_protocol_reconnect]
mirror-c: attempting reconnect
2007-10-19 20:36:42 D [ib-verbs-client.c:188:ib_verbs_client_connect]
mirror-c: connection on 5 success, attempting to handshake
2007-10-19 20:36:42 E [ib-verbs.c:762:ib_verbs_handshake]
transport/ib-verbs: mirror-c: could not send IB handshake data
2007-10-19 20:36:42 E [ib-verbs-client.c:197:ib_verbs_client_connect]
mirror-c: ib_verbs_handshake failed

Client:

2007-10-19 20:42:56 D [glusterfs.c:138:get_spec_fp] glusterfs: loading
spec from /usr/local/etc/glusterfs/client.vol
2007-10-19 20:42:56 W [fuse-bridge.c:2100:fuse_transport_notify]
glusterfs-fuse: Ignoring notify event 4
2007-10-19 20:42:56 D [spec.y:116:new_section] libglusterfs/parser: New
node for 'share'
2007-10-19 20:42:56 D [spec.y:132:section_type] libglusterfs/parser:
Type:share:protocol/client
2007-10-19 20:42:56 D [xlator.c:102:xlator_set_type] libglusterfs/xlator:
attempt to load type protocol/client
2007-10-19 20:42:56 D [xlator.c:109:xlator_set_type] libglusterfs/xlator:
attempt to load file
/usr/local/lib/glusterfs/1.3.6/xlator/protocol/client.so
2007-10-19 20:42:56 D [spec.y:152:section_option] libglusterfs/parser:
Option:share:transport-type:ib-verbs/client
2007-10-19 20:42:56 D [spec.y:152:section_option] libglusterfs/parser:
Option:share:remote-host:192.168.0.12
2007-10-19 20:42:56 D [spec.y:152:section_option] libglusterfs/parser:
Option:share:remote-subvolume:share
2007-10-19 20:42:56 D [spec.y:216:section_end] libglusterfs/parser:
end:share
2007-10-19 20:42:56 D [spec.y:116:new_section] libglusterfs/parser: New
node for 'writeback'
2007-10-19 20:42:56 D [spec.y:132:section_type] libglusterfs/parser:
Type:writeback:performance/write-behind
2007-10-19 20:42:56 D [xlator.c:102:xlator_set_type] libglusterfs/xlator:
attempt to load type performance/write-behind
2007-10-19 20:42:56 D [xlator.c:109:xlator_set_type] libglusterfs/xlator:
attempt to load file
/usr/local/lib/glusterfs/1.3.6/xlator/performance/write-behind.so
2007-10-19 20:42:56 W [xlator.c:156:xlator_set_type] libglusterfs/xlator:
dlsym(notify) on
/usr/local/lib/glusterfs/1.3.6/xlator/performance/write-behind.so:
undefined sym
bol: notify -- neglecting
2007-10-19 20:42:56 D [spec.y:152:section_option] libglusterfs/parser:
Option:writeback:aggregate-size:131072
2007-10-19 20:42:56 D [spec.y:201:section_sub] liglusterfs/parser:
child:writeback->share
2007-10-19 20:42:56 D [spec.y:216:section_end] libglusterfs/parser:
end:writeback
2007-10-19 20:42:56 D [spec.y:116:new_section] libglusterfs/parser: New
node for 'readahead'
2007-10-19 20:42:56 D [spec.y:132:section_type] libglusterfs/parser:
Type:readahead:performance/read-ahead
2007-10-19 20:42:56 D [xlator.c:102:xlator_set_type] libglusterfs/xlator:
attempt to load type performance/read-ahead
2007-10-19 20:42:56 D [xlator.c:109:xlator_set_type] libglusterfs/xlator:
attempt to load file
/usr/local/lib/glusterfs/1.3.6/xlator/performance/read-ahead.so
2007-10-19 20:42:56 W [xlator.c:156:xlator_set_type] libglusterfs/xlator:
dlsym(notify) on
/usr/local/lib/glusterfs/1.3.6/xlator/performance/read-ahead.so: undefined
symbo
l: notify -- neglecting
2007-10-19 20:42:56 D [spec.y:152:section_option] libglusterfs/parser:
Option:readahead:page-size:65536
2007-10-19 20:42:56 D [spec.y:152:section_option] libglusterfs/parser:
Option:readahead:page-count:16
2007-10-19 20:42:56 D [spec.y:201:section_sub] liglusterfs/parser:
child:readahead->writeback
2007-10-19 20:42:56 D [spec.y:216:section_end] libglusterfs/parser:
end:readahead
2007-10-19 20:42:56 W [inode.c:1099:inode_table_new] fuse: creating new
inode table with lru_limit=1024, sizeof(inode_t)=156
2007-10-19 20:42:56 D [inode.c:559:__create_inode] fuse/inode: create
inode(1)
2007-10-19 20:42:56 D [inode.c:351:__active_inode] fuse/inode: activating
inode(1), lru=0/1024
2007-10-19 20:42:56 D [client-protocol.c:4549:init] share: missing
'inode-lru-limit'. defaulting to 1000
2007-10-19 20:42:56 D [client-protocol.c:4566:init] share: defaulting
transport-timeout to 108
2007-10-19 20:42:56 D [transport.c:83:transport_load]
libglusterfs/transport: attempt to load type ib-verbs/client
2007-10-19 20:42:56 D [transport.c:88:transport_load]
libglusterfs/transport: attempt to load file
/usr/local/lib/glusterfs/1.3.6/transport/ib-verbs/client.so
2007-10-19 20:42:56 D [ib-verbs.c:1165:ib_verbs_options_init]
transport/ib-verbs: share: defaulting MTU to '2048'
2007-10-19 20:42:56 D [write-behind.c:845:init] writeback: using
aggregate-size = 131072
2007-10-19 20:42:56 D [read-ahead.c:841:init] readahead: Using
conf->page_size = 0x10000
2007-10-19 20:42:56 D [read-ahead.c:848:init] readahead: Using
conf->page_count = 0x10
2007-10-19 20:42:56 D [client-protocol.c:4847:notify] share: got
GF_EVENT_PARENT_UP, attempting connect on transport
2007-10-19 20:42:56 D [client-protocol.c:4294:client_protocol_reconnect]
share: attempting reconnect
2007-10-19 20:42:56 D [ib-verbs-client.c:70:ib_verbs_client_connect]
share: socket fd = 11
2007-10-19 20:42:56 D [ib-verbs-client.c:88:ib_verbs_client_connect]
share: finalized on port `1023'
2007-10-19 20:42:56 D [ib-verbs-client.c:109:ib_verbs_client_connect]
share: defaulting remote-port to 6996
2007-10-19 20:42:56 D [common-utils.c:161:gf_resolve_ip] resolver: DNS
cache not present, freshly probing hostname: 192.168.0.12
2007-10-19 20:42:56 D [common-utils.c:186:gf_resolve_ip] resolver:
returning IP:192.168.0.12[0] for hostname: 192.168.0.12
2007-10-19 20:42:56 D [common-utils.c:194:gf_resolve_ip] resolver:
flushing DNS cache
2007-10-19 20:42:56 D [ib-verbs-client.c:142:ib_verbs_client_connect]
share: connect on 11 in progress (non-blocking)
2007-10-19 20:42:56 D [ib-verbs-client.c:188:ib_verbs_client_connect]
share: connection on 11 success, attempting to handshake
2007-10-19 20:42:56 E [ib-verbs.c:772:ib_verbs_handshake]
transport/ib-verbs: share: could not recv IB handshake-2 data
2007-10-19 20:42:56 E [ib-verbs-client.c:197:ib_verbs_client_connect]
share: ib_verbs_handshake failed
2007-10-19 20:42:57 D [client-protocol.c:4294:client_protocol_reconnect]
share: attempting reconnect
2007-10-19 20:42:57 D [ib-verbs-client.c:188:ib_verbs_client_connect]
share: connection on 11 success, attempting to handshake
2007-10-19 20:42:57 E [ib-verbs.c:762:ib_verbs_handshake]
transport/ib-verbs: share: could not send IB handshake data
2007-10-19 20:42:57 E [ib-verbs-client.c:197:ib_verbs_client_connect]
share: ib_verbs_handshake failed
2007-10-19 20:43:00 D [client-protocol.c:4294:client_protocol_reconnect]
share: attempting reconnect
2007-10-19 20:43:00 D [ib-verbs-client.c:188:ib_verbs_client_connect]
share: connection on 11 success, attempting to handshake
2007-10-19 20:43:00 E [ib-verbs.c:762:ib_verbs_handshake]
transport/ib-verbs: share: could not send IB handshake data
2007-10-19 20:43:00 E [ib-verbs-client.c:197:ib_verbs_client_connect]
share: ib_verbs_handshake failed
2007-10-19 20:43:03 D [client-protocol.c:4294:client_protocol_reconnect]
share: attempting reconnect
2007-10-19 20:43:03 D [ib-verbs-client.c:188:ib_verbs_client_connect]
share: connection on 11 success, attempting to handshake
2007-10-19 20:43:03 E [ib-verbs.c:762:ib_verbs_handshake]
transport/ib-verbs: share: could not send IB handshake data
2007-10-19 20:43:03 E [ib-verbs-client.c:197:ib_verbs_client_connect]
share: ib_verbs_handshake failed
2007-10-19 20:43:08 D [client-protocol.c:4294:client_protocol_reconnect]
share: attempting reconnect
2007-10-19 20:43:08 D [ib-verbs-client.c:188:ib_verbs_client_connect]
share: connection on 11 success, attempting to handshake
2007-10-19 20:43:08 E [ib-verbs.c:762:ib_verbs_handshake]
transport/ib-verbs: share: could not send IB handshake data
2007-10-19 20:43:08 E [ib-verbs-client.c:197:ib_verbs_client_connect]
share: ib_verbs_handshake failed
2007-10-19 20:43:16 D [client-protocol.c:4294:client_protocol_reconnect]
share: attempting reconnect
2007-10-19 20:43:16 D [ib-verbs-client.c:188:ib_verbs_client_connect]
share: connection on 11 success, attempting to handshake
2007-10-19 20:43:16 E [ib-verbs.c:762:ib_verbs_handshake]
transport/ib-verbs: share: could not send IB handshake data
2007-10-19 20:43:16 E [ib-verbs-client.c:197:ib_verbs_client_connect]
share: ib_verbs_handshake failed