[Gluster-devel] Infiniband help

Sat Oct 20 13:00:22 UTC 2007

This is now working! I had an error in one of the config files.

><>
Nathan Stratton                         CTO, Voila IP Communications
nathan at robotics.net                  nathan at voilaip.com
http://www.robotics.net                 http://www.voilaip.com

On Fri, 19 Oct 2007, Nathan Allen Stratton wrote:

> On Fri, 19 Oct 2007, Anand Avati wrote:
>
> > Nathan,
> >   if you have IPoIB working, using ib-verbs should be straight forward. Just
> > use the IPoIB's IP addresses and transport-type as ib-verbs/{client,server}
> > and things should just work.
> >
> > ib-sdp was a 'stopgap' solution when the ib-verbs driver was not yet
> > implemented. You need ib_sdp kernel module for it to work. Also make sure
> > you have 'ib_uverbs' kernel module for ib-verbs to work. If still things
> > dont work, run both the server and client with -LDEBUG and attach the logs.
>
> Server:
>
> 2007-10-19 20:36:42 D [client-protocol.c:4294:client_protocol_reconnect]
> brick-a-ns: attempting reconnect
> 2007-10-19 20:36:42 D [ib-verbs-client.c:188:ib_verbs_client_connect]
> brick-a-ns: connection on 5 success, attempting to handshake
> 2007-10-19 20:36:42 E [ib-verbs.c:762:ib_verbs_handshake]
> transport/ib-verbs: brick-a-ns: could not send IB handshake data
> 2007-10-19 20:36:42 E [ib-verbs-client.c:197:ib_verbs_client_connect]
> brick-a-ns: ib_verbs_handshake failed
> 2007-10-19 20:36:42 D [client-protocol.c:4294:client_protocol_reconnect]
> brick-b-ns: attempting reconnect
> 2007-10-19 20:36:42 D [ib-verbs-client.c:188:ib_verbs_client_connect]
> brick-b-ns: connection on 5 success, attempting to handshake
> 2007-10-19 20:36:42 E [ib-verbs.c:762:ib_verbs_handshake]
> transport/ib-verbs: brick-b-ns: could not send IB handshake data
> 2007-10-19 20:36:42 E [ib-verbs-client.c:197:ib_verbs_client_connect]
> brick-b-ns: ib_verbs_handshake failed
> 2007-10-19 20:36:42 D [client-protocol.c:4294:client_protocol_reconnect]
> brick-a: attempting reconnect
> 2007-10-19 20:36:42 D [ib-verbs-client.c:188:ib_verbs_client_connect]
> brick-a: connection on 5 success, attempting to handshake
> 2007-10-19 20:36:42 E [ib-verbs.c:762:ib_verbs_handshake]
> transport/ib-verbs: brick-a: could not send IB handshake data
> 2007-10-19 20:36:42 E [ib-verbs-client.c:197:ib_verbs_client_connect]
> brick-a: ib_verbs_handshake failed
> 2007-10-19 20:36:42 D [client-protocol.c:4294:client_protocol_reconnect]
> mirror-a: attempting reconnect
> 2007-10-19 20:36:42 D [ib-verbs-client.c:188:ib_verbs_client_connect]
> mirror-a: connection on 5 success, attempting to handshake
> 2007-10-19 20:36:42 E [ib-verbs.c:762:ib_verbs_handshake]
> transport/ib-verbs: mirror-a: could not send IB handshake data
> 2007-10-19 20:36:42 E [ib-verbs-client.c:197:ib_verbs_client_connect]
> mirror-a: ib_verbs_handshake failed
> 2007-10-19 20:36:42 D [client-protocol.c:4294:client_protocol_reconnect]
> brick-b: attempting reconnect
> 2007-10-19 20:36:42 D [ib-verbs-client.c:188:ib_verbs_client_connect]
> brick-b: connection on 5 success, attempting to handshake
> 2007-10-19 20:36:42 E [ib-verbs.c:762:ib_verbs_handshake]
> transport/ib-verbs: brick-b: could not send IB handshake data
> 2007-10-19 20:36:42 E [ib-verbs-client.c:197:ib_verbs_client_connect]
> brick-b: ib_verbs_handshake failed
> 2007-10-19 20:36:42 D [client-protocol.c:4294:client_protocol_reconnect]
> mirror-c: attempting reconnect
> 2007-10-19 20:36:42 D [ib-verbs-client.c:188:ib_verbs_client_connect]
> mirror-c: connection on 5 success, attempting to handshake
> 2007-10-19 20:36:42 E [ib-verbs.c:762:ib_verbs_handshake]
> transport/ib-verbs: mirror-c: could not send IB handshake data
> 2007-10-19 20:36:42 E [ib-verbs-client.c:197:ib_verbs_client_connect]
> mirror-c: ib_verbs_handshake failed
>
> Client:
>
> 2007-10-19 20:42:56 D [glusterfs.c:138:get_spec_fp] glusterfs: loading
> spec from /usr/local/etc/glusterfs/client.vol
> 2007-10-19 20:42:56 W [fuse-bridge.c:2100:fuse_transport_notify]
> glusterfs-fuse: Ignoring notify event 4
> 2007-10-19 20:42:56 D [spec.y:116:new_section] libglusterfs/parser: New
> node for 'share'
> 2007-10-19 20:42:56 D [spec.y:132:section_type] libglusterfs/parser:
> Type:share:protocol/client
> 2007-10-19 20:42:56 D [xlator.c:102:xlator_set_type] libglusterfs/xlator:
> attempt to load type protocol/client
> 2007-10-19 20:42:56 D [xlator.c:109:xlator_set_type] libglusterfs/xlator:
> attempt to load file
> /usr/local/lib/glusterfs/1.3.6/xlator/protocol/client.so
> 2007-10-19 20:42:56 D [spec.y:152:section_option] libglusterfs/parser:
> Option:share:transport-type:ib-verbs/client
> 2007-10-19 20:42:56 D [spec.y:152:section_option] libglusterfs/parser:
> Option:share:remote-host:192.168.0.12
> 2007-10-19 20:42:56 D [spec.y:152:section_option] libglusterfs/parser:
> Option:share:remote-subvolume:share
> 2007-10-19 20:42:56 D [spec.y:216:section_end] libglusterfs/parser:
> end:share
> 2007-10-19 20:42:56 D [spec.y:116:new_section] libglusterfs/parser: New
> node for 'writeback'
> 2007-10-19 20:42:56 D [spec.y:132:section_type] libglusterfs/parser:
> Type:writeback:performance/write-behind
> 2007-10-19 20:42:56 D [xlator.c:102:xlator_set_type] libglusterfs/xlator:
> attempt to load type performance/write-behind
> 2007-10-19 20:42:56 D [xlator.c:109:xlator_set_type] libglusterfs/xlator:
> attempt to load file
> /usr/local/lib/glusterfs/1.3.6/xlator/performance/write-behind.so
> 2007-10-19 20:42:56 W [xlator.c:156:xlator_set_type] libglusterfs/xlator:
> dlsym(notify) on
> /usr/local/lib/glusterfs/1.3.6/xlator/performance/write-behind.so:
> undefined sym
> bol: notify -- neglecting
> 2007-10-19 20:42:56 D [spec.y:152:section_option] libglusterfs/parser:
> Option:writeback:aggregate-size:131072
> 2007-10-19 20:42:56 D [spec.y:201:section_sub] liglusterfs/parser:
> child:writeback->share
> 2007-10-19 20:42:56 D [spec.y:216:section_end] libglusterfs/parser:
> end:writeback
> 2007-10-19 20:42:56 D [spec.y:116:new_section] libglusterfs/parser: New
> node for 'readahead'
> 2007-10-19 20:42:56 D [spec.y:132:section_type] libglusterfs/parser:
> Type:readahead:performance/read-ahead
> 2007-10-19 20:42:56 D [xlator.c:102:xlator_set_type] libglusterfs/xlator:
> attempt to load type performance/read-ahead
> 2007-10-19 20:42:56 D [xlator.c:109:xlator_set_type] libglusterfs/xlator:
> attempt to load file
> /usr/local/lib/glusterfs/1.3.6/xlator/performance/read-ahead.so
> 2007-10-19 20:42:56 W [xlator.c:156:xlator_set_type] libglusterfs/xlator:
> dlsym(notify) on
> /usr/local/lib/glusterfs/1.3.6/xlator/performance/read-ahead.so: undefined
> symbo
> l: notify -- neglecting
> 2007-10-19 20:42:56 D [spec.y:152:section_option] libglusterfs/parser:
> Option:readahead:page-size:65536
> 2007-10-19 20:42:56 D [spec.y:152:section_option] libglusterfs/parser:
> Option:readahead:page-count:16
> 2007-10-19 20:42:56 D [spec.y:201:section_sub] liglusterfs/parser:
> child:readahead->writeback
> 2007-10-19 20:42:56 D [spec.y:216:section_end] libglusterfs/parser:
> end:readahead
> 2007-10-19 20:42:56 W [inode.c:1099:inode_table_new] fuse: creating new
> inode table with lru_limit=1024, sizeof(inode_t)=156
> 2007-10-19 20:42:56 D [inode.c:559:__create_inode] fuse/inode: create
> inode(1)
> 2007-10-19 20:42:56 D [inode.c:351:__active_inode] fuse/inode: activating
> inode(1), lru=0/1024
> 2007-10-19 20:42:56 D [client-protocol.c:4549:init] share: missing
> 'inode-lru-limit'. defaulting to 1000
> 2007-10-19 20:42:56 D [client-protocol.c:4566:init] share: defaulting
> transport-timeout to 108
> 2007-10-19 20:42:56 D [transport.c:83:transport_load]
> libglusterfs/transport: attempt to load type ib-verbs/client
> 2007-10-19 20:42:56 D [transport.c:88:transport_load]
> libglusterfs/transport: attempt to load file
> /usr/local/lib/glusterfs/1.3.6/transport/ib-verbs/client.so
> 2007-10-19 20:42:56 D [ib-verbs.c:1165:ib_verbs_options_init]
> transport/ib-verbs: share: defaulting MTU to '2048'
> 2007-10-19 20:42:56 D [write-behind.c:845:init] writeback: using
> aggregate-size = 131072
> 2007-10-19 20:42:56 D [read-ahead.c:841:init] readahead: Using
> conf->page_size = 0x10000
> 2007-10-19 20:42:56 D [read-ahead.c:848:init] readahead: Using
> conf->page_count = 0x10
> 2007-10-19 20:42:56 D [client-protocol.c:4847:notify] share: got
> GF_EVENT_PARENT_UP, attempting connect on transport
> 2007-10-19 20:42:56 D [client-protocol.c:4294:client_protocol_reconnect]
> share: attempting reconnect
> 2007-10-19 20:42:56 D [ib-verbs-client.c:70:ib_verbs_client_connect]
> share: socket fd = 11
> 2007-10-19 20:42:56 D [ib-verbs-client.c:88:ib_verbs_client_connect]
> share: finalized on port `1023'
> 2007-10-19 20:42:56 D [ib-verbs-client.c:109:ib_verbs_client_connect]
> share: defaulting remote-port to 6996
> 2007-10-19 20:42:56 D [common-utils.c:161:gf_resolve_ip] resolver: DNS
> cache not present, freshly probing hostname: 192.168.0.12
> 2007-10-19 20:42:56 D [common-utils.c:186:gf_resolve_ip] resolver:
> returning IP:192.168.0.12[0] for hostname: 192.168.0.12
> 2007-10-19 20:42:56 D [common-utils.c:194:gf_resolve_ip] resolver:
> flushing DNS cache
> 2007-10-19 20:42:56 D [ib-verbs-client.c:142:ib_verbs_client_connect]
> share: connect on 11 in progress (non-blocking)
> 2007-10-19 20:42:56 D [ib-verbs-client.c:188:ib_verbs_client_connect]
> share: connection on 11 success, attempting to handshake
> 2007-10-19 20:42:56 E [ib-verbs.c:772:ib_verbs_handshake]
> transport/ib-verbs: share: could not recv IB handshake-2 data
> 2007-10-19 20:42:56 E [ib-verbs-client.c:197:ib_verbs_client_connect]
> share: ib_verbs_handshake failed
> 2007-10-19 20:42:57 D [client-protocol.c:4294:client_protocol_reconnect]
> share: attempting reconnect
> 2007-10-19 20:42:57 D [ib-verbs-client.c:188:ib_verbs_client_connect]
> share: connection on 11 success, attempting to handshake
> 2007-10-19 20:42:57 E [ib-verbs.c:762:ib_verbs_handshake]
> transport/ib-verbs: share: could not send IB handshake data
> 2007-10-19 20:42:57 E [ib-verbs-client.c:197:ib_verbs_client_connect]
> share: ib_verbs_handshake failed
> 2007-10-19 20:43:00 D [client-protocol.c:4294:client_protocol_reconnect]
> share: attempting reconnect
> 2007-10-19 20:43:00 D [ib-verbs-client.c:188:ib_verbs_client_connect]
> share: connection on 11 success, attempting to handshake
> 2007-10-19 20:43:00 E [ib-verbs.c:762:ib_verbs_handshake]
> transport/ib-verbs: share: could not send IB handshake data
> 2007-10-19 20:43:00 E [ib-verbs-client.c:197:ib_verbs_client_connect]
> share: ib_verbs_handshake failed
> 2007-10-19 20:43:03 D [client-protocol.c:4294:client_protocol_reconnect]
> share: attempting reconnect
> 2007-10-19 20:43:03 D [ib-verbs-client.c:188:ib_verbs_client_connect]
> share: connection on 11 success, attempting to handshake
> 2007-10-19 20:43:03 E [ib-verbs.c:762:ib_verbs_handshake]
> transport/ib-verbs: share: could not send IB handshake data
> 2007-10-19 20:43:03 E [ib-verbs-client.c:197:ib_verbs_client_connect]
> share: ib_verbs_handshake failed
> 2007-10-19 20:43:08 D [client-protocol.c:4294:client_protocol_reconnect]
> share: attempting reconnect
> 2007-10-19 20:43:08 D [ib-verbs-client.c:188:ib_verbs_client_connect]
> share: connection on 11 success, attempting to handshake
> 2007-10-19 20:43:08 E [ib-verbs.c:762:ib_verbs_handshake]
> transport/ib-verbs: share: could not send IB handshake data
> 2007-10-19 20:43:08 E [ib-verbs-client.c:197:ib_verbs_client_connect]
> share: ib_verbs_handshake failed
> 2007-10-19 20:43:16 D [client-protocol.c:4294:client_protocol_reconnect]
> share: attempting reconnect
> 2007-10-19 20:43:16 D [ib-verbs-client.c:188:ib_verbs_client_connect]
> share: connection on 11 success, attempting to handshake
> 2007-10-19 20:43:16 E [ib-verbs.c:762:ib_verbs_handshake]
> transport/ib-verbs: share: could not send IB handshake data
> 2007-10-19 20:43:16 E [ib-verbs-client.c:197:ib_verbs_client_connect]
> share: ib_verbs_handshake failed
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>