[Gluster-users] simultaneous Gbe/IPoIB clients?

[Harry Mangalam]

We have a use case in our clusters where we're adding IB to some racks  
and would like to use gluster storage over IB in that mix.  If it 
works out on the cluster case, it would be a good argument to scale 
this out much further.  

So, is there an elegant, best-practices-way to use a gluster volume 
that was established using the IPoIB net to provide storage to the 
IPoGBe clients without routing thru a single gateway (and therefore 
losing the multipath advantage of gluster) or creating a DNS hairball?

ie: The IPoIB net is 1.2.3.0 (host names are referenced as 'pbsXib') 
and the IPoGbe net is 1.2.4.0 (names are 'pbsXgb'). All gluster 
storage nodes have 1 IB port on 1.2.3.0 and one Gbe port on 1.2.4.0)

The gluster volume was set up as follows and the host names are 
defined in the local /etc/hosts files for now):

gluster volume create gl \
    	transport tcp,rdma \
	  pbs1ib:/bducgl \
	  pbs2ib:/bducgl \
	  pbs3ib:/bducgl \
	  pbs4ib:/bducgl 

The IPoIB clients (which also have the same /etc/hosts info) have no 
problem accessing the volume as 'pbs1ib:/gl' (and have good large-file 
IO, maxing out at about 250MB/s on a single file copy)

The problem (from the gl.log (5.8K):
<http://moo.nac.uci.edu/~hjm/gl.log>

seems to be that the Gbe gluster client can contact the Gbe side of 
the gluster volume but cannot resolve the gluster volume since the 
gluster volume was created with pbsXib names.

I can bypass the DNS by pointing the pbsXib names -> pbsXgb addresses 
on a client (in /etc/hosts) and in fact this works fine and I have the 
gluster volume reading and writing clients from both IB and Gbe.

The problem is that this leaves a fragile, tangled nest of DNS 
overrides and manual edits.  Setting this up in the DNS tables would 
solve most of this but the servers would still have to have overrides 
set up in their /etc/hosts to override the DNS.

Is there a more elegant way of doing this?
This is 3.3b3, (which was fairly easy to set up).

Also, with dual-port IB cards, is it recommended to do channel-bonding 
or to use different IP #s?

hjm

-- 
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[ZOT 2225] / 92697  Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120416/8147fef3/attachment.html>