[Gluster-users] IPoIB Volume (3.3b3) started but not online, not mountable

[Joe Landman]

On 04/18/2012 06:58 PM, Harry Mangalam wrote:
> And one more observation that will probably be obvious in retrospect. If
> you enable auth.allow (on 3.3b3), it will do reverse lookups to verify
> hostnames so it will be more complicated to share an IPoIB gluster
> volume to IPoEth clients.
>
> I had been overriding DNS entries with /etc/hosts entries, but the
> auth.allow option will prevent that hack.
>
> If anyone knows how to share an IPoIB volume to ethernet clients in a
> more formally correct way, I'd be happy to learn of it.

After dealing with problems in multi-modal networks with slightly 
different naming schemes, I don't recommend using tcp and RDMA together 
(or even IPoIB with eth) for Gluster.  Very long, very painful saga. 
Executive summary:  here be dragons.

Also, IPoIB is very leaky.  So under heavy load, you can find your 
servers starting to run out of memory.  We've seen this with OFED 
through 1.5.3.x and Glusters as late as 3.2.6.

We'd recommend sticking to one fabric for the moment with Gluster.  Use 
real tcp with a 10 or 40 GbE backbone.  Far fewer problems.  Much less 
excitement.

Regards,

Joe


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615