[Gluster-users] Protocol stacking: gluster over NFS

[harry mangalam]

Well, it was too clever for me too :) - someone else suggested it when I was 
describing some of the options we were facing. I admit to initially thinking 
that it was silly to expect better performance by stacking protocols, but we 
tried it and it seems to have worked.

To your point:

the 'client' is the end node that uses the gluster storage - in our case it's 
a compute node (w/ limited storage) in a research cluster.

the 'server' is the collection of nodes that provides the gluster storage.

the client mounts the server with the native gluster client, providing all the 
gluster advantages of single namespace, scalability, reliability, etc. to 
'client:/glmount'

the client then exports that gluster fs via NFS to itself, so 
'client:/glmount' is listed in '/etc/exports' as rw to itself.

the client then mounts itself (innuendo and disturbing mental images 
notwithstanding) via NFS:
'mount -t nfs localhost:/glmount /glnfs'
so that the gluster fs (/glmount) is NFS-loopback-mounted on the client 
(itself):

from our test case, simplified:
-----------------------------------
hmangala at claw5:~
$ cat /etc/mtab  # (all non-gluster-related entries deleted)
...
pbs1ib:/gli    /glmount          fuse.glusterfs   \  
             rw,default_permissions,allow_other,max_read=131072  0 0 
...
claw5:/glmount      /glnfs   nfs       rw,addr=10.255.78.4        0 0
...
-----------------------------------

in the above extract, pbs1ib:/gli is the gluster fs that is mounted to 
'claw5:/glmount'.

claw5 then NFS-mounts claw5:/glmount onto /glnfs which users actually use to 
read/write.

I agree, not very intuitive... but it seems to work.  

This is with NFS3 clients.  NFS4 may provide an additional perf boost by 
allowing clients to work out of cache until it's forced to sync, but we 
haven't tried that yet and the test methodology we used wouldn't show a gain 
anyway.  I'll have to try to create a more realistic test harness.


hjm

On Friday, September 14, 2012 01:04:59 PM Whit Blauvelt wrote:
> On Fri, Sep 14, 2012 at 09:41:42AM -0700, harry mangalam wrote:
> > > > What I mean:
> > > > - mounting a gluster fs via the native client,
> > > > - then NFS-exporting the gluster fs to the client itself
> > > > - then mounting that gluster fs via NFS3 to take advantage of the
> > > > client-side caching.
> 
> Harry,
> 
> What is "the client itself" here? I'm having trouble picturing what's doing
> what with what. No doubt because it's too clever for me. Maybe a bit more
> description would clarify it nonetheless.
> 
> Thanks,
> Whit
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
-- 
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
--
What does it say about a society that would rather send its 
children to kill and die for oil than to get on a bike?