[Gluster-users] Protocol stacking: gluster over NFS

[John Mark Walker]

A note on recent history:

There were past attempts to export GlusterFS client mounts over NFS, but those used the GlusterFS NFS service. I believe this is the first instance "in the wild" of someone trying this with knfsd. 

With the former, while there was increased performance, there would invariably be race conditions that would lock up GlusterFS. See the ominous warnings posted on this Q&A thread: http://community.gluster.org/a/nfs-performance-with-fuse-client-redundancy/

I am curious to see if using knfsd, as opposed to GlusterFS' NFS service, yields a long-term solution for this type of workload. Please do continue to keep us updated.

Thanks,
JM



----- Original Message -----
> Well, it was too clever for me too :) - someone else suggested it
> when I was
> describing some of the options we were facing. I admit to initially
> thinking
> that it was silly to expect better performance by stacking protocols,
> but we
> tried it and it seems to have worked.
> 
> To your point:
> 
> the 'client' is the end node that uses the gluster storage - in our
> case it's
> a compute node (w/ limited storage) in a research cluster.
> 
> the 'server' is the collection of nodes that provides the gluster
> storage.
> 
> the client mounts the server with the native gluster client,
> providing all the
> gluster advantages of single namespace, scalability, reliability,
> etc. to
> 'client:/glmount'
> 
> the client then exports that gluster fs via NFS to itself, so
> 'client:/glmount' is listed in '/etc/exports' as rw to itself.
> 
> the client then mounts itself (innuendo and disturbing mental images
> notwithstanding) via NFS:
> 'mount -t nfs localhost:/glmount /glnfs'
> so that the gluster fs (/glmount) is NFS-loopback-mounted on the
> client
> (itself):
> 
> from our test case, simplified:
> -----------------------------------
> hmangala at claw5:~
> $ cat /etc/mtab  # (all non-gluster-related entries deleted)
> ...
> pbs1ib:/gli    /glmount          fuse.glusterfs   \
>              rw,default_permissions,allow_other,max_read=131072  0 0
> ...
> claw5:/glmount      /glnfs   nfs       rw,addr=10.255.78.4        0 0
> ...
> -----------------------------------
> 
> in the above extract, pbs1ib:/gli is the gluster fs that is mounted
> to
> 'claw5:/glmount'.
> 
> claw5 then NFS-mounts claw5:/glmount onto /glnfs which users actually
> use to
> read/write.
> 
> I agree, not very intuitive... but it seems to work.
> 
> This is with NFS3 clients.  NFS4 may provide an additional perf boost
> by
> allowing clients to work out of cache until it's forced to sync, but
> we
> haven't tried that yet and the test methodology we used wouldn't show
> a gain
> anyway.  I'll have to try to create a more realistic test harness.
> 
> 
> hjm
> 
> On Friday, September 14, 2012 01:04:59 PM Whit Blauvelt wrote:
> > On Fri, Sep 14, 2012 at 09:41:42AM -0700, harry mangalam wrote:
> > > > > What I mean:
> > > > > - mounting a gluster fs via the native client,
> > > > > - then NFS-exporting the gluster fs to the client itself
> > > > > - then mounting that gluster fs via NFS3 to take advantage of
> > > > > the
> > > > > client-side caching.
> > 
> > Harry,
> > 
> > What is "the client itself" here? I'm having trouble picturing
> > what's doing
> > what with what. No doubt because it's too clever for me. Maybe a
> > bit more
> > description would clarify it nonetheless.
> > 
> > Thanks,
> > Whit
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> --
> Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
> [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
> 415 South Circle View Dr, Irvine, CA, 92697 [shipping]
> MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
> --
> What does it say about a society that would rather send its
> children to kill and die for oil than to get on a bike?
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>