[Gluster-users] NFS replacement

[Stephan von Krawczynski]

On Tue, 01 Sep 2009 11:33:38 +0530
Shehjar Tikoo <shehjart at gluster.com> wrote:

> Stephan von Krawczynski wrote:
> > On Mon, 31 Aug 2009 19:48:46 +0530 Shehjar Tikoo 
> > <shehjart at gluster.com> wrote:
> > 
> >> Stephan von Krawczynski wrote:
> >>> Hello all,
> >>> 
> >>> after playing around for some weeks we decided to make some real
> >>>  world tests with glusterfs. Therefore we took a nfs-client and 
> >>> mounted the very same data with glusterfs. The client does some 
> >>> logfile processing every 5 minutes and needs around 3,5 mins 
> >>> runtime in a nfs setup. We found out that it makes no sense to 
> >>> try this setup with gluster replicate as long as we do not have 
> >>> the same performance in a single server setup with glusterfs. So
> >>>  now we have one server mounted (halfway replicate) and would
> >>> like to tune performance. Does anyone have experience with some 
> >>> simple replacement like that? We had to find out that almost all
> >>>  performance options have exactly zero effect. The only thing
> >>> that seems to make at least some difference is read-ahead on the
> >>>  server. We end up with around 4,5 - 5,5 minutes runtime of the 
> >>> scripts, which is on the edge as we need something quite below 5
> >>>  minutes (just like nfs was). Our goal is to maximise performance
> >>>  in this setup and then try a real replication setup with two 
> >>> servers. The load itselfs looks like around 100 scripts starting
> >>>  at one time and processing their data.
> >>> 
> >>> Any ideas?
> >>> 
> >> What nfs server are you using? The in-kernel one?
> > 
> > Yes.
> > 
> >> You could try the unfs3booster server, which is the original unfs3 
> >> with our modifications for bug fixes and slight performance 
> >> improvements. It should give better performance in certain cases 
> >> since it avoids the FUSE bottleneck on the server.
> >> 
> >> For more info, do take a look at this page: 
> >> http://www.gluster.org/docs/index.php/Unfs3boosterConfiguration
> >> 
> >> When using unfs3booster, please use GlusterFS release 2.0.6 since 
> >> that has the required changes to make booster work with NFS.
> > 
> > I read the docs, but I don't understand the advantage. Why should we
> >  use nfs as kind of a transport layer to an underlying glusterfs 
> > server, when we can easily export the service (i.e. glusterfs) 
> > itself. Remember, we don't want nfs on the client any longer, but a 
> > replicate setup with two servers (though we do not use it right now,
> >  but nevertheless it stays our primary goal).
> 
> Ok. My answer was simply under the impression that moving to NFS
> was the motive. unfs3booster-over-gluster is a better solution as
> opposed to having kernel-nfs-over-gluster because of the avoidance of
> the FUSE layer completely.

Sorry. To make that one clear again: I don't want to use NFS if not ultimately
necessary. I would be happy to use a complete glusterfs environment without
any patches and glues to nfs, cifs or the like. 
> > It sounds obvious to me
> > that a nfs-over-gluster must be slower than a pure kernel-nfs. On the
> >  other hand glusterfs per se may even have some advantages on the 
> > network side, iff performance tuning (and of course the options 
> > themselves) is well designed. The first thing we noticed is that load
> >  dropped dramatically both on server and client when not using 
> > kernel-nfs. Client dropped from around 20 to around 4. Server dropped
> >  from around 10 to around 5. Since all boxes are pretty much 
> > dedicated to their respective jobs a lot of caching is going on 
> > anyways.
> Thanks, that is useful information.
> 
> So I
> > would not expect nfs to have advantages only because it is 
> > kernel-driven. And the current numbers (loss of around 30% in 
> > performance) show that nfs performance is not completely out of 
> > reach.
> That is true, we do have setups performing as well and in some
> cases better than kernel NFS despite the replication overhead. It
> is a matter of testing and arriving at a config that works for your
> setup.
> 
> 
> > 
> > What advantages would you expect from using unfs3booster at all?
> > 
> To begin with, unfs3booster must be compared against kernel nfsd and not
> against a GlusterFS-only config. So when comparing with kernel-nfsd, one
> should understand that knfsd involves the FUSE layer, kernel's VFS and
> network layer, all of which have their advantages and also
> disadvantages, especially FUSE when using with the kernel nfsd. Those
> bottlenecks with FUSE+knfsd interaction are well documented elsewhere.
> 
> unfs3booster enables you to avoid the FUSE layer, the VFS, etc and talk
> directly to the network and through that, to the GlusterFS server. In
> our measurements, we found that we could perform better than kernel
> nfs-over-gluster by avoiding FUSE and using our own caching(io-cache),
> buffering(write-behind, read-ahead) and request scheduling(io-threads).
> 
> > Another thing we really did not understand is the _negative_ effect 
> > of adding iothreads on client or server. Our nfs setup needs around 
> > 90 nfs kernel threads to run smoothly. Every number greater than 8 
> > iothreads reduces the performance of glusterfs measurably.
> > 
> 
> The main reason why knfsds need a higher number of threads is simply
> because knfsd threads are highly io-bound, that is they wait for for the
> disk IO to complete in order to serve each NFS request.
> 
> On the other hand, with io-threads, the right number actually depends on
> the point at which io-threads are being used. For eg, if you're using
> io-threads just above the posix or  features/locks, the scenario is much
> like kernel nfsd threads, where each io-thread blocks till the disk IO
> is complete. Is this is where you've observed that 8 iothread drop-off?
> If so, then it is something we'll need to investigate.
> 
> The other place where you can you can use io-threads is on the GlusterFS
> client side, it is here that the 8 thread drop-off seems possible since
> the client side in GlusterFS is more CPU hungry than the server, and it
> is possible that 8 io-threads are able to consume as much CPU as is
> available for GlusterFS. Have you observed what the CPU usage figures
> are as you increase the number of io-threads?
> 
> How many CPUs did the machine have when you observed the drop-off beyond
> 8 threads?

The client box has a Quad Core 2 CPU, the server is dual AMD Opteron 246.
I currently try a pretty simple setup with this client vol file:

volume remote1
  type protocol/client
  option transport-type tcp
  option remote-host 192.168.82.1
  option remote-subvolume testfs
end-volume

volume remote2
  type protocol/client
  option transport-type tcp
  option remote-host 192.168.82.2
  option remote-subvolume testfs
end-volume

volume replicate
  type cluster/replicate
  option data-self-heal on
  option metadata-self-heal on
  option entry-self-heal on
  subvolumes remote1 remote2
end-volume

volume writebehind
  type performance/write-behind
  #option aggregate-size 1MB # option is unkown
  # option window-size 1MB
  option cache-size 2MB
  #option block-size 1MB # option is unkown
  option flush-behind on
  subvolumes replicate
end-volume

I tried read-ahead and others but they don't boost performance at all.
Only writebehind has some positive effect of around 30s runtime (drops from
around 5 mins to 4:30 mins).
Unfortunately I need a further improvement down to around 4 mins runtime,
because now every now and then the 5 mins barrier is hit.

Remember that the remote2 server is down in this setup. We use only remote1
currently.
Do you have any ideas how to improve performance?
 
> -Shehjar

-- 
Regards,
Stephan