[Gluster-devel] Performance as compared to NFS

[François Thiebolt]

Hi Paul,

On 9/7/07, Paul Jochum <jochum at alcatel-lucent.com> wrote:
>
> Hi All:
>
>     Time to move from functionality (i.e. it is working), to performance
> (need to speed it up now).
>
>     Basically, I have a one server that can run NFS and gluster, and one
> client that can access that server (either NFS or Glusterfs).  In a
> simple set of tests (time of rsyncs, and then time of rm's), I see the
> following:
>
> For NFS:  rsync time is 0m52.304s and rm time is 0m12.615s
>
> For Glusterfs: rsync time is 1m29.312s and rm time is 0m33.901s
> (these were the fastest gluster times, slowest for the rsync was
> 5m23.9s, depending on the perfomance translators I used)
>
> So, how do I improve Gluster, which is running about twice as slow as NFS?


NFS (whatever release) is a kernel implementation whereas fuse-based
filesystems are userland implementations. You can't have better performance
due to the multiples memory copies along with context switching penalties.
Moreover, switching to direct-io avoid kernel page cache duplication but
won't allow you to run application (execve needs byte code to be kernel page
cache resident).

Our scientific paper related to VisageFS wide-area filesystem to appear in
proceedings of PDCS2007, has a fuse overhead evaluation. Roughtly speaking,
it's about 15% penalty related to 128KB reading and only a few percent when
using small read size. When modifying kernel and fuse to read 512KB read
syscalls, application's performance step up from 66Mb/s to 80Mb/s with a 17%
overhead (Ethernet Gigabit Switch).

I recently noticed a glusterFS vs Lustre evaluation that looks strange.
Lustre is one of the most *powerfull* distributed filesystem featuring full
kernel implementation along with out-of-band architecture, write behind,
file joining and a centralized metadata server (OK, glusterFS does not have
a centralized MDS), and optimized specific network drivers. So unless lustre
installation was completly messed up, i can't believe glusterFS has the
ability to outperforms Lustre.

Now, NFSv4.1 tooks ideas from Lustre and pNFS (Panasas NFS). One of the most
awaited feature is the stripping capability, so it'll be of interest to
perform evaluations of userlands FS against such NFS version (anyone
GlusterFS vs pNFS ?)

I think GlusterFS is a geart work but outperforming kernel based FS
featuring out-of-band architecture come as a suprise.

François.

Here is my background info
>
> *Gluster Server:*
> No messages in the log files (either gluster's log file or
> /var/log/messages)
>
> glusterfsd -V
> glusterfs 1.3.1
>
> spec file:
> volume brick
>   type storage/posix
>   option directory /nfs/gluster
> end-volume
> volume server
>   type protocol/server
>   option transport-type tcp/server
>   subvolumes brick
>   option auth.ip.brick.allow *
> end-volume
>
> machine's OS = Scientific Linux SL release 3.0.8 (SL)
> 2.4.21-47.0.1.ELsmp #1 SMP Thu Oct 19 10:38:33 CDT 2006 x86_64 x86_64
> x86_64 GNU/Linux
>
> Gluster Client:
> No messages in the log files (either gluster's log file or
> /var/log/messages)
>
> glusterfs -V
> glusterfs 1.3.1
>
> fuse version is fuse-2.7.0-glfs3
>
> spec file:
> volume client
>   type protocol/client
>   option transport-type tcp/client     # for TCP/IP transport
>   option remote-host 135.1.29.152       # IP address of the remote brick
>   option remote-subvolume brick        # name of the remote volume
> end-volume
> #### Add readahead feature
> volume readahead
>   type performance/read-ahead
>   option page-size 128KB     # unit in bytes
>   subvolumes client
> end-volume
> ### Add IO-Threads feature
> volume iothreads
>   type performance/io-threads
>   option thread-count 4         # deault is 1
>   option cache-size 64MB
>   subvolumes readahead
> end-volume
>
> machine's OS = CentOS release 5 (Final)
> 2.6.18-2.6.18-8.1.8.el5.028stab039.1.prj4 #1 SMP Mon Aug 13 16:31:27 CDT
> 2007 i686 athlon i386 GNU/Linux
>
> thanks,
>
> Paul Jochum
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>



-- 
-------------------------------------------------------------
You think your computer is overloaded ?
Check if nobody's asked for tea !
"The Hitchhiker's Guide to the Galaxy" D.Adams