[Gluster-users] Protocol stacking: gluster over NFS

[harry mangalam]

Hi All,

We have been experimenting with 'protocol stacking' - that is, running gluster 
over NFS.

What I mean:
- mounting a gluster fs via the native client,
- then NFS-exporting the gluster fs to the client itself
- then mounting that gluster fs via NFS3 to take advantage of the client-side 
caching.

We've tried it on a limited basis (single client) and not only does it work, 
but it works surprisingly well, gaining about 2-3X the write performance 
relative to the native gluster mount on uncompressed data, using small writes.  
Using compressed data (piping thru gzip, for example) is more variable - if 
the data is highly compressible, it tends to increase performance; if less 
compressible, it tends to decrease performance.  As I noted previously 
<http://goo.gl/7G7k3>, piping small writes thru gzip /can/ tremendously 
increase performance on a gluster fs in some bioinformatics applications.

A graph of the performance on various file sizes (created by a trivial program 
that does zillions of tiny writes - a sore point in the gluster performance 
spectrum) is shown here:

http://moo.nac.uci.edu/~hjm/fs_perf_gl-tmp-glnfs.png

The graphs show the time to complete and sync on a set of writes from 10MB to 
30GB on 3 fs's:

- /tmp on the client's system disk (a single 10K USCSI)
- /gl, a 4 server, 4 brick gluster (distributed-only) fs 
- /gl/nfs, the same gluster fs, loopback-mounted via NFS3 on the client

The results show that using a gluster fs loopback-mounted to itself increased 
performance by 2-3X, increasing as the file size increased to 30GB.

The client (64GB RAM) was otherwise idle when I did these tests.  

In addition (data not shown), I also tested how compression (piping the output 
thru gzip) affected the total time-to-complete.  In one case, due to the 
identical string being written, gzip managed about 1000X compression, so the 
eventual file size sent to the disk was almost inconsequential.  Nevertheless, 
the extra time for the compression was more than made up for by the reduced 
data and adding gzip decreased the time-to-complete significantly.  In other 
testing with less compressible data (shown above), the compression time 
overwhelmed the write time and all the fs had essentially identical times per 
file size. 

In all cases, the writes were followed by a 'sync' to flush the cache.  

It seems that the loopback NFS mounting of the gluster fs is a fairly obvious 
win (overall, about 2-3x times the write speed) in terms of taking avantage of 
gluster's fs scaling and namespace with NFS3's client-side caching, but I'd 
like to hear from other gluster users as to possible downsides of this 
approach.

hjm
-- 
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
--