[Gluster-users] Protocol stacking: gluster over NFS

[harry mangalam]

Hi Venky - thank for the link to this translator.  I'll take a look at it, but 
right now, we don't have too much trouble with reads - it's the 'zillions of 
tiny writes' problem that's hosing us and the NFS solution gives us a bit more 
headroom.

We'll be moving this out to part of our cluster today (unless someone can 
convince me otherwise) and we'll see if it shows real-world improvements.

hjm

On Friday, September 14, 2012 11:33:34 AM Venky Shankar wrote:
> Hi Harry,
> 
> There is a compression translator in Gerrit which you might be
> interested in: http://review.gluster.org/#change,3251
> 
> It compresses data (using zlib library) before it is sent out to the
> network from the server and on the other side (client; FUSE mount) it
> decompresses it. Also, note that it only does it for data transferred as
> part of the read fop and the volfiles needs to be hand edited (CLI
> support is still pending)
> 
> I've not performed any performance run till now but I plan to do so soon.
> 
> -Venky
> 
> On Friday 14 September 2012 10:25 AM, harry mangalam wrote:
> > Hi All,
> > 
> > We have been experimenting with 'protocol stacking' - that is, running
> > gluster over NFS.
> > 
> > What I mean:
> > - mounting a gluster fs via the native client,
> > - then NFS-exporting the gluster fs to the client itself
> > - then mounting that gluster fs via NFS3 to take advantage of the
> > client-side caching.
> > 
> > We've tried it on a limited basis (single client) and not only does it
> > work, but it works surprisingly well, gaining about 2-3X the write
> > performance relative to the native gluster mount on uncompressed data,
> > using small writes. Using compressed data (piping thru gzip, for example)
> > is more variable - if the data is highly compressible, it tends to
> > increase performance; if less compressible, it tends to decrease
> > performance.  As I noted previously <http://goo.gl/7G7k3>, piping small
> > writes thru gzip /can/ tremendously increase performance on a gluster fs
> > in some bioinformatics applications.
> > 
> > A graph of the performance on various file sizes (created by a trivial
> > program that does zillions of tiny writes - a sore point in the gluster
> > performance spectrum) is shown here:
> > 
> > http://moo.nac.uci.edu/~hjm/fs_perf_gl-tmp-glnfs.png
> > 
> > The graphs show the time to complete and sync on a set of writes from 10MB
> > to 30GB on 3 fs's:
> > 
> > - /tmp on the client's system disk (a single 10K USCSI)
> > - /gl, a 4 server, 4 brick gluster (distributed-only) fs
> > - /gl/nfs, the same gluster fs, loopback-mounted via NFS3 on the client
> > 
> > The results show that using a gluster fs loopback-mounted to itself
> > increased performance by 2-3X, increasing as the file size increased to
> > 30GB.
> > 
> > The client (64GB RAM) was otherwise idle when I did these tests.
> > 
> > In addition (data not shown), I also tested how compression (piping the
> > output thru gzip) affected the total time-to-complete.  In one case, due
> > to the identical string being written, gzip managed about 1000X
> > compression, so the eventual file size sent to the disk was almost
> > inconsequential.  Nevertheless, the extra time for the compression was
> > more than made up for by the reduced data and adding gzip decreased the
> > time-to-complete significantly.  In other testing with less compressible
> > data (shown above), the compression time overwhelmed the write time and
> > all the fs had essentially identical times per file size.
> > 
> > In all cases, the writes were followed by a 'sync' to flush the cache.
> > 
> > It seems that the loopback NFS mounting of the gluster fs is a fairly
> > obvious win (overall, about 2-3x times the write speed) in terms of
> > taking avantage of gluster's fs scaling and namespace with NFS3's
> > client-side caching, but I'd like to hear from other gluster users as to
> > possible downsides of this approach.
> > 
> > hjm
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
-- 
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
--
What does it say about a society that would rather send its 
children to kill and die for oil than to get on a bike?