[Gluster-users] Gluster on ZFS performance concerns

[Tony Maro]

Off and on for the past 8 months I've been working on setting up a rather
sizable Gluster configuration on top of ZFS.  I wanted to report back my
experiences.

First, I'm running Ubuntu 12.04 with zfsonlinux.org packages.  Gluster is
installed from the semiosis PPA.

My test config is as follows:

Two brick Gluster in mirroring (replica 2)
Each brick has 8 TB of hard drives configured with ZFS using RAID-Z.
Each brick has a 256 GB SSD drive to use as cache for ZFS.
A third server for geo-replication slave also with RAID-Z storage.
Private Gigabit network segment for the gluster servers (and the
application servers that talk to them)
Geo-rep will be placed three states away in a colo center, but is currently
on the same private segment.

Data:
About 6 TB of test data, the final deploy will be much larger.  Most files
are around 40MB in size.  Files are stored in directories going 4 deep, no
more than 255 files in a single directory.

All data is already compressed and encrypted with AES-256 prior to storing
on the filesystem, so the comrpession feature of ZFS isn't useful to me.

The default configs for ZFS caused this to be a miserable failure.
 Configuring Gluster geo-replication over SSH was also a difficult task,
because the documentation is wrong on many counts when compared to the
version in the PPA.

With a default config, directory listings (remember, less than 255 files
and folders in each directory) would take about 15 seconds to complete over
the Gluster share.

By simply tweaking the following ZFS configs:
zfs set atime=off [volname]
zfs set xattr=sa [volname]
zfs set exec=off [volname]
zfs set sync=disabled [volname]

I was able to get near instantaneous directory listings and dramatically
improved read performance, although I haven't bothered getting hard numbers
on that.

I just wanted to pass this along.  Here's my blog posts about ZFS and
Gluster, which pretty much says the same thing I just did but offers more
about installing ZFS:

http://www.ossramblings.com/zfs-filesystem-on-linux
http://www.ossramblings.com/gluster-geo-replication-with-zfs

-Tony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131227/cfa75c19/attachment.html>