[Gluster-users] Gluster on ZFS performance concerns

[bernhard glomm]

Thanks a lot for sharing your results!
I'm to looking into a similar test scenario.

According to
http://www.gluster.org/community/documentation/index.php/GlusterOnZFS
one should apply this patch
https://github.com/zfsonlinux/zfs/issues/1648
Did you do that? Or don't you use symlinks on your glustermounts?
From your blogs I understand you didn't, did you experienced ny issues?
I like to use this scenario as a backend fro VM images.
The glusterserver mounts the glustervolumes 
and are also the hosts for the VM, sort of self describing/self providing approach.
I created an individual filesystems and along with it a separate glustervolume 
for each VM. This way I can snapshot the volumes/VMs individually.
I doubt it would make any performance difference 
if I would put all images in one Volume/zfs-Filesystem. ?
I ran several VMs in parallel and let them all perform bonnie++ tests
The whole setup seems quite stable so far
including life migrating of the VMs and reboot of the hosts during the tests.

TIA
Bernhard


	 Bernhard Glomm
IT Administration

Phone:	 +49 (30) 86880 134
Fax:	 +49 (30) 86880 100
Skype:	 bernhard.glomm.ecologic
               
Ecologic Institut gemeinnützige GmbH | Pfalzburger Str. 43/44 | 10717 Berlin | Germany
GF: R. Andreas Kraemer | AG: Charlottenburg HRB 57947 | USt/VAT-IdNr.: DE811963464
Ecologic™ is a Trade Mark (TM) of Ecologic Institut gemeinnützige GmbH

On Dec 27, 2013, at 5:52 PM, Tony Maro <tonym at evrichart.com> wrote:

> Off and on for the past 8 months I've been working on setting up a rather sizable Gluster configuration on top of ZFS.  I wanted to report back my experiences.
> 
> First, I'm running Ubuntu 12.04 with zfsonlinux.org packages.  Gluster is installed from the semiosis PPA.
> 
> My test config is as follows:
> 
> Two brick Gluster in mirroring (replica 2)
> Each brick has 8 TB of hard drives configured with ZFS using RAID-Z.
> Each brick has a 256 GB SSD drive to use as cache for ZFS.
> A third server for geo-replication slave also with RAID-Z storage.
> Private Gigabit network segment for the gluster servers (and the application servers that talk to them)
> Geo-rep will be placed three states away in a colo center, but is currently on the same private segment.
> 
> Data:
> About 6 TB of test data, the final deploy will be much larger.  Most files are around 40MB in size.  Files are stored in directories going 4 deep, no more than 255 files in a single directory.
> 
> All data is already compressed and encrypted with AES-256 prior to storing on the filesystem, so the comrpession feature of ZFS isn't useful to me.
> 
> The default configs for ZFS caused this to be a miserable failure.  Configuring Gluster geo-replication over SSH was also a difficult task, because the documentation is wrong on many counts when compared to the version in the PPA.
> 
> With a default config, directory listings (remember, less than 255 files and folders in each directory) would take about 15 seconds to complete over the Gluster share.
> 
> By simply tweaking the following ZFS configs:
> zfs set atime=off [volname]
> zfs set xattr=sa [volname]
> zfs set exec=off [volname]
> zfs set sync=disabled [volname]
> 
> I was able to get near instantaneous directory listings and dramatically improved read performance, although I haven't bothered getting hard numbers on that.
> 
> I just wanted to pass this along.  Here's my blog posts about ZFS and Gluster, which pretty much says the same thing I just did but offers more about installing ZFS:
> 
> http://www.ossramblings.com/zfs-filesystem-on-linux
> http://www.ossramblings.com/gluster-geo-replication-with-zfs
> 
> -Tony
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131230/a02c792e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131230/a02c792e/attachment.sig>