[Gluster-users] GlusterFS on ZFS

[Pascal Suter]

Hi Cody

i'm still new to Gluster myself, so take my input with the necessary 
skepticism:

if you care about performance (and it looks like you do), use zfs mirror 
pairs and not raidz volumes. in my experience (outside of gluster), 
raidz pools perform significantly worse than a hardware raid5 or 6. if 
you combine a mirror on zfs with a 3x replication on gluster, you need 
6x the amount of raw disk space to get your desired redundancy.. you 
could do with 3x the amount of diskspace, if you left the zfs mirror 
away and accept the rebuild of a lost disk over the network or you could 
end up somewhere beween 3x and 6x if you used hardware raid6 instead of 
zfs on the bricks. When using hardware raid6 make sure you align your 
lvm volumes properly, it makes a huge difference in performance. Okay, 
deduplication might give you some of it back, but benchmark the zfs 
deduplication process first before deciding on it. in theory it could 
add to your write perofrmance, but i'm not sure if that's going to 
happen in reality.

snapshotting might be tricky.. afaik gluster natively supports 
snapshotting with thin provisioned lvm volumes only. this lets you 
create snapshots with the "gluster" cli tool. gluster will then handle 
consistency across all your bricks so that each snapshot (as a whole, 
across all bricks) is consistent in itself. this includes some 
challenges about handling open file sessions etc. I'm not familiar with 
what gluster actually does but by reading the documentation and some 
discussion about snapshots it seems that there is more to it than simply 
automate a couple of lvcreate statements. so i would expect some 
challenges when doing it yourself on zfs rather than letting gluster 
handle it. Restoring a single file from a snapshot also seems alot 
easier if you go with the lvm thin setup.. you can then mount a snapshot 
(of your entire gluster volume, not just of a brick) and simply copy the 
file.. while with zfs it seems you need to find out which bricks your 
file resided on, then copy the necessary raw data to your live bricks 
which is something i would not feel comfortable doing and it is a lot 
more work and prone to error.

also, if things go wrong (for example when dealing with the snapshots), 
there are probably not so many people around to help you.

again, i am no expert, that's just what i'd be concerned about with the 
little knowledge i have at the moment :)

cheers

Pascal

On 17.04.19 00:09, Cody Hill wrote:
> Hey folks.
>
> I’m looking to deploy GlusterFS to host some VMs. I’ve done a lot of reading and would like to implement Deduplication and Compression in this setup. My thought would be to run ZFS to handle the Compression and Deduplication.
>
> ZFS would give me the following benefits:
> 1. If a single disk fails rebuilds happen locally instead of over the network
> 2. Zil & L2Arc should add a slight performance increase
> 3. Deduplication and Compression are inline and have pretty good performance with modern hardware (Intel Skylake)
> 4. Automated Snapshotting
>
> I can then layer GlusterFS on top to handle distribution to allow 3x Replicas of my storage.
> My question is… Why aren’t more people doing this? Is this a horrible idea for some reason that I’m missing? I’d be very interested to hear your thoughts.
>
> Additional thoughts:
> I’d like to use Ganesha pNFS to connect to this storage. (Any issues here?)
> I think I’d need KeepAliveD across these 3x nodes to store in the FSTAB (Is this correct?)
> I’m also thinking about creating a “Gluster Tier” of 512GB of Intel Optane DIMM to really smooth out write latencies… Any issues here?
>
> Thank you,
> Cody Hill
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users