<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 23, 2019 at 11:38 PM Cody Hill <<a href="mailto:cody@platform9.com">cody@platform9.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><br>Thanks for the info Karli,<div><br></div><div>I wasn’t aware ZFS Dedup was such a dog. I guess I’ll leave that off. My data get’s 3.5:1 savings on compression alone. I was aware of stripped sets. I will be doing 6x Striped sets across 12x disks. </div><div><br></div><div>On top of this design I’m going to try and test Intel Optane DIMM (512GB) as a “Tier” for GlusterFS to try and get further write acceleration. And issues with GlusterFS “Tier” functionality that anyone is aware of?</div><div><br></div><div></div></div></blockquote><div><br></div><div>Hi Cody, I wanted to be honest about GlusterFS 'Tier' functionality. While it is functional and works, we had not seen the actual benefit we expected with the feature, and noticed it is better to use the tiering on each host machine (ie, on bricks) and use those bricks as glusterfs bricks. (like dmcache). </div><div><br></div><div>Also note that from glusterfs-6.x releases, Tier feature is deprecated.</div><div><br></div><div>-Amar</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div>Thank you,</div><div>Cody Hill <br><div><br><blockquote type="cite"><div>On Apr 18, 2019, at 2:32 AM, Karli Sjöberg <<a href="mailto:karli@inparadise.se" target="_blank">karli@inparadise.se</a>> wrote:</div><br class="gmail-m_-6291573082511852447Apple-interchange-newline"><div><div dir="auto"><div><br><div class="gmail_extra"><br><div class="gmail_quote">Den 17 apr. 2019 16:30 skrev Cody Hill <<a href="mailto:cody@platform9.com" target="_blank">cody@platform9.com</a>>:<br type="attribution"><blockquote class="gmail-m_-6291573082511852447quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Hey folks.<br><br>I’m looking to deploy GlusterFS to host some VMs. I’ve done a lot of reading and would like to implement Deduplication and Compression in this setup. My thought would be to run ZFS to handle the Compression and Deduplication.<br></div></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">You _really_ don't want ZFS doing dedup for any reason.</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail-m_-6291573082511852447quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br>ZFS would give me the following benefits:<br>1. If a single disk fails rebuilds happen locally instead of over the network<br>2. Zil & L2Arc should add a slight performance increase<br></div></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">Adding two really good NVME SSD's as a mirrored SLOG vdev does a huge deal for synchronous write performance, turning every random write into large streams that the spinning drives handle better.</div><div dir="auto"><br></div><div dir="auto">Don't know how picky Gluster is about synchronicity though, most "performance" tweaking suggests setting stuff to async, which I wouldn't recommend, but it's a huge boost for throughput obviously; not having to wait for stuff to actually get written, but it's dangerous.</div><div dir="auto"><br></div><div dir="auto">With mirrored NVME SLOG's, you could probably get that throughput without going asynchronous, which saves you from potential data corruption in a sudden power loss.</div><div dir="auto"><br></div><div dir="auto">L2ARC on the other hand does a bit for read latency, but for a general purpose file server- in practice- not a huge difference, the working set is just too large. Also keep in mind that L2ARC isn't "free". You need more RAM to know where you've cached stuff...</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail-m_-6291573082511852447quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>3. Deduplication and Compression are inline and have pretty good performance with modern hardware (Intel Skylake)<br></div></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">ZFS deduplication has terrible performance. Watch your throughput automatically drop from hundreds or thousands of MB/s down to, like 5. It's a feature;)</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail-m_-6291573082511852447quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>4. Automated Snapshotting<br><br>I can then layer GlusterFS on top to handle distribution to allow 3x Replicas of my storage.<br>My question is… Why aren’t more people doing this? Is this a horrible idea for some reason that I’m missing?</div></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">While it could save a lot of space in some hypothetical instance, the drawbacks can never motivate it. E.g. if you want one node to suddenly die and never recover because of RAM exhaustion, go with ZFS dedup ;)</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail-m_-6291573082511852447quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div> I’d be very interested to hear your thoughts.<br></div></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">Avoid ZFS dedup at all costs. LZ4 compression on the hand is awesome, definitely use that! It's basically a free performance enhancer the also saves space :)</div><div dir="auto"><br></div><div dir="auto">As another person has said, the best performance layout is RAID10- striped mirrors. I understand you'd want to get as much volume as possible with RAID-Z/RAID(5|6) since gluster also replicates/distributes, but it has a huge impact on IOPS. If performance is the main concern, do striped mirrors with replica 3 in Gluster. My advice is to test thoroughly with different pool layouts to see what gives acceptable performance against your volume requirements.</div><div dir="auto"><br></div><div dir="auto">/K</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail-m_-6291573082511852447quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br>Additional thoughts:<br>I’d like to use Ganesha pNFS to connect to this storage. (Any issues here?)<br>I think I’d need KeepAliveD across these 3x nodes to store in the FSTAB (Is this correct?)<br>I’m also thinking about creating a “Gluster Tier” of 512GB of Intel Optane DIMM to really smooth out write latencies… Any issues here?<br><br>Thank you,<br>Cody Hill<br><br><br><br><br><br><br><br><br><br><br><br>_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></div></blockquote></div><br></div></div></div></div></blockquote></div><br></div></div>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div></div>