<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Gagan:</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Throwing my $0.02 in --</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
It depends on the system environment of how you are planning on deploying Gluster (and/or Ceph).</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I have Ceph running on my three node HA Proxmox cluster using three OASLOA Mini PCs that only has the Intel N95 Processor (4-core/4-thread) with 16 GB of RAM and a cheap Microcenter store brand 512 GB NVMe M.2 2230 SSD and my Ceph cluster has been running without
any issues.</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
As someone else mentioned, to state or claim that Ceph is "hardware demanding" isn't wholly accurate.</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
As for management, you can install the <span style="font-family: "Courier New", monospace;">
ceph-mgr-dashboard</span> package (there is a video that apalrd's adventures put together on YouTube which goes over the installation process for this package, if you're running Debian and/or Proxmox (which runs on top of Debian anyways).)</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
From there, you can use said Ceph manager dashboard to do everything else, so that you don't have to deploy Ceph via the CLI.<br>
<br>
I was able to create my erasure coded CRUSH rules using the dashboard, and then create my RBD pool and also my CephFS pool as well. (The metadata needs a replicate CRUSH rule, but the data itself can use erasure coded CRUSH rule.)</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
If your environment is such that you can do this, then Ceph might be a better option for you.<br>
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
If you look at the benchmarks that tech YouTuber ElectronicsWizardry ran, ZFS is actually
<i>not</i> all that performant. But what ZFS <i>is</i> good for are some of the other features like snapshots, replications, and it's copy-on-write schema for modifying files (which again, based on the testing that ElectronicsWizardry ran,
<i>does</i> indeed create a write amplification effect as a result of the copy-on-write architecture)<br>
<br>
Conversely, if you're looking for reliability, the more nodes that you have in the Ceph cluster, the more reliable and resilient to failures the Ceph backend will be.</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks.</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Sincerely,</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Ewen</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Gluster-users <gluster-users-bounces@gluster.org> on behalf of gagan tiwari <gagan.tiwari@mathisys-india.com><br>
<b>Sent:</b> April 17, 2025 2:14 AM<br>
<b>To:</b> Alexander Schreiber <als@thangorodrim.ch><br>
<b>Cc:</b> gluster-users@gluster.org <gluster-users@gluster.org><br>
<b>Subject:</b> Re: [Gluster-users] Gluster with ZFS</font>
<div> </div>
</div>
<div>
<div dir="ltr">HI Alexander,
<div> Thanks for the update. Initially, I also thought of deploying Ceph but ceph is quite difficult to set-up and manage. Moreover, it's also hardware demanding. I think it's most suitable for a very large set-up with hundreds
of clients. </div>
<div><br>
</div>
<div>What do you think of MooseFS ? Have you or anyone else tried MooseFS. If yes, how was its performance?</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Gagan</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div class="x_gmail_quote x_gmail_quote_container">
<div dir="ltr" class="x_gmail_attr">On Thu, Apr 17, 2025 at 1:45 PM Alexander Schreiber <<a href="mailto:als@thangorodrim.ch">als@thangorodrim.ch</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
On Thu, Apr 17, 2025 at 09:40:08AM +0530, gagan tiwari wrote:<br>
> Hi Guys,<br>
> We have been using OpenZFS in our HPC environment for<br>
> quite some time. And OpenZFS was going fine.<br>
> <br>
> But we are now running into scalability issues since OpenZFS can't be<br>
> scaled out.<br>
<br>
Since ZFS is a local FS, you are essentially limited to how much storage<br>
you can stick into one machine, yes.<br>
<br>
> So, I am planning to use Gluster on top of OpenZFS.<br>
<br>
I don't think that is giving you the kind of long term scalability<br>
you might expect.<br>
<br>
> So, I wanted to know if anyone has tried it. if yes, how it was and any<br>
> deployment guide for it.<br>
<br>
I'm running GlusterFS in a small cluster for backup storage.<br>
<br>
> We have an HPC environment . Data security and extremely fast read<br>
> performance is very important for us.<br>
> <br>
> So, please advise.<br>
<br>
For that use case I would actually recommend Ceph over GlusterFS, since<br>
that can be pretty easily scaled out to very large setups, e.g. CERN is<br>
using multiple Ceph clusters sized at several PB and their use cases<br>
usually include very fast I/O.<br>
<br>
Another concern is that Ceph is being quite actively developed whereas<br>
GlusterFS development seems to have slowed down to ... not much, these days.<br>
<br>
Kind regards,<br>
Alex.<br>
-- <br>
"Opportunity is missed by most people because it is dressed in overalls and<br>
looks like work." -- Thomas A. Edison<br>
</blockquote>
</div>
</div>
</body>
</html>