[Gluster-users] Gluster with ZFS

[Ewen Chan]

Gagan:

I actually tried what Alexander mentioned below, as a separate experiment.

I have two AMD Ryzen 9 5950X compute nodes and one AMD Ryzen 9 7950X compute node, and each node has 128 GB of RAM and also a Mellanox ConnectX-4 100 Gbps Infiniband network card, and I was using I think one Intel 670p 1 TB NVMe SSD and two Silicon Power US70 1 TB NVMe SSD.

>From the Ceph perspective, giving it quote "this much hardware", didn't significantly improve the performance.

Of course, 100 Gbps Infiniband is faster than 1 GbE, but Ceph barely even noticed.

Where it played a more significant role was when I was doing live migrations of VMs and LXCs where it was able to use the 100 Gbps IB network speeds more so for migrating RAM states between nodes than the actual virtual disk drive of the VM. Ceph won't help with the RAM state. And since the VM disk was already sitting on the shared, distributed, Ceph storage, I wasn't moving the disk over 100 Gbps IB, just the RAM state.

So depending on how you've set it up, throwing more hardware at Ceph, might not improve performance much.

Thanks.

Sincerely,
Ewen
________________________________
From: Gluster-users <gluster-users-bounces at gluster.org> on behalf of Alexander Schreiber <als at thangorodrim.ch>
Sent: April 17, 2025 7:54 AM
To: gagan tiwari <gagan.tiwari at mathisys-india.com>
Cc: gluster-users at gluster.org <gluster-users at gluster.org>
Subject: Re: [Gluster-users] Gluster with ZFS

On Thu, Apr 17, 2025 at 02:44:28PM +0530, gagan tiwari wrote:
> HI Alexander,
>                               Thanks for the update. Initially, I  also
> thought of deploying Ceph but ceph is quite difficult to set-up and manage.
> Moreover, it's also  hardware demanding. I think it's most suitable for a
> very large set-up  with hundreds of clients.

I strongly disagree. I run a small (3 nodes) Ceph cluster in my homelab
and following the official docs it's pretty easy to set up. The hardware
demands mostly depend on what performance one needs - the more performance
(e.g. NVMe storage and 100 GBit networking) one wants, the more powerful
hardware one has to provide, as usual. My nodes are Intel D-1521 with
64G of (ECC, of course) RAM and Connect-X 4 cards running at 10 GBit
and storage on HDDs which provide reasonable performance for my needs - not
a HPC setup, of course.

> What do you think of MooseFS ?  Have you or anyone else tried MooseFS. If
> yes, how was its performance?

Last time I looked, MooseFS needs a commercial license for the full feature
set (e.g. highly available metadata (_that_ is not negotiable!), erasure
oding, Windows clients) which killed it for my non-commercial use case.

Kind regards,
            Alex.
--
"Opportunity is missed by most people because it is dressed in overalls and
 looks like work."                                      -- Thomas A. Edison
________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users at gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20250417/8c34a86b/attachment.html>