<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head><meta content="text/html;charset=UTF-8" http-equiv="Content-Type"></head><body ><div style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt;"><div>Only Replica 2 or Distributed Gluster volumes can be created with two servers. High chance of split brain with Replica 2 compared to Replica 3 volume.<br></div><div><br></div><div>For NFS Ganesha, no issue exporting the volume even if only one server is available. Run NFS Ganesha servers in Gluster server nodes and NFS clients from the network can connect to any NFS Ganesha server.<br></div><div><br></div><div>You can use Haproxy + Keepalived (or any other load balancer) if high availability required for the NFS Ganesha connections (Ex: If a server node goes down, then nfs client can connect to other NFS ganesha server node).<br></div><div><br></div><div>--</div><div id="Zm-_Id_-Sgn" data-sigid="3848334000000010003" data-zbluepencil-ignore="true"><div>Aravinda<br></div><div>Kadalu Technologies</div></div><div><br></div><div class="zmail_extra_hr" style="border-top: 1px solid rgb(204, 204, 204); height: 0px; margin-top: 10px; margin-bottom: 10px; line-height: 0px;"><br></div><div class="zmail_extra" data-zbluepencil-ignore="true"><div><br></div><div id="Zm-_Id_-Sgn1">---- On Wed, 13 Dec 2023 01:42:11 +0530 <b>Gilberto Ferreira <gilberto.nunes32@gmail.com></b> wrote ---<br></div><div><br></div><blockquote id="blockquote_zmail" style="margin: 0px;"><div><div dir="ltr">Ah that's nice.<div>Somebody knows this can be achieved with two servers?<br><br><div><div dir="ltr" class="x_-238876503gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>---<br></div><div><div><div>Gilberto Nunes Ferreira<br></div></div><div><span class="size" style="font-size:12.8px">(47) 99676-7530 - Whatsapp / Telegram</span><br></div><div><p style="margin: 0px;"><span class="size" style="font-size: 12.8px; margin: 0px;"></span><br></p><p style="margin: 0px;"><span class="size" style="font-size: 12.8px; margin: 0px;"><br></span></p><p style="margin: 0px;"><span class="size" style="font-size: 12.8px; margin: 0px;"><br></span></p></div></div><div><br></div></div></div></div></div></div></div></div><br></div></div><br><div class="x_-238876503gmail_quote"><div dir="ltr" class="x_-238876503gmail_attr">Em ter., 12 de dez. de 2023 às 17:08, Danny <<a href="mailto:dbray925%2Bgluster@gmail.com" target="_blank">dbray925+gluster@gmail.com</a>> escreveu:<br></div><div id="zmail_block"><br></div></div>________<br> <br> <br> <br>Community Meeting Calendar: <br> <br>Schedule - <br>Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC <br>Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a> <br>Gluster-users mailing list <br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <br><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a> <br></div><blockquote class="x_-238876503gmail_quote" style="margin : 0px 0px 0px 0.8ex; border-left : 1px solid rgb(204,204,204); padding-left : 1ex;"><div dir="ltr"><div>Wow, HUGE improvement with NFS-Ganesha!<br></div><div><br></div><div><br></div><div><div><code>sudo</code> <code>dnf -y </code><code>install</code> <code>glusterfs-ganesha</code><br></div><div><div title="Hint: double-click to select code"><div><code>sudo</code> <code>vim </code><code>/etc/ganesha/ganesha</code><code>.conf</code><br></div></div></div><div><br></div><div><div title="Hint: double-click to select code"><div><code>NFS_CORE_PARAM {</code><br></div><div><code> </code><code>mount_path_pseudo = true;</code><br></div><div><code> </code><code>Protocols = 3,4;</code><br></div><div><code>}</code><br></div><div><code>EXPORT_DEFAULTS {</code><br></div><div><code> </code><code>Access_Type = RW;</code><br></div><div><code>}</code><br></div><div> <br></div><div><code>LOG {</code><br></div><div><code> </code><code>Default_Log_Level = WARN;</code><br></div><div><code>}</code><br></div><div> <br></div><div><code>EXPORT{</code><br></div><div><code> </code><code>Export_Id = 1 ; # Export ID unique to each export</code><br></div><div><code> </code><code>Path = "/data"; # Path of the volume to be exported</code><br></div><div> <br></div><div><code> </code><code>FSAL {</code><br></div><div><code> </code><code>name = GLUSTER;</code><br></div><div><code> </code><code>hostname = "localhost"; # IP of one of the nodes in the trusted pool</code><br></div><div><code> </code><code>volume = "data"; # Volume name. Eg: "test_volume"</code><br></div><div><code> </code><code>}</code><br></div><div> <br></div><div><code> </code><code>Access_type = RW; # Access permissions</code><br></div><div><code> </code><code>Squash = No_root_squash; # To enable/disable root squashing</code><br></div><div><code> </code><code>Disable_ACL = TRUE; # To enable/disable ACL</code><br></div><div><code> </code><code>Pseudo = "/data"; # NFSv4 pseudo path for this export</code><br></div><div><code> </code><code>Protocols = "3","4" ; # NFS protocols supported</code><br></div><div><code> </code><code>Transports = "UDP","TCP" ; # Transport protocols supported</code><br></div><div><code> </code><code>SecType = "sys"; # Security flavors supported</code><br></div><div><code>}</code><br></div></div></div><div><br></div><div><br></div><div><div title="Hint: double-click to select code"><div><code>sudo</code> <code>systemctl </code><code>enable</code> <code>--now nfs-ganesha</code><br></div></div></div><div><code>s</code><code>udo</code> <code>vim </code><code>/etc/fstab</code> <br></div><div><br></div><div><div><code>localhost:/data /data nfs defaults,_netdev 0 0</code><br></div></div><div><br></div><div><br></div><div><div title="Hint: double-click to select code"><div><code>sudo</code> <code>systemctl daemon-reload</code><br></div><div><code>sudo</code> <code>mount</code> <code>-a</code><br></div></div></div><div><code><br></code></div><div><div><code>fio --name=</code><code>test</code> <code>--filename=</code><code>/data/wow</code> <code>--size=1G --readwrite=write</code><br></div></div><div><br></div><div>Run status group 0 (all jobs):<br> WRITE: bw=2246MiB/s (2355MB/s), 2246MiB/s-2246MiB/s (2355MB/s-2355MB/s), io=1024MiB (1074MB), run=456-456msec</div><div><br></div><div>Yeah
2355MB/s is much better than the original 115MB/s<br></div><div><br></div><div>So in the end, I guess FUSE isn't the best choice. <br></div></div></div><br><div class="x_-238876503gmail_quote"><div dir="ltr" class="x_-238876503gmail_attr">On Tue, Dec 12, 2023 at 3:00 PM Gilberto Ferreira <<a href="mailto:gilberto.nunes32@gmail.com" target="_blank">gilberto.nunes32@gmail.com</a>> wrote:<br></div><blockquote class="x_-238876503gmail_quote" style="margin : 0px 0px 0px 0.8ex; border-left : 1px solid rgb(204,204,204); padding-left : 1ex;"><div dir="ltr">Fuse there some overhead.<div>Take a look at libgfapi:<br></div><div><a href="https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/libgfapi/" target="_blank">https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/libgfapi/</a><br></div><div><br></div><div>I know this doc somehow is out of date, but could be a hint<br></div><div><br></div><div><br><div><div dir="ltr" class="x_-238876503gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>---<br></div><div><div><div>Gilberto Nunes Ferreira<br></div></div><div><span class="size" style="font-size:12.8px">(47) 99676-7530 - Whatsapp / Telegram</span><br></div><div><p style="margin: 0px;"><span class="size" style="font-size: 12.8px; margin: 0px;"></span><br></p><p style="margin: 0px;"><span class="size" style="font-size: 12.8px; margin: 0px;"><br></span></p><p style="margin: 0px;"><span class="size" style="font-size: 12.8px; margin: 0px;"><br></span></p></div></div><div><br></div></div></div></div></div></div></div></div><br></div></div><br><div class="x_-238876503gmail_quote"><div dir="ltr" class="x_-238876503gmail_attr">Em ter., 12 de dez. de 2023 às 16:29, Danny <<a href="mailto:dbray925%2Bgluster@gmail.com" target="_blank">dbray925+gluster@gmail.com</a>> escreveu:<br></div><blockquote class="x_-238876503gmail_quote" style="margin : 0px 0px 0px 0.8ex; border-left : 1px solid rgb(204,204,204); padding-left : 1ex;"><div dir="ltr"><div>Nope, not a caching thing. I've tried multiple different types of fio tests, all produce the same results. Gbps when hitting the disks locally, slow MB\s when hitting the Gluster FUSE mount.<br></div><div><br></div><div>I've been reading up on glustr-ganesha, and will give that a try.<br></div></div><br><div class="x_-238876503gmail_quote"><div dir="ltr" class="x_-238876503gmail_attr">On Tue, Dec 12, 2023 at 1:58 PM Ramon Selga <<a href="mailto:ramon.selga@gmail.com" target="_blank">ramon.selga@gmail.com</a>> wrote:<br></div><blockquote class="x_-238876503gmail_quote" style="margin : 0px 0px 0px 0.8ex; border-left : 1px solid rgb(204,204,204); padding-left : 1ex;"><u></u><div><span class="font" style="font-family:"Futura Bk BT"">Dismiss my first question: you have SAS
12Gbps SSDs Sorry!</span><br> <br> <div>El 12/12/23 a les 19:52, Ramon Selga ha
escrit:<br></div><blockquote><span class="font" style="font-family:"Futura Bk BT"">May ask you which kind of disks you have
in this setup? rotational, ssd SAS/SATA, nvme?<br> <br> Is there a RAID controller with writeback caching?<br> <br> It seems to me your fio test on local brick has a unclear result
due to some caching.<br> <br> Try something like (you can consider to increase test file size
depending of your caching memory) :<br> <br> fio --size=16G --name=test --filename=/gluster/data/brick/wow
--bs=1M --nrfiles=1 --direct=1 --sync=0 --randrepeat=0
--rw=write --refill_buffers --end_fsync=1 --iodepth=200
--ioengine=libaio<br> <br> </span>Also remember a replica 3 arbiter 1 volume writes
synchronously to two data bricks, halving throughput of your
network backend.<br> <br> Try similar fio on gluster mount but I hardly see more than
300MB/s writing sequentially on only one fuse mount even with nvme
backend. On the other side, with 4 to 6 clients, you can easily
reach 1.5GB/s of aggregate throughput <br> <br> To start, I think is better to try with default parameters for
your replica volume.<br> <br> Best regards!<br> <br> Ramon<br> <br> <br> <div>El 12/12/23 a les 19:10, Danny ha
escrit:<br></div><blockquote><div dir="ltr">Sorry, I noticed that too after I posted, so I
instantly upgraded to 10. Issue remains. <br></div><br><div class="x_-238876503gmail_quote"><div dir="ltr" class="x_-238876503gmail_attr">On Tue, Dec 12, 2023 at
1:09 PM Gilberto Ferreira <<a href="mailto:gilberto.nunes32@gmail.com" target="_blank">gilberto.nunes32@gmail.com</a>>
wrote:<br></div><blockquote class="x_-238876503gmail_quote" style="margin : 0px 0px 0px 0.8ex; border-left : 1px solid rgb(204,204,204); padding-left : 1ex;"><div dir="ltr">I strongly suggest you update to version 10
or higher. <br> It's come with significant improvement
regarding performance.<br> <div><div dir="ltr" class="x_-238876503gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>---<br></div><div><div><div>Gilberto Nunes Ferreira<br></div></div><div><span class="size" style="font-size:12.8px">(47)
99676-7530 - Whatsapp / Telegram</span><br></div><div><p style="margin: 0px;"><span class="size" style="font-size: 12.8px; margin: 0px;"><br></span></p><p style="margin: 0px;"><span class="size" style="font-size: 12.8px; margin: 0px;"><br></span></p></div></div><div><br></div></div></div></div></div></div></div></div><br></div><br><div class="x_-238876503gmail_quote"><div dir="ltr" class="x_-238876503gmail_attr">Em ter., 12 de dez. de
2023 às 13:03, Danny <<a href="mailto:dbray925%2Bgluster@gmail.com" target="_blank">dbray925+gluster@gmail.com</a>>
escreveu:<br></div><blockquote class="x_-238876503gmail_quote" style="margin : 0px 0px 0px 0.8ex; border-left : 1px solid rgb(204,204,204); padding-left : 1ex;"><div dir="ltr">MTU is already 9000, and as you can see
from the IPERF results, I've got a nice, fast
connection between the nodes.<br></div><br><div class="x_-238876503gmail_quote"><div dir="ltr" class="x_-238876503gmail_attr">On Tue, Dec 12, 2023
at 9:49 AM Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" target="_blank">hunter86_bg@yahoo.com</a>>
wrote:<br></div><blockquote class="x_-238876503gmail_quote" style="margin : 0px 0px 0px 0.8ex; border-left : 1px solid rgb(204,204,204); padding-left : 1ex;"><div>Hi, <div><br></div><div>Let’s try the simple things:<br></div><div><br></div><div>Check if you can use MTU9000 and if it’s
possible, set it on the Bond Slaves and the bond
devices:<br></div><div><span> ping GLUSTER_PEER </span><span>-c 10
-M do -s 8972</span><br></div><div><br>Then try to follow up the recommendations from <a href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance" target="_blank">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance</a> <div><br></div><div><br></div>Best Regards,</div><div>Strahil Nikolov <br> <br> <p style="padding-top: 15px; margin-top: 0px;"><span class="colour" style="color:rgb(113, 95, 250)"><span class="size" style="font-size: 15px; padding-top: 15px; margin-top: 0px;">On
Monday, December 11, 2023, 3:32 PM, Danny <<a href="mailto:dbray925%2Bgluster@gmail.com" target="_blank">dbray925+gluster@gmail.com</a>>
wrote:</span></span><br></p><blockquote><div id="x_-238876503m_-5986430612906196748m_5440043245792196623m_3596866081645582176m_-4149661103284099357m_-3283052153129726045m_98609290250261231m_810164770777586547yiv1869372156"><div dir="ltr"><div>Hello list, I'm hoping someone can
let me know what setting I missed.<br></div><div><br></div><div>Hardware:<br></div><div>Dell R650 servers, Dual 24 Core Xeon
2.8 GHz, 1 TB RAM<br></div><div>8x SSD s <span>Negotiated Speed</span> 12 Gbps<br></div><div>PERC H755 Controller - RAID 6 <br></div><div>Created virtual "data" disk from the
above 8 SSD drives, for a ~20 TB
/dev/sdb<br></div><div><br></div><div>OS:<br></div><div>CentOS Stream<br></div><div>kernel-4.18.0-526.el8.x86_64<br></div><div>glusterfs-7.9-1.el8.x86_64<br></div><div><br></div><div>IPERF Test between nodes:<br> [ ID] Interval Transfer
Bitrate Retr<br> [ 5] 0.00-10.00 sec 11.5 GBytes
9.90 Gbits/sec 0 sender<br> [ 5] 0.00-10.04 sec 11.5 GBytes
9.86 Gbits/sec
receiver<br></div><div><br></div><div>All good there. ~10 Gbps, as
expected.<br></div><div><br></div><div>LVM Install:<br></div><div>export DISK="/dev/sdb"<br> sudo parted --script $DISK "mklabel gpt"<br> sudo parted --script $DISK "mkpart
primary 0% 100%"<br> sudo parted --script $DISK "set 1 lvm
on"</div><div>sudo pvcreate --dataalignment 128K
/dev/sdb1<br> sudo vgcreate --physicalextentsize 128K
gfs_vg /dev/sdb1<br> sudo lvcreate -L 16G -n gfs_pool_meta
gfs_vg<br> sudo lvcreate -l 95%FREE -n gfs_pool
gfs_vg<br> sudo lvconvert --chunksize 1280K
--thinpool gfs_vg/gfs_pool
--poolmetadata gfs_vg/gfs_pool_meta<br> sudo lvchange --zero n gfs_vg/gfs_pool<br> sudo lvcreate -V 19.5TiB --thinpool
gfs_vg/gfs_pool -n gfs_lv<br> sudo mkfs.xfs -f -i size=512 -n
size=8192 -d su=128k,sw=10
/dev/mapper/gfs_vg-gfs_lv<br> sudo vim /etc/fstab</div><div>/dev/mapper/gfs_vg-gfs_lv
/gluster/data/brick xfs
rw,inode64,noatime,nouuid 0 0<br></div><div><br></div><div>sudo systemctl daemon-reload
&& sudo mount -a<br> fio --name=test
--filename=/gluster/data/brick/wow
--size=1G --readwrite=write<br></div><div><br></div><div>Run status group 0 (all jobs):<br> WRITE: bw=2081MiB/s (2182MB/s),
2081MiB/s-2081MiB/s (2182MB/s-2182MB/s),
io=1024MiB (1074MB), run=492-492msec<br></div><div><br></div><div>All good there. 2182MB/s =~ 17.5
Gbps. Nice!<br></div><div><br></div><div><br></div><div>Gluster install:<br></div><div>export NODE1='10.54.95.123'<br> export NODE2='10.54.95.124'<br> export NODE3='10.54.95.125'<br> sudo gluster peer probe $NODE2<br> sudo gluster peer probe $NODE3<br> sudo gluster volume create data replica
3 arbiter 1 $NODE1:/gluster/data/brick
$NODE2:/gluster/data/brick
$NODE3:/gluster/data/brick force<br> sudo gluster volume set data
network.ping-timeout 5<br> sudo gluster volume set data
performance.client-io-threads on<br> sudo gluster volume set data group
metadata-cache<br> sudo gluster volume start data<br> sudo gluster volume info all<br></div><div><br>Volume Name: data<br> Type: Replicate<br> Volume ID:
b52b5212-82c8-4b1a-8db3-52468bc0226e<br> Status: Started<br> Snapshot Count: 0<br> Number of Bricks: 1 x (2 + 1) = 3<br> Transport-type: tcp<br> Bricks:<br> Brick1: 10.54.95.123:/gluster/data/brick<br> Brick2: 10.54.95.124:/gluster/data/brick<br> Brick3: 10.54.95.125:/gluster/data/brick
(arbiter)<br> Options Reconfigured:<br> network.inode-lru-limit: 200000<br> performance.md-cache-timeout: 600<br> performance.cache-invalidation: on<br> performance.stat-prefetch: on<br> features.cache-invalidation-timeout: 600<br> features.cache-invalidation: on<br> network.ping-timeout: 5<br> transport.address-family: inet<br> storage.fips-mode-rchecksum: on<br> nfs.disable: on<br> performance.client-io-threads: on</div><div><br></div><div>sudo vim /etc/fstab<br></div><div>localhost:/data /data
glusterfs defaults,_netdev
0 0<br></div><div><br></div><div>sudo systemctl daemon-reload
&& sudo mount -a<br></div><div>fio --name=test --filename=/data/wow
--size=1G --readwrite=write<br></div><div><br></div><div>Run status group 0 (all jobs):<br> WRITE: bw=109MiB/s (115MB/s),
109MiB/s-109MiB/s (115MB/s-115MB/s),
io=1024MiB (1074MB), run=9366-9366msec</div><div><br></div><div>Oh no, what's wrong? From 2182MB/s
down to only 115MB/s? What am I missing?
I'm not expecting the above ~17 Gbps,
but I'm thinking it should at least be
close(r) to ~10 Gbps. <br></div><div><br></div><div>Any suggestions?<br></div></div></div>________<br> <br> <br> <br> Community Meeting Calendar:<br> <br> Schedule -<br> Every 2nd and 4th Tuesday at 14:30 IST / 09:00
UTC<br> Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br> Gluster-users mailing list<br> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br> <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div></div></blockquote></div>________<br> <br> <br> <br> Community Meeting Calendar:<br> <br> Schedule -<br> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br> Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br> Gluster-users mailing list<br> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br> <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div></blockquote></div><br><pre>________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a>
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a>
<br></pre></blockquote><br></blockquote><br></div>________<br> <br> <br> <br> Community Meeting Calendar:<br> <br> Schedule -<br> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br> Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br> Gluster-users mailing list<br> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br> <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div>________<br> <br> <br> <br> Community Meeting Calendar:<br> <br> Schedule -<br> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br> Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br> Gluster-users mailing list<br> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br> <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div></blockquote></div>________<br> <br> <br> <br> Community Meeting Calendar:<br> <br> Schedule -<br> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br> Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br> Gluster-users mailing list<br> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br> <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></blockquote></div></div><br></body></html>