<div dir="ltr">Sorry, I noticed that too after I posted, so I instantly upgraded to 10. Issue remains. <br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 12, 2023 at 1:09 PM Gilberto Ferreira <<a href="mailto:gilberto.nunes32@gmail.com">gilberto.nunes32@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I strongly suggest you update to version 10 or higher. <br>It's come with significant improvement regarding performance.<br clear="all"><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>---</div><div><div><div>Gilberto Nunes Ferreira</div></div><div><span style="font-size:12.8px">(47) 99676-7530 - Whatsapp / Telegram</span><br></div><div><p style="font-size:12.8px;margin:0px"></p><p style="font-size:12.8px;margin:0px"><br></p><p style="font-size:12.8px;margin:0px"><br></p></div></div><div><br></div></div></div></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Em ter., 12 de dez. de 2023 às 13:03, Danny <<a href="mailto:dbray925%2Bgluster@gmail.com" target="_blank">dbray925+gluster@gmail.com</a>> escreveu:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">
MTU is already 9000, and as you can see from the IPERF results, I've got a nice, fast connection between the nodes.
</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 12, 2023 at 9:49 AM Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" target="_blank">hunter86_bg@yahoo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
Hi,<div><br></div><div>Let’s try the simple things:</div><div><br></div><div>Check if you can use MTU9000 and if it’s possible, set it on the Bond Slaves and the bond devices:</div><div><span> ping GLUSTER_PEER </span><span>-c 10 -M do -s 8972</span></div><div><br>Then try to follow up the recommendations from <a href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance" target="_blank">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance</a> <div><br></div><div><br></div>Best Regards,</div><div>Strahil Nikolov <br><div></div><br><p style="font-size:15px;color:rgb(113,95,250);padding-top:15px;margin-top:0px">On Monday, December 11, 2023, 3:32 PM, Danny <<a href="mailto:dbray925%2Bgluster@gmail.com" target="_blank">dbray925+gluster@gmail.com</a>> wrote:</p><blockquote><div id="m_-3283052153129726045m_98609290250261231m_810164770777586547yiv1869372156"><div dir="ltr">
<div>Hello list, I'm hoping someone can let me know what setting I missed.</div><div><br></div><div>Hardware:</div><div>Dell R650 servers, Dual 24 Core Xeon 2.8 GHz, 1 TB RAM<br></div><div>8x SSD s
<span>Negotiated Speed</span> 12 Gbps</div><div>PERC H755 Controller - RAID 6 <br></div><div>Created virtual "data" disk from the above 8 SSD drives, for a ~20 TB /dev/sdb<br></div><div><br></div><div>OS:</div><div>CentOS Stream</div><div>kernel-4.18.0-526.el8.x86_64</div><div>glusterfs-7.9-1.el8.x86_64</div><div><br></div><div>IPERF Test between nodes:<br>[ ID] Interval Transfer Bitrate Retr<br>[ 5] 0.00-10.00 sec 11.5 GBytes 9.90 Gbits/sec 0 sender<br>[ 5] 0.00-10.04 sec 11.5 GBytes 9.86 Gbits/sec receiver<br></div><div><br></div><div>All good there. ~10 Gbps, as expected.<br></div><div><br></div><div>LVM Install:</div><div>export DISK="/dev/sdb"<br>sudo parted --script $DISK "mklabel gpt"<br>sudo parted --script $DISK "mkpart primary 0% 100%"<br>sudo parted --script $DISK "set 1 lvm on"</div><div>sudo pvcreate --dataalignment 128K /dev/sdb1<br>sudo vgcreate --physicalextentsize 128K gfs_vg /dev/sdb1<br>sudo lvcreate -L 16G -n gfs_pool_meta gfs_vg<br>sudo lvcreate -l 95%FREE -n gfs_pool gfs_vg<br>sudo lvconvert --chunksize 1280K --thinpool gfs_vg/gfs_pool --poolmetadata gfs_vg/gfs_pool_meta<br>sudo lvchange --zero n gfs_vg/gfs_pool<br>sudo lvcreate -V 19.5TiB --thinpool gfs_vg/gfs_pool -n gfs_lv<br>sudo mkfs.xfs -f -i size=512 -n size=8192 -d su=128k,sw=10 /dev/mapper/gfs_vg-gfs_lv<br>sudo vim /etc/fstab</div><div>/dev/mapper/gfs_vg-gfs_lv /gluster/data/brick xfs rw,inode64,noatime,nouuid 0 0</div><div><br></div><div>sudo systemctl daemon-reload && sudo mount -a<br>fio --name=test --filename=/gluster/data/brick/wow --size=1G --readwrite=write<br></div><div><br></div><div>Run status group 0 (all jobs):<br> WRITE: bw=2081MiB/s (2182MB/s), 2081MiB/s-2081MiB/s (2182MB/s-2182MB/s), io=1024MiB (1074MB), run=492-492msec<br></div><div><br></div><div>All good there. 2182MB/s =~ 17.5 Gbps. Nice!<br></div><div><br></div><div><br></div><div>Gluster install:</div><div>export NODE1='10.54.95.123'<br>export NODE2='10.54.95.124'<br>export NODE3='10.54.95.125'<br>sudo gluster peer probe $NODE2<br>sudo gluster peer probe $NODE3<br>sudo
gluster volume create data replica 3 arbiter 1
$NODE1:/gluster/data/brick $NODE2:/gluster/data/brick
$NODE3:/gluster/data/brick force<br>sudo gluster volume set data network.ping-timeout 5<br>sudo gluster volume set data performance.client-io-threads on<br>sudo gluster volume set data group metadata-cache<br>sudo gluster volume start data<br>sudo gluster volume info all<br></div><div><br>Volume Name: data<br>Type: Replicate<br>Volume ID: b52b5212-82c8-4b1a-8db3-52468bc0226e<br>Status: Started<br>Snapshot Count: 0<br>Number of Bricks: 1 x (2 + 1) = 3<br>Transport-type: tcp<br>Bricks:<br>Brick1: 10.54.95.123:/gluster/data/brick<br>Brick2: 10.54.95.124:/gluster/data/brick<br>Brick3:
10.54.95.125:/gluster/data/brick (arbiter)<br>Options Reconfigured:<br>network.inode-lru-limit: 200000<br>performance.md-cache-timeout: 600<br>performance.cache-invalidation: on<br>performance.stat-prefetch: on<br>features.cache-invalidation-timeout: 600<br>features.cache-invalidation: on<br>network.ping-timeout: 5<br>transport.address-family: inet<br>storage.fips-mode-rchecksum: on<br>nfs.disable: on<br>performance.client-io-threads: on</div><div><br></div><div>sudo vim /etc/fstab<br></div><div>localhost:/data /data glusterfs defaults,_netdev 0 0</div><div><br></div><div>sudo systemctl daemon-reload && sudo mount -a</div><div>fio --name=test --filename=/data/wow --size=1G --readwrite=write</div><div><br></div><div>Run status group 0 (all jobs):<br> WRITE: bw=109MiB/s (115MB/s), 109MiB/s-109MiB/s (115MB/s-115MB/s), io=1024MiB (1074MB), run=9366-9366msec</div><div><br></div><div>Oh
no, what's wrong? From
2182MB/s down to only 115MB/s? What am I missing? I'm not expecting the
above ~17 Gbps, but I'm thinking it should at least be close(r) to ~10
Gbps. <br></div><div><br></div><div>Any suggestions?</div>
</div>
</div>________<br><br><br><br>Community Meeting Calendar:<br><br>Schedule -<br>Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br><blockquote></blockquote></blockquote></div>
</div></blockquote></div>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" rel="noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div>
</blockquote></div>