<html xmlns="http://www.w3.org/1999/xhtml" xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office"><head><!--[if gte mso 9]><xml><o:OfficeDocumentSettings><o:AllowPNG/><o:PixelsPerInch>96</o:PixelsPerInch></o:OfficeDocumentSettings></xml><![endif]--></head><body>
Hi,<div><br></div><div>Let’s try the simple things:</div><div><br></div><div>Check if you can use MTU9000 and if it’s possible, set it on the Bond Slaves and the bond devices:</div><div><span style="-webkit-text-size-adjust: auto;"> ping GLUSTER_PEER </span><span style="-webkit-text-size-adjust: auto;">-c 10 -M do -s 8972</span></div><div><br>Then try to follow up the recommendations from <a href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance" target="_blank" class="enhancr2_A6E0775D-02AB-4B81-9762-F1F1F8931B8F">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance</a> <div><br></div><div><br></div>Best Regards,</div><div>Strahil Nikolov <br><div class="yahoo-signature"></div><br><p class="yahoo-quoted-begin" style="font-size: 15px; color: #715FFA; padding-top: 15px; margin-top: 0">On Monday, December 11, 2023, 3:32 PM, Danny <dbray925+gluster@gmail.com> wrote:</p><blockquote class="iosymail"><div id="yiv1869372156"><div dir="ltr">
<div>Hello list, I'm hoping someone can let me know what setting I missed.</div><div><br></div><div>Hardware:</div><div>Dell R650 servers, Dual 24 Core Xeon 2.8 GHz, 1 TB RAM<br></div><div>8x SSD s
<span>Negotiated Speed</span> 12 Gbps</div><div>PERC H755 Controller - RAID 6 <br></div><div>Created virtual "data" disk from the above 8 SSD drives, for a ~20 TB /dev/sdb<br></div><div><br></div><div>OS:</div><div>CentOS Stream</div><div>kernel-4.18.0-526.el8.x86_64</div><div>glusterfs-7.9-1.el8.x86_64</div><div><br></div><div>IPERF Test between nodes:<br>[ ID] Interval Transfer Bitrate Retr<br>[ 5] 0.00-10.00 sec 11.5 GBytes 9.90 Gbits/sec 0 sender<br>[ 5] 0.00-10.04 sec 11.5 GBytes 9.86 Gbits/sec receiver<br></div><div><br></div><div>All good there. ~10 Gbps, as expected.<br></div><div><br></div><div>LVM Install:</div><div>export DISK="/dev/sdb"<br>sudo parted --script $DISK "mklabel gpt"<br>sudo parted --script $DISK "mkpart primary 0% 100%"<br>sudo parted --script $DISK "set 1 lvm on"</div><div>sudo pvcreate --dataalignment 128K /dev/sdb1<br>sudo vgcreate --physicalextentsize 128K gfs_vg /dev/sdb1<br>sudo lvcreate -L 16G -n gfs_pool_meta gfs_vg<br>sudo lvcreate -l 95%FREE -n gfs_pool gfs_vg<br>sudo lvconvert --chunksize 1280K --thinpool gfs_vg/gfs_pool --poolmetadata gfs_vg/gfs_pool_meta<br>sudo lvchange --zero n gfs_vg/gfs_pool<br>sudo lvcreate -V 19.5TiB --thinpool gfs_vg/gfs_pool -n gfs_lv<br>sudo mkfs.xfs -f -i size=512 -n size=8192 -d su=128k,sw=10 /dev/mapper/gfs_vg-gfs_lv<br>sudo vim /etc/fstab</div><div>/dev/mapper/gfs_vg-gfs_lv /gluster/data/brick xfs rw,inode64,noatime,nouuid 0 0</div><div><br></div><div>sudo systemctl daemon-reload && sudo mount -a<br>fio --name=test --filename=/gluster/data/brick/wow --size=1G --readwrite=write<br></div><div><br></div><div>Run status group 0 (all jobs):<br> WRITE: bw=2081MiB/s (2182MB/s), 2081MiB/s-2081MiB/s (2182MB/s-2182MB/s), io=1024MiB (1074MB), run=492-492msec<br></div><div><br></div><div>All good there. 2182MB/s =~ 17.5 Gbps. Nice!<br></div><div><br></div><div><br></div><div>Gluster install:</div><div>export NODE1='10.54.95.123'<br>export NODE2='10.54.95.124'<br>export NODE3='10.54.95.125'<br>sudo gluster peer probe $NODE2<br>sudo gluster peer probe $NODE3<br>sudo
gluster volume create data replica 3 arbiter 1
$NODE1:/gluster/data/brick $NODE2:/gluster/data/brick
$NODE3:/gluster/data/brick force<br>sudo gluster volume set data network.ping-timeout 5<br>sudo gluster volume set data performance.client-io-threads on<br>sudo gluster volume set data group metadata-cache<br>sudo gluster volume start data<br>sudo gluster volume info all<br></div><div><br>Volume Name: data<br>Type: Replicate<br>Volume ID: b52b5212-82c8-4b1a-8db3-52468bc0226e<br>Status: Started<br>Snapshot Count: 0<br>Number of Bricks: 1 x (2 + 1) = 3<br>Transport-type: tcp<br>Bricks:<br>Brick1: 10.54.95.123:/gluster/data/brick<br>Brick2: 10.54.95.124:/gluster/data/brick<br>Brick3:
10.54.95.125:/gluster/data/brick (arbiter)<br>Options Reconfigured:<br>network.inode-lru-limit: 200000<br>performance.md-cache-timeout: 600<br>performance.cache-invalidation: on<br>performance.stat-prefetch: on<br>features.cache-invalidation-timeout: 600<br>features.cache-invalidation: on<br>network.ping-timeout: 5<br>transport.address-family: inet<br>storage.fips-mode-rchecksum: on<br>nfs.disable: on<br>performance.client-io-threads: on</div><div><br></div><div>sudo vim /etc/fstab<br></div><div>localhost:/data /data glusterfs defaults,_netdev 0 0</div><div><br></div><div>sudo systemctl daemon-reload && sudo mount -a</div><div>fio --name=test --filename=/data/wow --size=1G --readwrite=write</div><div><br></div><div>Run status group 0 (all jobs):<br> WRITE: bw=109MiB/s (115MB/s), 109MiB/s-109MiB/s (115MB/s-115MB/s), io=1024MiB (1074MB), run=9366-9366msec</div><div><br></div><div>Oh
no, what's wrong? From
2182MB/s down to only 115MB/s? What am I missing? I'm not expecting the
above ~17 Gbps, but I'm thinking it should at least be close(r) to ~10
Gbps. <br></div><div><br></div><div>Any suggestions?</div>
</div>
</div>________<br><br><br><br>Community Meeting Calendar:<br><br>Schedule -<br>Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>Gluster-users mailing list<br><a ymailto="mailto:Gluster-users@gluster.org" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br><blockquote></blockquote></blockquote></div>
</body></html>