<div dir="auto"><div>great hints, thanks a lot, going to test all this tomorrow.</div><div dir="auto">I have an appointment with IT department for enabling the LACP bonds on 10gig dual port interfaces, so next will test the latency, and ssd+xfs+glusterfs.</div><div dir="auto">we did not touch default mtu, i did some tests with CentOS7.3 on IB with 65k mtu connected mode ipoib, was ok but not stable on the large workloads, maybe now the situation has been changed.</div><div dir="auto">are there any suggestions on lacp-bonds and mtu sizes?</div><div dir="auto"><br></div><div dir="auto"><br><br><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com">hunter86_bg@yahoo.com</a>> schrieb am Mo., 29. März 2021, 19:02:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I guess there is no need to mention that lattency is the real killer of Gluster. What is the DC-to-DC lattency and MTU (ethernet) ?<div id="m_-3785411564711768569yMail_cursorElementTracker_1617036805306">Also, if you use SSDs , consider using noop/none I/O schedulers.</div><div id="m_-3785411564711768569yMail_cursorElementTracker_1617036874320"><br></div><div id="m_-3785411564711768569yMail_cursorElementTracker_1617036874513">Also, you can obtain the tuned profiles used in Red Hat Gluster Storage via this source rpm:</div><div id="m_-3785411564711768569yMail_cursorElementTracker_1617037064167"><br></div><div id="m_-3785411564711768569yMail_cursorElementTracker_1617037064645"><a href="http://ftp.redhat.com/redhat/linux/enterprise/7Server/en/RHS/SRPMS/redhat-storage-server-3.5.0.0-7.el7rhgs.src.rpm" target="_blank" rel="noreferrer">http://ftp.redhat.com/redhat/linux/enterprise/7Server/en/RHS/SRPMS/redhat-storage-server-3.5.0.0-7.el7rhgs.src.rpm</a></div><div id="m_-3785411564711768569yMail_cursorElementTracker_1617037067890"><br></div><div id="m_-3785411564711768569yMail_cursorElementTracker_1617037068340">You can combine the settings from the tuned profile for Hypervisor and combine it with the gluster random I/O tuned profile.</div><div id="m_-3785411564711768569yMail_cursorElementTracker_1617037182745"><br></div><div id="m_-3785411564711768569yMail_cursorElementTracker_1617037182916">Also worth mentioning, RHGS uses 512M shard size, while default in upstream gluster is just 64M. Some oVirt users have reported issues and suspect is gluster's inability to crwate enough shards.</div><div id="m_-3785411564711768569yMail_cursorElementTracker_1617037311178"><br></div><div id="m_-3785411564711768569yMail_cursorElementTracker_1617037311375">WARNING: ONCE SHARDING IS ENABLED, NEVER EVER DISABLE IT.</div><div id="m_-3785411564711768569yMail_cursorElementTracker_1617037149614"><br></div><div id="m_-3785411564711768569yMail_cursorElementTracker_1617037149834">Best Regards,</div><div id="m_-3785411564711768569yMail_cursorElementTracker_1617037154491">Strahil Nikolov<br> <br> <blockquote style="margin:0 0 20px 0"> <div style="font-family:Roboto,sans-serif;color:#6d00f6"> <div>On Mon, Mar 29, 2021 at 11:03, Arman Khalatyan</div><div><<a href="mailto:arm2arm@gmail.com" target="_blank" rel="noreferrer">arm2arm@gmail.com</a>> wrote:</div> </div> <div style="padding:10px 0 0 20px;margin:10px 0 0 0;border-left:1px solid #6d00f6"> <div id="m_-3785411564711768569yiv7383663816"><div><div dir="ltr"><div><div>Thanks Strahil,</div><div>good point on choose-local, definitely we will try.</div><div>the connection is: 10Gbit, also FDR Infiniband( ipoib will be used).</div><div>we are still experimenting with 2 buildings and 8 nodes ovirt+ changing the bricks number on glusterfs. <br clear="none"><br clear="none"><div><div dir="ltr">Strahil Nikolov <<a rel="nofollow noopener noreferrer noreferrer" shape="rect" href="mailto:hunter86_bg@yahoo.com" target="_blank">hunter86_bg@yahoo.com</a>> schrieb am So., 28. März 2021, 00:35:<br clear="none"></div><div id="m_-3785411564711768569yiv7383663816yqt50377"><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">It's worth mentioning that if your network bandwidth is smaller than the raid bandwidth, you can consider to enable the cluster.choose-local (which oVirt's optimizations disable) for faster reads.<div id="m_-3785411564711768569yiv7383663816m_-5271630689013369476m_-159553019428442139yMail_cursorElementTracker_1616887990856"><br clear="none"></div><div id="m_-3785411564711768569yiv7383663816m_-5271630689013369476m_-159553019428442139yMail_cursorElementTracker_1616888030346">Some people would also consider going with JBOD (replica 3) mode. I guess you can test both prior moving to prod phase </div><div id="m_-3785411564711768569yiv7383663816m_-5271630689013369476m_-159553019428442139yMail_cursorElementTracker_1616888080152"><br clear="none"></div><div id="m_-3785411564711768569yiv7383663816m_-5271630689013369476m_-159553019428442139yMail_cursorElementTracker_1616888080396"><br clear="none"></div><div id="m_-3785411564711768569yiv7383663816m_-5271630689013369476m_-159553019428442139yMail_cursorElementTracker_1616887999939">P.S.: Don't forget to align the LVM/FS layer to the hardware raid.</div><div id="m_-3785411564711768569yiv7383663816m_-5271630689013369476m_-159553019428442139yMail_cursorElementTracker_1616888027871"><br clear="none"></div><div id="m_-3785411564711768569yiv7383663816m_-5271630689013369476m_-159553019428442139yMail_cursorElementTracker_1616887991058">Best Regards,</div><div id="m_-3785411564711768569yiv7383663816m_-5271630689013369476m_-159553019428442139yMail_cursorElementTracker_1616887995872">Strahil Nikolov</div></blockquote></div></div></div></div>
</div>
</div></div> </div> </blockquote></div></blockquote></div></div></div>