<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0cm;
        margin-bottom:.0001pt;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:purple;
        text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
        {mso-style-name:msonormal;
        mso-margin-top-alt:auto;
        margin-right:0cm;
        mso-margin-bottom-alt:auto;
        margin-left:0cm;
        font-size:11.0pt;
        font-family:"Calibri",sans-serif;}
span.E-MailFormatvorlage19
        {mso-style-type:personal-reply;
        font-family:"Calibri",sans-serif;
        color:windowtext;}
.MsoChpDefault
        {mso-style-type:export-only;
        font-size:10.0pt;}
@page WordSection1
        {size:612.0pt 792.0pt;
        margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
        {page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">@Stefan: Thank you. I‘ll stick to IPoIB for now, trusting that real bottlenecks will probably be elsewhere. At the moment on an almost empty node, gluster uses 0.5-1 cores of 102.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">I still hope that RDMA will be back sometime.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">@Saju: Thanks for the clarifications. The main reason I “complained” publicly is that RDMA support (<a href="https://docs.gluster.org/en/latest/Administrator-Guide/RDMA-Transport/">https://docs.gluster.org/en/latest/Administrator-Guide/RDMA-Transport/</a>)
in the official and current documentation has no indication in which versions it works or not. At least tiering is only mentioned as experimental in 3.7 and discontinued in release 6.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Rupert<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>Von:</b> Stefan Solbrig <stefan.solbrig@ur.de> <br>
<b>Gesendet:</b> Donnerstag, 25. Februar 2021 16:37<br>
<b>An:</b> gluster-users@gluster.org<br>
<b>Cc:</b> Schlick Rupert <Rupert.Schlick@ait.ac.at><br>
<b>Betreff:</b> Re: [EXT] [Gluster-users] RDMA support in gluster 9<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi Rupert,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">If you try RMDA (or work on the older, now removed RMDA code in glusterfs that use(d) ibverbs, be aware that there was a glitch, at least as of glusterfs v3.12.7: <a href="https://lists.gluster.org/pipermail/gluster-users/2018-May/034135.html">https://lists.gluster.org/pipermail/gluster-users/2018-May/034135.html</a>
A directory with tens of thousands of files did not show all directory entries.) I couldn't fix this back then and instead switched to ipoib. Although the measured per-file bandwith back then was better with RDMA than with ipoib by about 50%, I stayed with
ipoib because in the aggreate bandwith (many clients reading many files simultaneously) it didn't make a difference. (Our bottlenecks are elsewhere, not in the network bandwidth between server and client.) I've used ipoib since then to run a glusterfs with
more than 2 PB and haven't had serious problems so far.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">best wishes,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Stefan<o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Am 24.02.2021 um 10:59 schrieb Saju Mohammed Noohu <<a href="mailto:sajmoham@redhat.com">sajmoham@redhat.com</a>>:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Hi Rupert,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Apologies for the inconvenience, even though this was discussed, somehow it got missed in official release notes or documentation. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Few old discussions are below:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://github.com/gluster/glusterfs/issues/692">https://github.com/gluster/glusterfs/issues/692</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://lists.gluster.org/pipermail/gluster-users/2018-July/034400.html">https://lists.gluster.org/pipermail/gluster-users/2018-July/034400.html</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Will surely document this as part of Gluster docs on an appropriate heading.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">As evident from the above discussions, we deprecated it since we had no means to support it. If that changes, we'll be happy to work on re-enabling it, maybe as a compile-time option.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Saju<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Wed, Feb 24, 2021 at 1:23 PM Schlick Rupert <<a href="mailto:Rupert.Schlick@ait.ac.at">Rupert.Schlick@ait.ac.at</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">What surprised me is more that, if not looking into the developer channels, there is no indication that the feature is gone till it crashes when you try to use
it. I did not think of Gluster as a developer-only tool, even in the “community version”.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Native RDMA support was the main reason for me to choose GlusterFS over MooseFS – although it’s just a feature on “paper” and I do not really know yet the real
needs and bottlenecks of our new cluster. With both RDMA and tiering gone, I based my decision on completely outdated information about gluster features.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">@strahil: I hope to find some time to try your suggestion next week, thanks. But even if it works, I will most probably not risk using something the developers
consider not fit for use anymore. There seem also to be issues with mixed mode and multiple IP addresses anyway – our computation and storage nodes are on a infiniband (RDMA and IP over Infiniband) island network, but all are also connected to the outside
via IP over Ethernet. The login nodes, which would also need access to the cluster file system are in a VM and only accessible via IP over Ethernet</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Rupert</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>Von:</b> Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" target="_blank">hunter86_bg@yahoo.com</a>>
<br>
<b>Gesendet:</b> Dienstag, 23. Februar 2021 11:20<br>
<b>An:</b> Schlick Rupert <<a href="mailto:Rupert.Schlick@ait.ac.at" target="_blank">Rupert.Schlick@ait.ac.at</a>>; Saju Mohammed Noohu <<a href="mailto:sajmoham@redhat.com" target="_blank">sajmoham@redhat.com</a>><br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Betreff:</b> Re: [Gluster-users] RDMA support in gluster 9<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">As everything deprecated, it never get's completely removed from source.<o:p></o:p></p>
<div id="gmail-m_-1736521010177902879yMail_cursorElementTracker_1614075471935">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div id="gmail-m_-1736521010177902879yMail_cursorElementTracker_1614075472100">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Can you try to 'make' from source and check if v9 by default has 'Infiniband verbs' disabled ?<o:p></o:p></p>
</div>
<div id="gmail-m_-1736521010177902879yMail_cursorElementTracker_1614075514421">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">If yes, then you can compile with enabled supportand try that.<o:p></o:p></p>
</div>
<div id="gmail-m_-1736521010177902879yMail_cursorElementTracker_1614075547333">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div id="gmail-m_-1736521010177902879yMail_cursorElementTracker_1614075547537">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Best Regards,<o:p></o:p></p>
</div>
<div id="gmail-m_-1736521010177902879yMail_cursorElementTracker_1614075551597">
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Strahil Nikolov<o:p></o:p></p>
<blockquote style="margin-left:0cm;margin-top:5.0pt;margin-right:0cm;margin-bottom:15.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Times New Roman",serif;color:#6D00F6">On Tue, Feb 23, 2021 at 10:40, Schlick Rupert</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Times New Roman",serif;color:#6D00F6"><<a href="mailto:Rupert.Schlick@ait.ac.at" target="_blank">Rupert.Schlick@ait.ac.at</a>> wrote:</span><o:p></o:p></p>
</div>
</div>
<div style="border:none;border-left:solid #6D00F6 1.0pt;padding:0cm 0cm 0cm 15.0pt;margin-top:7.5pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></p>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal">________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://meet.google.com/cpu-eiue-hvk">https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>