<div dir="ltr">If it is for academic purposes, I suggest using glusterfs-3.12.x or glusterfs-5.x series with RDMA... That would give an idea.<div><br></div><div>-Amar</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Oct 8, 2019 at 7:44 PM Liu, Changcheng <<a href="mailto:changcheng.liu@intel.com">changcheng.liu@intel.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Thanks Rafi-KC & Tumbali.<br>
I'm working on developing RDMA in Ceph and doing some investigating about RDMA usage in distributed storage system.<br>
It's unlucky that GlusterFs won't support RDMA now.<br>
<br>
For IPoIB, I'll try to evaluate its performance versus RDMA(RoCEv2 & iWARP).<br>
<br>
-----Original Message-----<br>
From: Rafi Kavungal Chundattu Parambil [mailto:<a href="mailto:rkavunga@redhat.com" target="_blank">rkavunga@redhat.com</a>] <br>
Sent: Monday, October 7, 2019 5:07 PM<br>
To: Liu, Changcheng <<a href="mailto:changcheng.liu@intel.com" target="_blank">changcheng.liu@intel.com</a>><br>
Cc: <a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>; <a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a><br>
Subject: Re: RDMA in GlusterFs<br>
<br>
Hi Liu,<br>
<br>
RDMA support in GlusterFS has been dropped from the latest release and removed from the codebase. You can find more details here in the mail sent as the proposal to drop the feature[1].<br>
<br>
<snippet from proposal mail><br>
Gluster<br>
started supporting RDMA while ib-verbs was still new, and very high-end infra around that time were using Infiniband. Engineers did work with Mellanox, and got the technology into GlusterFS for better data migration, data copy. While current day kernels support very good speed with IPoIB module itself, and there are no more bandwidth for experts in these area to maintain the feature, we recommend migrating over to TCP (IP based) network for your volume.If you are successfully using RDMA transport, do get in touch with us to prioritize the migration plan for your volume. Plan is to work on this after the release, so by version 6.0, we will have a cleaner transport code, which just needs to support one type.<br>
</end od snippet><br>
<br>
That said, Officially GlusterFs doesn't support RDMA on master branch.<br>
<br>
[1] : <a href="https://lists.gluster.org/pipermail/gluster-users/2018-July/034400.html" rel="noreferrer" target="_blank">https://lists.gluster.org/pipermail/gluster-users/2018-July/034400.html</a><br>
<br>
<br>
Regards<br>
Rafi KC<br>
><br>
><br>
> Does anyone use RDMA in Glusterfs before? What’s RDMA usability in <br>
> Glusterfs now?<br>
><br>
><br>
><br>
> *From:* Liu, Changcheng<br>
> *Sent:* Tuesday, October 1, 2019 11:24 PM<br>
> *To:* 'Yaniv Kaul' <<a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>>; 'Amar Tumballi' <br>
> <<a href="mailto:amarts@redhat.com" target="_blank">amarts@redhat.com</a>><br>
> *Cc:* '<a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a>' <<a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a>><br>
> *Subject:* RE: RDMA in GlusterFs<br>
><br>
><br>
><br>
> Does anyone know whether Glusterfs support RDMA or not in master <br>
> branch now?<br>
><br>
><br>
><br>
> *From:* Liu, Changcheng<br>
> *Sent:* Tuesday, October 1, 2019 12:16 AM<br>
> *To:* Yaniv Kaul <<a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>>; Amar Tumballi <<a href="mailto:amarts@redhat.com" target="_blank">amarts@redhat.com</a>><br>
> *Cc:* <a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a><br>
> *Subject:* RDMA in GlusterFs<br>
><br>
><br>
><br>
> Hi Yaniv Kaul & Amar Tumbali<br>
><br>
> Upstream question:<br>
><br>
> Does Glusterfs support RDMA after below PR is merged?<br>
><br>
> ibverbs/rdma: remove from build:<br>
> <a href="https://github.com/gluster/glusterfs/commit/8f37561d665b" rel="noreferrer" target="_blank">https://github.com/gluster/glusterfs/commit/8f37561d665b</a><br>
><br>
><br>
><br>
> B.R.<br>
><br>
> Changcheng<br>
><br>
</blockquote></div>