[Gluster-devel] Suggestion needed to make use of iobuf_pool as rdma buffer.

[Ben England]

Rafi,

it totally makes sense to me that you need to pre-allocate i/o buffers that will be used by RDMA, and you don't want to constantly change (i.e. allocate and deallocate) these buffers.  Since a remote RDMA controller can be reading and writing to them, we have to be very careful about deallocating in particular.  So an "arena" of pre-registered RDMA buffers makes perfect sense.

Am I understanding you correctly that io-cache translator is soaking up all the RDMA-related buffers?   How important is io-cache translator to Gluster performance at this point?  Given that FUSE caching is now enabled, it seems to me that io-cache translator would accomplish very little.  Should we have it disabled by default?  If so, would that solve your problem?

So how do read-ahead translator and write-behind translator interact with RDMA buffering?

-ben

----- Original Message -----
> From: "Mohammed Rafi K C" <rkavunga at redhat.com>
> To: gluster-devel at gluster.org
> Sent: Tuesday, January 13, 2015 9:29:56 AM
> Subject: [Gluster-devel] Suggestion needed to make use of iobuf_pool as rdma	buffer.
> 
> Hi All,
> 
> When using RDMA protocol, we need to register the buffer which is going
> to send through rdma with rdma device. In fact, it is a costly
> operation, and a performance killer if it happened in I/O path. So our
> current plan is to register pre-allocated iobuf_arenas from  iobuf_pool
> with rdma when rdma is getting initialized. The problem comes when all
> the iobufs are exhausted, then we need to dynamically allocate new
> arenas from libglusterfs module. Since it is created in libglusterfs, we
> can't make a call to rdma from libglusterfs. So we will force to
> register each of the iobufs from the newly created arenas with rdma in
> I/O path. If io-cache is turned on in client stack, then all the
> pre-registred arenas will use by io-cache as cache buffer. so we have to
> do the registration in rdma for each i/o call for every iobufs,
> eventually we cannot make use of pre registered arenas.
> 
> To address the issue, we have two approaches in mind,
> 
>  1) Register each dynamically created buffers in iobuf by bringing
> transport layer together with libglusterfs.
> 
>  2) create a separate buffer for caching and offload the data from the
> read response to the cache buffer in background.
> 
> If we could make use of preregister memory for every rdma call, then we
> will have approximately 20% increment for write and 25% of increment for
> read.
> 
> Please give your thoughts to address the issue.
> 
> Thanks & Regards
> Rafi KC
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>