[Gluster-devel] RDMA: Patch to make use of pre registered memory

[Anand Avati]

Couple of comments -

1. rdma can register init/fini functions (via pointers) into iobuf_pool.
Absolutely no need to introduce rdma dependency into libglusterfs.

2. It might be a good idea to take a holistic approach towards zero-copy
with libgfapi + RDMA, rather than a narrow goal of "use pre-registered
memory with RDMA". Do keep the options open for RDMA'ing user's memory
pointer (passed to glfs_write()) as well.

3. It is better to make io-cache and write-behind use a new iobuf_pool for
caching purpose. There could be an optimization where they could just do
iobuf/iobref_ref() when safe - e.g io-cache can cache with iobuf_ref when
transport is socket, or write-behind can unwind by holding onto data with
iobuf_ref() when the topmost layer is FUSE or server (i.e no gfapi).

4. Next step for zero-copy would be introduction of a new fop readto()
where the destination pointer is passed from the caller (gfapi being the
primary use case). In this situation RDMA ought to register that memory if
necessary and request server to RDMA_WRITE into the pointer provided by
gfapi caller.

2. and 4. require changes in the code you would be modifying if you were to
just do "pre-registered memroy", so it is better we plan for the bigger
picture upfront. Zero-copy can improve performance (especially read) in
qemu use case.

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20150124/d57662e9/attachment.html>