[Gluster-devel] Proposal for new glfs api for zero copy read/write.

[Mohammed Rafi K C]

Hi All,

As per the current design of glfs read/write, it incur an extra level
copy for both read and write. To avoid this extra level copy and to work
well with rdma protocol we are come up with an idea of bringing new
api's that include two buffer management API's ,based on the discussion
in gluster devel[1] by Ben England and personal discussion with him.


[1]:https://www.mail-archive.com/gluster-devel@gluster.org/msg03338.html

API Proto type :

1) void** glfs_get_buffer (char *handler, int64_t size, int count);
2)int glfs_free_buffer (char *handler);
3)ssize_t glfs_zero_readv ( struct glfs_fd *glfd, const struct iovec
*iovec, int iovcnt,
                                           off_t offset, int flags, void
*handler);
4)ssize_t glfs_zero_writev ( struct glfs_fd *glfd, const struct iovec
*iovec, int iovcnt,
                                           off_toffset , int flags, void
*handler);

API in details:


1) void** glfs_get_buffer ( struct glfs fs, char **handler, int64_t size, int count);


glfs_get_buffer will return an array of void* that contains count number
of entries,each address will hold a buffer of size "size". The iobref
will be returned as handler, this can be used for freeing the iobufs in
iobrefs. So the handler should be NULL for the first buffer request. if
some one need a different size of buffers, they can request for the
buffer with an existing handler. Then the new buffer will be added in
the iobref.


2)int glfs_free_buffer (char *handler);

we will do an unref of iobref from this function. Every one should call
this function once they are done with buffer returned by glfs_get_buffer
and glfs_zero_readv.

3)ssize_t glfs_zero_readv ( struct glfs_fd *glfd, const struct iovec
*iovec, int iovcnt,
                                           off_t offset, int flags, void
*handler);

This function will read the data to an iobuf and the pointer will be set
in the iovec. the corresponding iobref will be returned through handler.
Once we copmleted the use of read buffer, user should call
glfs_free_bufferto return the iobufs to the pool.

4)ssize_t glfs_zero_writev ( struct glfs_fd *glfd, const struct iovec
*iovec, int iovcnt,
                                           off_toffset , int flags, void
*handler);

glfs_zero_writev should call with a buffer allocated through
glfs_get_buffer and the same handler should use as handler argument.



By this method we can bring zero copy for read/write and we can make use
of pre registered iobufs.

Please provide your thoughts.

Thanks & Regards
Rafi KC