[Gluster-devel] libgfapi zero copy write - application in samba, nfs-ganesha

Tue Sep 27 08:02:34 UTC 2016

W.r.t Samba consuming this, it requires a great deal of code change in Samba.
Currently samba has no concept of getting buf from the underlying file system,
the filesystem comes into picture only at the last layer(gluster plugin),
where system calls are replaced by libgfapi calls. Hence, this is not readily
consumable by Samba, and i think same will be the case with NFS_Ganesha, will
let the Ganesha folksc comment on the same.

Regards,
Poornima

----- Original Message -----
> From: "Ric Wheeler" <ricwheeler at gmail.com>
> To: "Raghavendra Gowdappa" <rgowdapp at redhat.com>, "Ric Wheeler" <rwheeler at redhat.com>
> Cc: "Ben England" <bengland at redhat.com>, "Gluster Devel" <gluster-devel at gluster.org>, "Ben Turner"
> <bturner at redhat.com>
> Sent: Tuesday, September 27, 2016 2:25:40 AM
> Subject: Re: [Gluster-devel] libgfapi zero copy write - application in samba, nfs-ganesha
> 
> On 09/27/2016 08:53 AM, Raghavendra Gowdappa wrote:
> >
> > ----- Original Message -----
> >> From: "Ric Wheeler" <rwheeler at redhat.com>
> >> To: "Raghavendra Gowdappa" <rgowdapp at redhat.com>, "Saravanakumar Arumugam"
> >> <sarumuga at redhat.com>
> >> Cc: "Gluster Devel" <gluster-devel at gluster.org>, "Ben Turner"
> >> <bturner at redhat.com>, "Ben England"
> >> <bengland at redhat.com>
> >> Sent: Tuesday, September 27, 2016 10:51:48 AM
> >> Subject: Re: [Gluster-devel] libgfapi zero copy write - application in
> >> samba, nfs-ganesha
> >>
> >> On 09/27/2016 07:56 AM, Raghavendra Gowdappa wrote:
> >>> +Manoj, +Ben turner, +Ben England.
> >>>
> >>> @Perf-team,
> >>>
> >>> Do you think the gains are significant enough, so that smb and
> >>> nfs-ganesha
> >>> team can start thinking about consuming this change?
> >>>
> >>> regards,
> >>> Raghavendra
> >> This is a large gain but I think that we might see even larger gains (a
> >> lot
> >> depends on how we implement copy offload :)).
> > Can you elaborate on what you mean "copy offload"? If it is the way we
> > avoid a copy in gfapi (from application buffer), following is the
> > workflow:
> >
> > <commit>
> >
> > Work flow of zero copy write operation:
> > --------------------------------------
> >
> > 1) Application requests a buffer of specific size. A new buffer is
> > allocated from iobuf pool, and this buffer is passed on to application.
> >     Achieved using "glfs_get_buffer"
> >
> > 2) Application writes into the received buffer, and passes that to
> > libgfapi, and libgfapi in turn passes the same buffer to underlying
> > translators. This avoids a memcpy in glfs write
> >     Achieved using "glfs_zero_write"
> >
> > 3) Once the write operation is complete, Application must take the
> > responsibilty of freeing the buffer.
> >     Achieved using "glfs_free_buffer"
> >
> > </commit>
> >
> > Do you've any suggestions/improvements on this? I think Shyam mentioned an
> > alternative approach (for zero-copy readv I think), let me look up at that
> > too.
> >
> > regards,
> > Raghavendra
> 
> Both NFS and SMB support a copy offload that allows a client to produce a new
> copy of a file without bringing data over the wire. Both, if I remember
> correctly, do a ranged copy within a file.
> 

Yup, also referred to as Server side copy, Niels is working on having this for Gluster.

> The key here is that since the data does not move over the wire from server
> to
> client, we can shift the performance bottleneck to the storage server.
> 
> If we have a slow (1GB) link between client and server, we should be able to
> do
> that copy as if it happened just on the server itself. For a single NFS
> server
> (not a clustered, scale out server), that usually means we are as fast as the
> local file system copy.
> 
> Note that there are also servers that simply "reflink" that file, so we have
> a
> very small amount of time needed on the server to produce that copy.  This
> can
> be a huge win for say a copy of a virtual machine guest image.
> 
> Gluster and other distributed servers won't benefit as much as a local server
> would I suspect because of the need to do things internally over our networks
> between storage server nodes.
> 
> Hope that makes my thoughts clearer?
> 
> Here is a link to a brief overview of the new Linux system call:
> 
> https://kernelnewbies.org/Linux_4.5#head-6df3d298d8e0afa8e85e1125cc54d5f13b9a0d8c
> 
> Note that block devices or pseudo devices can also implement a copy offload.
> 
> Regards,
> 
> Ric
> 
> >
> >> Worth looking at how we can make use of it.
> >>
> >> thanks!
> >>
> >> Ric
> >>
> >>> ----- Original Message -----
> >>>> From: "Saravanakumar Arumugam" <sarumuga at redhat.com>
> >>>> To: "Gluster Devel" <gluster-devel at gluster.org>
> >>>> Sent: Monday, September 26, 2016 7:18:26 PM
> >>>> Subject: [Gluster-devel] libgfapi zero copy write - application in
> >>>> samba,
> >>>> 	nfs-ganesha
> >>>>
> >>>> Hi,
> >>>>
> >>>> I have carried out "basic" performance measurement with zero copy write
> >>>> APIs.
> >>>> Throughput of zero copy write is 57 MB/sec vs default write 43 MB/sec.
> >>>> ( I have modified Ben England's gfapi_perf_test.c for this. Attached the
> >>>> same
> >>>> for reference )
> >>>>
> >>>> We would like to hear how samba/ nfs-ganesha who are libgfapi users can
> >>>> make
> >>>> use of this.
> >>>> Please provide your comments. Refer attached results.
> >>>>
> >>>> Zero copy in write patch: http://review.gluster.org/#/c/14784/
> >>>>
> >>>> Thanks,
> >>>> Saravana
> >>
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-devel
> 
> 
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
> 