[Gluster-devel] Reduce memcpy in glfs read and write

Wed Jun 22 00:34:59 UTC 2016

On Wed, Jun 22, 2016 at 5:50 AM, Sachin Pandit <spandit at commvault.com>
wrote:

> Hey Pranith, I am good, I hope you are doing good too.
>
> Please find the comments inline.
>
>
>
> *From:* Pranith Kumar Karampuri [mailto:pkarampu at redhat.com]
> *Sent:* Tuesday, June 21, 2016 5:58 AM
> *To:* Sachin Pandit <spandit at commvault.com>
> *Cc:* gluster-devel at gluster.org
> *Subject:* Re: [Gluster-devel] Reduce memcpy in glfs read and write
>
>
>
> Hey!!
>
>        Hope you are doing good. I took a look at the bt. So when flush
> comes write-behind has to flush all the writes down. I see the following
> frame hung in iob_unref:
> Thread 7 (Thread 0x7fa601a30700 (LWP 16218)):
> #0  0x00007fa60cc55225 in pthread_spin_lock () from
> /lib64/libpthread.so.0  <<---- Does it always hang there?
>
> ---------------------------------
>
> >>It does always hang here.
>
> ---------------------------------
> #1  0x00007fa60e1f373e in iobref_unref (iobref=0x19dc7e0) at iobuf.c:907
> #2  0x00007fa60e246fb2 in args_wipe (args=0x19e70ec) at
> default-args.c:1593
> #3  0x00007fa60e1ea534 in call_stub_wipe_args (stub=0x19e709c) at
> call-stub.c:2466
> #4  0x00007fa60e1ea5de in call_stub_destroy (stub=0x19e709c) at
> call-stub.c:2482
>
> Is this on top of master branch? It seems like we missed an unlock of the
> spin-lock or the iobref has junk value which gives the feeling that it is
> in locked state (May be double free?). Do you have any extra patches you
> have in your repo which make changes in iobuf?
>
> ----------------------------------
>
> >>I have implemented a method to reduce memcpy in libgfapi (My patch is on
> top of master branch), by making use of buffer from iobuf pool and passing
> the buffer to application. However, I have not made any changes in iobuf
> core feature. I don’t  think double free is happening anywhere in the code
> (I did check this using logs)
>
>
>
>      Method that I have implemented:
>
> 1)      Application asks for a buffer of specific size, and the buffer is
> allocated from the iobuf pool.
>
> 2)      Buffer is passed on to application, and the application writes
> the data into that buffer.
>
> 3)      Buffer with data in it is passed from application to libgfapi and
> the underlying translators (no memcpy in glfs_write)
>
>
>
> I have couple of questions, and observations:
>
>
>
> Observations:
>
> ------------------
>
> 1)      For every write if I get a fresh buffer then I don’t see any
> problem. All the writes are going through.
>
> 2)      If I try to make use of buffer for consecutive writes, then I am
> seeing the hang in flush.
>
>
>
> Question1: Is it fine if I reuse the buffer for consecutive writes??
>
> Question2: Is it always ensured that the data is written to the file when
> I get a response from syncop_writev.
>

Will it be possible to share the patch on master and a test program which
can recreate this issue?

>
>
> Thanks,
>
> Sachin Pandit.
>
>
>
> ----------------------------------
>
>
>
> On Tue, Jun 21, 2016 at 4:07 AM, Sachin Pandit <spandit at commvault.com>
> wrote:
>
> Hi all,
>
>
>
> I bid adieu to you all with the hope of crossing path again, and the time
> has come rather quickly. It feels great to work on GlusterFS again.
>
>
>
> Currently we are trying to write data backed up by Commvault Simpana to
> glusterfs volume (Disperse volume). To improve the performance, I have
> implemented the proposal put forward my Rafi  K C [1]. I have some
> questions regarding libgfapi and iobuf pool.
>
>
>
> To reduce an extra level of copy in glfs read and write, I have
> implemented few APIs to request a buffer (similar to the one represented in
>  [1]) from iobuf pool which can be used by the application to write data
> to. With this implementation, when I try to reuse the buffer for
> consecutive writes, I could see a hang in syncop_flush of glfs_close (BT of
> the hang can be found in [2]). I wanted to know if reusing the buffer is
> recommended. If not, do we need to request buffer for each writes?
>
>
>
> Setup : Distributed-Disperse ( 4 * (2+1)). Bricks scattered over 3 nodes.
>
>
>
> [1]
> http://www.gluster.org/pipermail/gluster-devel/2015-February/043966.html
>
> [2] Attached file -  bt.txt
>
>
>
> Thanks & Regards,
>
> Sachin Pandit.
>
>
>
> ***************************Legal Disclaimer***************************
>
> "This communication may contain confidential and privileged material for the
>
> sole use of the intended recipient. Any unauthorized review, use or distribution
>
> by others is strictly prohibited. If you have received the message by mistake,
>
> please advise the sender by reply email and delete the message. Thank you."
>
> **********************************************************************
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
>
>
>
> --
>
> Pranith
>
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material for the
> sole use of the intended recipient. Any unauthorized review, use or distribution
> by others is strictly prohibited. If you have received the message by mistake,
> please advise the sender by reply email and delete the message. Thank you."
> **********************************************************************
>
>

-- 
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160622/db1130fa/attachment-0001.html>