[Gluster-devel] FreeBSD port of GlusterFS racks up a lot of CPU usage

[Rick Macklem]

Raghavendra Gowdappa wrote:
> 
> 
> ----- Original Message -----
> > From: "Rick Macklem" <rmacklem at uoguelph.ca>
> > To: "Jeff Darcy" <jdarcy at redhat.com>
> > Cc: "Raghavendra G" <raghavendra at gluster.com>, "freebsd-fs"
> > <freebsd-fs at freebsd.org>, "Hubbard Jordan"
> > <jkh at ixsystems.com>, "Xavier Hernandez" <xhernandez at datalab.es>, "Gluster
> > Devel" <gluster-devel at gluster.org>
> > Sent: Saturday, January 9, 2016 7:29:59 AM
> > Subject: Re: [Gluster-devel] FreeBSD port of GlusterFS racks up a lot of
> > CPU usage
> > 
> > Jeff Darcy wrote:
> > > > > I don't know anything about gluster's poll implementation so I may
> > > > > be totally wrong, but would it be possible to use an eventfd (or a
> > > > > pipe if eventfd is not supported) to signal the need to add more
> > > > > file descriptors to the poll call ?
> > > > >
> > > > >
> > > > > The poll call should listen on this new fd. When we need to change
> > > > > the fd list, we should simply write to the eventfd or pipe from
> > > > > another thread.  This will cause the poll call to return and we will
> > > > > be able to change the fd list without having a short timeout nor
> > > > > having to decide on any trade-off.
> > > > 
> > > >
> > > > Thats a nice idea. Based on my understanding of why timeouts are being
> > > > used, this approach can work.
> > > 
> > > The own-thread code which preceded the current poll implementation did
> > > something similar, using a pipe fd to be woken up for new *outgoing*
> > > messages.  That code still exists, and might provide some insight into
> > > how to do this for the current poll code.
> > I took a look at event-poll.c and found something interesting...
> > - A pipe called "breaker" is already set up by event_pool_new_poll() and
> >   closed by event_pool_destroy_poll(), however it never gets used for
> >   anything.
> 
> I did a check on history, but couldn't find any information on why it was
> removed. Can you send this patch to http://review.gluster.org ? We can
> review and merge the patch over there. If you are not aware, development
> work flow can be found at:
> 
> http://www.gluster.org/community/documentation/index.php/Developers
> 
Actually, the patch turned out to be a flop. Sometimes a fuse mount would end
up with an empty file system with the patch. (I don't know why it was broken,
but maybe the original author tan into issues as well?)

Anyhow, I am now using the 3.7.6 event-poll.c code except that I have increased
the timeout from 1msec->10msec. (Going from 1->5->10 didn't seem to cause a
problem, but I got slower test runs when I increased to 20msec, so I've settled on
10mses. This does reduce the CPU usage when the GlusterFS file systems aren't active.)
I will submit this one line change to your workflow if it continues to test ok.

Thanks for everyone's input, rick

> > 
> > So, I added a few lines of code that writes a byte to it whenever the list
> > of
> > file descriptors is changed and read when poll() returns, if its revents is
> > set.
> > I also changed the timeout to -1 (infinity) and it seems to work for a
> > trivial
> > test.
> > --> Btw, I also noticed the "changed" variable gets set to 1 on a change,
> > but
> >     never reset to 0. I didn't change this, since it looks "racey". (ie. I
> >     think you could easily get a race between a thread that clears it and
> >     one
> >     that adds a new fd.)
> > 
> > A slightly safer version of the patch would set a long (100msec ??) timeout
> > instead
> > of -1.
> > 
> > Anyhow, I've attached the patch in case anyone would like to try it and
> > will
> > create a bug report for this after I've had more time to test it.
> > (I only use a couple of laptops, so my testing will be minimal.)
> > 
> > Thanks for all the help, rick
> > 
> > > _______________________________________________
> > > freebsd-fs at freebsd.org mailing list
> > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
> > > 
> > 
>