[Gluster-devel] FreeBSD port of GlusterFS racks up a lot of CPU usage

[Niels de Vos]

On Wed, Dec 30, 2015 at 09:00:32AM -0500, Rick Macklem wrote:
> Niels de Vos wrote:
> > On Tue, Dec 29, 2015 at 08:12:40PM -0500, Rick Macklem wrote:
> > > Hi,
> > > 
> > > I'm been playing with the FreeBSD port of GlusterFS and it seems
> > > to be working ok. I do notice that the daemons use a lot of CPU,
> > > even when there is nothing to do (no volumes started, etc).
> > > When I ktrace the daemon, I see a small number of nanosleep() and
> > > select() syscalls and lots of poll() syscalls (close to 1000/sec).
> > > 
> > > Looking at libglusterfs/src/event-poll.c, I find:
> > >    ret = poll(ufds, size, 1);
> > > in a loop. The only thing the code seems to do when poll() times
> > > out is a call to event_dispatch_poll_resize().
> > > 
> > > So, is it necessary to call event_dispatch_poll_resize() 1000 times
> > > per second?
> > > Or is there a way to make event_dispatch_poll_resize() return quickly
> > > when there is nothing to do?
> > 
> > I do not think this is critical. A longer timeout should be well
> > acceptable.
> > 
> > > I'm guessing that Linux uses the event-epoll stuff instead of event-poll,
> > > so it wouldn't exhibit this. Is that correct?
> > 
> > Well, both. most (if not all) Linux builds will use event-poll. But,
> > that calls epoll_wait() with a timeout of 1 millisecond as well.
> > 
> Actually, when I look at the 3.7.6 sources in libglusterfs/src/event-epoll.c
> I only find one epoll_wait() at line#668:
>    ret = epoll_wait (event_pool->fd, &event, 1, -1);
> so the timeout never happens in this code.
> (It does have code after the call to handle the timeout case.)
> All it seems to do (if it were to timeout) is adjust the # of threads in the
> event-epoll case, so my hunch is that a timeout isn't needed?

Oh, yes, indeed, my mistake.

> For the event-poll.c case, it calls event_dispatch_poll_size() which looks like
> it might add new file descriptors, so someone more familiar with this code would
> need to decide if the timeout can be disabled (my hunch is no, but I'm not familiar
> with the code).

Maybe Shyam (on CC) knows more how event-poll works. He is definitely
one of the developers that could explain epoll, and I guess he looked
into poll too.

Niels

> > > Thanks for any information on this, rick
> > > ps: I am tempted to just crank the timeout of 1msec up to 10 or 20msec.
> > 
> > Yes, that is probably what I would do too. And have both poll functions
> > use the same timeout, have it defined in libglusterfs/src/event.h. We
> > could make it a configurable option too, but I do not think it is very
> > useful to have.
> > 
> > Could you file a bug and/or send a patch for this?
> > 
> I will try bumping the timeout up in event-poll.c and if it seems to reduce
> CPU usage and not cause any obvious grief, I will file a bug report.
> 
> Thanks for your help with this, rick
> 
> > Thanks,
> > Niels
> > 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160107/9fe9cc33/attachment.sig>