[Gluster-devel] Possible improvements to Gluster epoll-based event layer

[Krishnan Parthasarathi]

All,

The following is a list of things that come to my mind
whenever I (re)visit our epoll-based event implementation
and compare it with how other open-source server projects
use epoll(7). The following list assumes familiarity with
epoll(7) and how we use it. Feel free to email gluster-devel with
your question. I will answer them to the best of my knowledge.


GlusterFS epoll subsystem
-------------------------

    GlusterFS uses epoll(7) on GNU/Linux systems for receiving notifications on
    sockets.

Current design
--------------

    We use a single epoll fd, which is being 'waited' by a configurable no. of
    threads (via {client,server}.event-threads option). All sockets are configured
    to be NONBLOCKing and added to this epoll fd. We use level-triggered mode with
    EPOLLONESHOT.

 
Open issues
------------

    0. Find a way to profile GlusterFS' epoll-based event system.

        a. to measure its contribution to latency of a FOP, IOPs. 
        b. to measure lock contention among `worker` threads.

    1. Move to edge-triggered mode for benefits mentioned in epoll(7)

    2. Increase `maxevents` handled per `epoll_wait(2)`. This helps in batching
       notifications with one system call.

~kp