[Gluster-devel] Brick port allocation by GlusterD

[Atin Mukherjee]

As of now GlusterD maintains its own portmap table and is responsible
for allocating ports for the services like brick process, snapd. The
flow with the portmap goes like this:

When a volume start is triggered, GlusterD checks whether the brick has
been already assigned a port earlier, if so then the same port is been
passed to the brick process, otherwise GlusterD picks up a free port
from the portmap table.

Now say if the node reboots, then GlusterD first starts the daemons
followed by the brick process. Now given brick process tries to bind to
the same persisted port there is no guarantee that the same port
wouldn't be consumed by some other application (be it from gluster
application or not) and this is exactly what we noticed in one of the BZ
[1]. We hit this very frequently when number of brick processes go high.

I think bringing up a process binding to a persisted port is not a good
idea since its prone to fail considering processes (clients) contend for
the same ports.

I've sent a patch [2] which follows the same approach what snapd
currently does for port. the ports will continue to get persisted but on
every brick restart a fresh port will be allocated for the brick. The
only reason of persisting the brick is that the same will be attempted
to be removed from the portmap in case the brick hasn't been shutdown
gracefully and a pmap_registry_remove () hasn't been invoked.

We'd also need another patch [3] to get this work as currently we don't
mark the port as free in pmap_registry_remove.

Please note that [2] doesn't fully eliminate the probability of other
process stealing the port allocated by GlusterD as there is still a
small time window where GlusterD allocates the port and brick process
binds to it.

As a complete/long term solution we think that GlusterD has to give up
managing the port allocation and the same has to be done by brick/daemon
process and GlusterD will be doing a book keeping of those ports.

Your comments/suggestion is more than welcome here :)

~Atin

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1322805
[2] http://review.gluster.org/#/c/13865/
[3] http://review.gluster.org/#/c/10785/