[Gluster-devel] GlusterFS (network) messaging layer - possible next steps

[Jeff Darcy]

> nanomsg wouldn't be an easy drop in replacement to our existing messaging
> infrastructure. We need to understand how our code would be structured if we
> decide to use nanomsg. I am considering the Go implementation of nanomsg
> (protocol) called mangos[3] for inter-GlusterD communication in GlusterD 2.0.
> I will be sharing our experience when we get there. Meanwhile, please share
> what you think about this.

As I've written elsewhere, I believe it would be a good thing to get away
from our current (XDR) serialization format/method.  Replacing the next
level up - managing connections, detecting when they break, etc. - is a
bit trickier.  One issue there is what to do about RDMA.  Another is that
the rest of our code makes a lot of assumptions (many implicit) about
things like event ordering and parallelism levels.  It has proven fragile
even with modest changes in this area (e.g. SSL/TLS), and would probably
be even more so in the face of greater change.

That said, I did actually look into nanomsg a while ago (mostly for NSR
internal communication).  It did look like one of the better alternatives
in its space, and might be a good fit for GlusterD specifically.  There's
no reason GlusterD needs to use the same networking infrastructure as the
I/O path, just like there's no reason it needs to use the same translator
infrastructure.  Since it does reduce our own burden on maintaining the
current networking code for both I/O path and GlusterD usage (which are
quite different) and does offer some more powerful abstractions that we
could use, I would definitely encourage further investigation in this
area.