[Gluster-devel] GlusterFS (network) messaging layer - possible next steps

[Krishnan Parthasarathi]

All,

I have been exploring different ways of addressing GlusterFS messaging layer
recently. My motivations for this are to find a library/framework that provides
messaging capabilities over the network with the following characteristics,

  - Better expressibility - provide useful abstractions for messaging

  - Maintainability - resulting code handling application and messaging logic
                      should be easy to reason with 
  - Performance 

The current messaging layer is built on top of non-blocking sockets and a
callback based model. While this does the trick for problems described in C10K[1]
it is not the most convenient abstraction. This is where nanomsg[2] comes in.

What is nanomsg?

nanomsg is a socket library that provides several common communication
patterns. It aims to make the networking layer fast, scalable, and easy to use.

Following are the communication patterns that come out of the box,

  PAIR - simple one-to-one communication

  BUS - simple many-to-many communication

  REQREP - allows to build clusters of stateless services to process user
           requests

  PUBSUB - distributes messages to large sets of interested subscribers

  PIPELINE - aggregates messages from multiple sources and load balances them
             among many destinations

  SURVEY - allows to query state of multiple applications in a single go

(source: http://nanomsg.org)


What does this mean to Gluster?

nanomsg's PUBSUB and SURVEY pattern appear in Gluster too often to miss.
PUBSUB will fit in code that sends out a multicast notifications to interested
remote entities. e.g notifications for volume configuration changes.
SURVEY will fit in where we send out the same request to many processes and
act on their responses to proceed. e.g cluster translators performing the same
operation (i.e, same RPC messages) on different bricks.

What next?

nanomsg wouldn't be an easy drop in replacement to our existing messaging
infrastructure. We need to understand how our code would be structured if we
decide to use nanomsg. I am considering the Go implementation of nanomsg
(protocol) called mangos[3] for inter-GlusterD communication in GlusterD 2.0.
I will be sharing our experience when we get there. Meanwhile, please share
what you think about this.


[1] - http://www.kegel.com/c10k.html
[2] - http://nanomsg.org/
[3] - https://github.com/gdamore/mangos

thanks,
kp