[Gluster-devel] GlusterD 2.0 status updates

Mon Sep 7 12:20:22 UTC 2015

Hi Richard,
Thanks a lot for you feedback. I've done my replies inline.

On Sat, Sep 5, 2015 at 5:46 AM, Richard Wareing <rwareing at fb.com> wrote:
> Hey Atin (and the wider community),
>
> This looks interesting, though I have a couple questions:
>
> 1. Language choice - Why the divergence from Python (which I'm no fan of) which is already heavily used in GlusterFS?  It seems a bit strange to me to introduce yet another language into the GlusterFS code base.  Doing this will make things less cohesive, harder to test, make it more difficult for contributors to understand the code base and improve their coding skills to be effective contributors.  I'm a bit concerned we are setting a precedent that development will switch to the new flavor of the day.  If a decision has been made to shift away from Python for the portions of GlusterFS where performance isn't a concern, will the portions currently written in Python be re-written as well?  I also question the wisdom of a language will a shallow developer pool, and a less development process (somewhat of an ironic choice IMHO).
>

One of our aims for GlusterD-2.0 was to switch to a higher level
language. While C is good for solving lower level performance critical
problems, it isn't best suited for the kind of management tasks we
want GlusterD to focus on. The choice of Go over Python as the higher
level language, was mainly driven by the following
- Go is easier to a hang of for a developer coming from a C
background. IMO for a developer new to both Go and Python, it's easier
to start producing working code in Go. The structure and syntax of the
language and the related tools make it easier.
- Go has built into the language support (think goroutines, channels)
for easily implementing the concurrency patterns that we have in the
current GlusterD codebase. This makes it easier for us think about
newer designs based on our understanding of the existing
implementation.
- We have concerns about the concurrency and threading capabilities of
Python. We have faced a lot of problems with doing concurrency and
threading in GlusterD (though this is mostly down to bad design).
Python has known issues with threading, which doesn't give us
confidence as python novices.
- Go has a pretty good standard library (possibly the best standard
library), which provides us with almost everything required. This
reduces the number of dependencies that we pull in.

That's not to say Go doesn't have it's drawbacks. The major drawback
that I see currently with Go is that it doesn't have a way to do
plugins (dynamically load and execute binaries). But there have been
recent developments in this area. Go 1.5 landed support for building
Go packages as dynamic libraries. There is design ready to support
runtime loadability (plugins) ready, which is targeted for inclusion
in Go 1.6. As Go follows a 6 month release cycle, 1.6 is scheduled for
a Feb 2016 release, we need not wait too long for this to land. In the
meantime, we plan to build up the rest of the infrastructure required.

> 2. RPC framework - What's the reasoning behind using Protocol Buffers vs Thrift?  I'm admittedly biased here (since it's heavily used here at FB), however Thrift supports far more languages, has a larger user-base, features better data structure support, has exceptions and has a more open development process (it's an Apache project).  It's mentioned folks are "uncomfortable" with GLib, exactly why?  Has anyone done any latency benchmarks on the serialization/de-serialization to ensure we don't shoot ourselves in the foot by moving away from XDR for brick<->client/gNFSd communication?  The low latency communication between bricks & clients is to me a _critical_ component to GlusterFS's success; adding weight to the protocol or (worse) making it easier to add weight to me is unwise.
>

I'd like to clarify that we are not moving away from sunrpc/xdr for
brick - client communication. This will remain as is. The new RPC
framework is only for communication with GlusterD. We started looking
for an alternative rpc mechanism because Go doesn't have an existing
sunrpc implementation. Go does have an XDR implementation, but there
isn't a stable code generator to generate Go code from .x files.

We are looking for certain features in the RPC framework,
- Language support for C and Go
- Code generation - we didn't want to write message data structures
for each language ourselves
- Support for SSL transports - thought we wouldn't be using this
initially, we would require it later on.
- Support for non-blocking I/O - to better integrate with current
glusterfsd code, which mainly uses non-blocking I/O for network
operations.

At a high level thrift ticks all the boxes. But we initially were
weary of using thrift because of it's usage of Glib in the C
implementation. Glib brings in its own programming patterns and idioms
of which we were a little uncomfortable. But we did revisit Thrift
later (after my original mail), once we felt more comfortable. We
found that the C implementation of thrift is lacking in many features,
the major one among them is that it doesn't have SSL transports. Also,
the C implementation just has a simple blocking server. It doesn't
have either the multi-threaded server or the non-blocking server which
are present in the implementations of other languages.

As we couldn't find any suitable existing RPC frameworks, we went
forward with implementing a simple, easy to use framework ourselves.
We chose protobuf over thrift, as the protobuf-c implementation brings
in less dependencies and works just as well as thrift for
serialization. For the transport, we've tested using libevent for C,
which provides non-blocking I/O and support for encryption. We are in
the process of completing a usable library with this. We've also
started investigating if it's worth using nanomsg for our transports.

But, we are still ready to use thrift, if we can get help implementing
the missing features.

> So far things are moving towards 3-4 languages (Python, C, Go, sprinkle of BASH) and 2 RPC frameworks.  No language or RPC mechanism is perfect, but the proficiency of the coder at the keyboard is _far_ more important.  IMHO we should focus on 1 low level high-performance language (C) and 1 higher level language for other components where high performance isn't required (geo-rep, glusterd etc), as it will encourage higher proficiency in the chosen languages and less fractured knowledge amongst developers.
>

I agree with this. But I believe, with what I've said above, that we
have valid reasons to justify our decisions.

But, even then I'd like to apologize for not having had these
discussions and decisions made transparently. We did say that we'd be
doing GlusterD-2.0 in the open during the Gluster summit, but we
haven't been following our own words. This has been a mistake on our
part, and we will correct it.

> My 2 cents.
>
> Richard
>

Cheers,
Kaushal

>
> ________________________________________
> From: gluster-devel-bounces at gluster.org [gluster-devel-bounces at gluster.org] on behalf of Atin Mukherjee [amukherj at redhat.com]
> Sent: Monday, August 31, 2015 10:04 PM
> To: Gluster Devel
> Subject: [Gluster-devel] GlusterD 2.0 status updates
>
> Here is a quick summary of what we accomplished over last one month:
>
> 1. The skeleton of GlusterD 2.0 codebase is now available @ [1] and the
> same is integrated with gerrithub.
>
> 2. Rest end points for basic commands like volume
> create/start/stop/delete/info/list have been implemented. Needs little
> bit of more polishing to strictly follow the heketi APIs
>
> 3. Team has worked on coming up with a cross language light weight RPC
> framework using pbrpc and the same can be found at [2]. The same also
> has pbcodec package which provides a protobuf based rpc.ClientCodec and
> rpc.ServerCodec that can be used with rpc package in Go's standard library
>
> 4. We also worked on the first cut of volfile generation and its
> integrated in the repository.
>
>
> The plan for next month is as follows:
>
> 1. Focus on the documentation along with publishing the design document
> 2. Unit tests
> 3. Come up with the initial design & a basic prototype for transaction
> framework.
>
> [1] https://github.com/kshlm/glusterd2
> [2] https://github.com/kshlm/pbrpc
>
> Thanks,
> Atin
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://urldefense.proofpoint.com/v1/url?u=http://www.gluster.org/mailman/listinfo/gluster-devel&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=eSXH2j44tGdWnHEbaIk1Tg%3D%3D%0A&m=%2Fqy%2B94CzpNhZn9QOWkfc%2FZkbPJPDiR9uYJNVtG%2BgZPA%3D%0A&s=68e118c111403736815ea0ddf1c756a6c66800a66cbc5e1d14e0586c24ceb695
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel