[Gluster-devel] An update on GlusterD-2.0

[Deepak Shetty]

On Wed, Jun 17, 2015 at 8:35 PM, Kaushal M <kshlmster at gmail.com> wrote:

> At the Gluster Summit, everyone agreed that GlusterD should be the
> first component to be targeted for GlusterFS-4.0. We had good
> discussions on what GlusterD lacks currently and what is required for
> GlusterD-2.0. KP and I had promised to send an update to mailing list
> summarizing the discussions, and this is it.
>
> Along with the summary, we'll also be discussing our plans for Manila
> integration and how we are planning to do it with GlusterD-2.0.
>
> ## Gluster Summit Summary
> In the summit, KP and I presented a talk titled *GLUSTERD NEXT GEN
> OR: HOW WE CAN STOP BEING A ROADBLOCK AND INSTEAD BECOME THE ROAD TO
> SCALABILITY*. The slides can be viewed at [1][1]. There is no video
> recording of it unfortunately.
>
> The summary of the presentation is below.
>
> ### Problems
> GlusterD, as it is currently, is not scalable which will prevent
> GlusterFS as whole from scaling. The scaling issues can be classified
> into,
> - Node scalability
> - Maintenance scalability
> - Integration scalability
> - Contribution scalability
>
> #### Node scalability
> This is the general scalability we all think about, scale in terms of
> node/machine/clients. GlusterD scalability is help back in this
> because of the store, transaction and synchronization mechanisms used
> in GlusterD.
>
> #### Maintenance scalability
> Maintenance scalability is to with the problems we as GlusterD
> maintainers faced. This is mainly related to the huge, monolithic code
> base of the current GlusterD, which makes splitting of maintenance and
> ownership tasks hard.
>
> #### Integration scalability
> Integration scalability can split into internal and external integration.
> Internal integration is the integration dealing with new features
> being added to GlusterFS. Every new feature being added needs to touch
> GlusterD or CLI in some way. This has generally been done with
> copy/paste coding which has added to the maintenence overhead.
> External integration is the integration of Gluster with other projects
> or other projects with Gluster. This is hurt due to the
> non-availability of a proper API for GlusterD operations. All
> interaction with GlusterD currently happens only via the CLI. And the
> output we provide is generally inconsistent to be programatically
> parsed.
>
> #### Contribution scalability
> Getting new contributors for GlusterD is hard. New contributors are
> put off because GlusterD is hard to understand, and because there
> isn't enough documentation.
>
>
> So GlusterD-2.0 will need to
> - be scalable to 1000s of nodes
> - have lower maintenance costs
> - enable better external and internal integrations
> - make it easier for newer contributors
>
> ### Design characteristics for GlusterD-2.0
> For GlusterD-2.0 to satisfy the above listed requirements and solve
> the problems listed before, it should have the following
> characteristics.
>

One other design characteristic I can think of is to have GlusterD-2.0
be able to make best use of Nodes based on the Node topology, not sure
if this is covered by any other feature below ?

IOW, admin should be able to provide a hierarchy of how nodes are
provisioned
(rack, site, zone, region, geo etc) and glusterd should use this info while
provisioning bricks for volume creation



> #### Centralized store
> This should help with our numbers scalability issues. GlusterD-2.0
> will be built around a centralized store. This means, instead of the
> GlusterD volume and peer information being persisted on all nodes, it
> will be stored only on a subset of the nodes in a trusted storage
> pool.
>
> We are looking at solutions like etcd and consul, both of which
> provide a distributed key/value store (and some more useful features
> on top), to provide the centralized store. The transaction mechanism
> for Gluster operations will be built around this centralized store.
>
> Moving to an external store provider and a transaction framework built
> around it will reduce a lot of the complexity in GlusterD.
>
> #### Plugins
> This mainly for the maintainability and internal integration aspects.
> GlusterD-2.0 will have a plug-gable, modular design. We expect all the
> commands of GlusterD to be implemented as plugins. Certain other parts
> of GlusterD, including things like volgen, volume-set, rest api, etc.
> This will allow new features to be integrated in to GlusterD easily.
> The code for these plugins is expected to live with the feature, and
> not in GlusterD.
>
> Doing a plugin design requires the defining of well defined plugin
> interfaces to not just plug into GlusterD-2.0, but also interact with
> GlusterD well. I'll be posting a document on this to the mailing list
> soon.
>
> The GlusterD maintainers/team will be doing the implementation of a
> few core commands required including volume create, start, stop,
> delete, expand, shrink and set/reset options.
>
> #### Rest API
> To provide better a better method for external projects to hook on to.
> GlusterD-2.0 will provide a HTTP REST API, with JSON output and proper
> API versioning. We will be looking for inspiration from other storage
> solutions to define the APIs presented to the users and admins.
>
> #### Logging
> To help with debuggability, GlusterD-2.0 will provide better logs that
> should allow easier tracking of operations across machines. We are
> currently looking at the oVirt logging mechanism, which tags all logs
> of an operation with a unique id, for inspiration.
>
> #### Documentation
> We hope to do a much better job of documenting the technical details
> of GlusterD-2.0, starting with design documents. We still don't have
>
>
> ## Manila and GlusterD-2.0
>
> Manila is the File System as a Service component of OpenStack. Manila
> has two GlusterFS providers neither of which currently support the
> full Manila API. One uses a sub-directory over NFS approach and the
> other uses a volume based approach. Both of them require specific
> features from GlusterFS to be completely functional by the next
> OpenStack release, which is expected for October 2015. Csaba listed
> these requirements at [2][2]. This lists two possible approaches,
> 1. Implement directory level operations (required by the sub-directory
> approach)
> 2. Implement intelligent volume creation (IVC) (required for the
> volume based approach)
> (Please read the linked mail-thread for more context)
>
>
I have 2 things to say here :

1) Manila uses GlusterFS SSL auth mode, so is there any discussion happening
around adding support in GlusterD-2.0 for managing SSL certificates ?

For eg: Other (eg Netapp) storage arrays do provide basic digital cert mgmt
support for server and client auth.

I feel it would be good to have glusterd provide support  to generate,
install, manage
self-signed and/or CA signed digital certs.

This will not just help Manila usecase, but also help GlusterFS provide a
complete
solution for digital cert management which will aid the SSL auth support
feature of GlusterFS

In fact, this can be done in a modular way where in the default implm will
be that of GlusterD
cert module, but optionally one can use openstack Barbican service too as a
cert mgmt service

2) In the manila community there was some intense discussion on the
definition of multi-tenancy
when applied to storage and network level multi-tenancy was voted as being
very important in
Manila cloud usecase. Do we have any thoughts on how GlusterD-2.0 /
GlusterFS-4.0 can
look at providing tenant separation at network layer ?

thanx,
deepak
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20150625/c0c47bb7/attachment.html>