[Gluster-devel] [Update] GD2 - what's been happening

[Kaushal M]

Hello!
We're restarting regular GD2 updates. This is the first one, and I
expect to send these out every other week.

In the last month, we've identified a few core areas that we need to
focus on. With solutions in place for these, we believe we're ready to
start more deeper integration with glusterfs, that would be requiring
changes in the rest of the code base.

As of release v4.0dev-7, GD2 provided the following features that
would be required for writing/developing Gluster management commands.

- A transaction framework to run orchestrate actions across the cluster
- A rest framework to add new rest endpoints to handle
- A central store based on etcd, to store cluster information
- An auto forming and scaling etcd cluster.

Using these features, we can currently form and run basic GlusterFS
clusters and volumes. While this works, this is not the most usable
yet. Nor is it ready for further integration with rest of GlusterFS.

We've identified and begun working on 3 specific features, that will
make GD2 more usable, and become ready for integration.

- CLI and ReST client packages [1][2][3]
  Aravinda has begun working on creating a CLI application for GD2,
that talks to GD2 using ReST. As a related effort, he's also creating
a GD2 rest-client Go package. With this available, users will be more
easily able to form and use a GD2 cluster. The client package will
also help us write tests using the end-to-end test framework.

- Volume set [3][4][5]
  Prashanth has been working on implementing the volume set
functionality. This is necessary because it allows volumes to be
customized after creation. Xlator options will be read directly from
the xlators, instead of being having a mapping table in GD2. This
means that xlator developers will not need any changes in GD2 to add
new options to their xlator. What this also means is that we will
require some changes to the xlator options table to add some
information that used to be available in the GD options table. We will
be detailing the required changes soon.

- Volgen [6]
  I've been working on a getting a volgen package and framework ready.
We had a very ambitious design earlier [7] involving a dynamic graph
generator with dependency resolution. Work was done on this long back
[8], but was stopped as it turned out to be too complex. The new plan
is much simpler, with graphs order being described using template
files, and having different template files for different volume types.
While this will not be as flexible to the end-user, it is much easier
for developers to add new xlators to graphs. As with volume set,
xlator developers will not need to change anything in GD2. But there
will be changes necessary in the xlators themselves to make the
xlators ready to be automatically picked up and used by GD2. We are
still figuring out the changes needed and finalizing the design. I
will update the wiki with more details on the design, and share
details on the changes required.

In addition to the above, we've had bug fixes to our store and etcd
packages, that make cluster scaling more reliable. Aravinda has also
started initial work on a geo-replication plugin [9], that will help
us develop our plugin infrastructure and be a demo/example of a GD2
plugin for developers.

This concludes the updates since the last update. Thanks for reading.

~kaushal

[1] https://github.com/gluster/glusterd2/pull/334
[2] https://github.com/gluster/glusterd2/pull/337
[3] https://github.com/gluster/glusterd2/pull/335
[4] https://github.com/gluster/glusterd2/pull/339
[5] https://github.com/gluster/glusterd2/pull/345
[6] https://github.com/gluster/glusterd2/pull/351
[7] https://github.com/gluster/glusterd2/wiki/Flexible-Volgen-(Old)#systemd-units-style-1
[8] https://github.com/kshlm/glusterd2-volgen/tree/volgen-systemd-style
[9] https://github.com/gluster/glusterd2/pull/349