[Gluster-devel] Kluster for Kubernetes (was Announcing Gluster for Container Storage)
joe at julianfamily.org
Fri Aug 24 16:25:59 UTC 2018
On 8/24/18 8:24 AM, Michael Adam wrote:
> On 2018-08-23 at 13:54 -0700, Joe Julian wrote:
>> Personally, I'd like to see the glusterd service replaced by a k8s native controller (named "kluster").
> If you are exclusively interested in gluster for kubernetes
> storage, this might seem the right approach. But I think
> this is much too narrow. The standalone, non-k8s deployments
> still are important and will be for some time.
> So what we've always tried to achieve (this is my personal
> very firm credo, and I think several of the other gluster
> developers are on the same page), is to keep any business
> logic of *how* to manage bricks, create volumes, how to do a
> mount, how to grow, shrink and grow volumes and clusters,
> etc... close to the core gluster project, so that these
> features are usable irrespective of whether gluster is
> used in kubernetes or not.
> The kubernetes components just need to make use of these,
> and so they can stay nicely small, too:
> * The provisioners and csi drivers mainly do api translation
> between k8s and gluster(heketi in the old style) and are
> rather trivial.
> * The operator would implement the logic "when" and "why"
> to invoke the gluster operations, but should imho not
> bother about the "how".
> What can not be implemented with that nice separation
> of responsibilies?
> Thinking about this a bit more, I do actually feel
> more and more that it would be wrong to put all of
> gluster into k8s even if we were only interested
> in k8s. And I'm really curious how you want to do
> that: I think you would have to rewrite more parts
> of how gluster actually works. Currently glusterd
> mananges (spawns) other gluster processes. Clients
> for mounting first connect to glusterd to get the
> volfile and maintain a connection to glusterd
> throughout the whole lifetime of the mount, etc...
> Really interested to hear your thoughts about the above!
> Cheers - Michael
To be clear, I'm not saying throw away glusterd and only do gluster for
Kubernetes and nothing else. That would be silly.
On k8s, a native controller would still need to use some of what
glusterd2 does as libraries, however things like spawning processes
would be relegated to the scheduler. Glusterfsd, glustershd, gsyncd,
etc. would just be pods in the cluster (probably with affinities set for
storage localization). This allows better resource and fault management,
better logging, and better monitoring.
Through Kubernetes custom resource definitions (CRDs), volumes would
declarative and the controller would be responsible for converging the
declaration and the state. I admit this goes opposite to what some
developers in the gluster community have strong feelings about, but the
industry has been moving away from having human managed resources and
toward declarative state engines for good reason. It scales, is less
prone to error, and allows for simpler interfaces.
Volume definitions (vol files, not the CRD) could be stored in
ConfigMaps or Secrets. The client (both glusterfsd and glusterfs) could
be made k8s aware and retrieve these directly or as an easier first step
the cm/secret could be mounted into the pod and the client could load
its vol from a file (the client would need to be altered to reload the
graph if the file changes).
As an aside, the "maintained" connection to glusterd is only true as
long as glusterd always lives at the same IP address. There's a
long-standing bug where the client will never try to find another
glusterd if the one it first connected to ever goes away.
There are still a lot of questions that I don't have answers to. I think
this could be done in a way that's complementary with glusterd and does
not create a bunch of double-work. Most importantly, I think this is
something that could get community buy-in and would suit a need in
Kubernetes that's not well supported at this time.
More information about the Gluster-devel