[heketi-devel] [heketi] Pre-existing GlusterFS cluster

[Jose A. Rivera]

Poking at this again. This time putting forth the idea of having two
heketi-controlled clusters overlapping in nodes (but not in devices,
of course). This facilitates, for example, a node that has both HDDs
and SSD, and being able to group the HDDs in come cluster and SSDs in
another so that users could have some ability to select the underlying
storage type if it matters to them.

--Jose

On Wed, Feb 8, 2017 at 11:51 AM, Jose A. Rivera <jarrpa at redhat.com> wrote:
> Ping :)
>
> On Thu, Jan 26, 2017 at 8:59 AM, Jose A. Rivera <jarrpa at redhat.com> wrote:
>> Sure thing!
>>
>> The ask is for an OpenShift use case where I want to create a
>> GlusterFS volume to store the local Docker registry before any
>> containers are running. I was thinking of doing this by running
>> Gluster natively on the OpenShift nodes, outside of containers,
>> creating a cluster of them, then selecting a directory on each node to
>> serve as bricks for the volume. The idea here is that I would still
>> want to deploy heketi in a container later on, and just use these same
>> nodes in the topology file. heketi would still need to be given
>> dedicated storage devices on each node.
>>
>> As far as the algorithm, I figure it should be something like:
>>
>> For adding a node of a pre-existing cluster, watch for the return
>> code/value from the exec of gluster peer probe and if it says peer
>> already in list we return success. If for some reason the pre-existing
>> cluster only overlaps on a subset of nodes with the heketi cluster,
>> gluster can handle this.
>>
>> In the inverse, when you remove a node from heketi, watch for a
>> message that peer cannot be detached because it has bricks and remove
>> the node from heketi anyway. heketi already does its own checks to see
>> if a volume is on a particular node, so we can't get to the point
>> where a heketi-managed brick is still extant on a heketi-managed node
>> unless something goes really wrong (and then we have to resort to the
>> backend command line anyway, I'd imagine?).
>>
>> Feedback, concerns, or flames welcome. :)
>> --Jose
>>
>> On Wed, Jan 25, 2017 at 10:55 PM, Luis Pabon <lpabon at gmail.com> wrote:
>>> I think that people keep asking because they want to continue what they
>>> know.
>>>
>>> I think it would work better if you provide a set of requirements and
>>> preconditions and the algorithm which satisfies what you would like to do.
>>> Provide it here, and let's discuss it.  No code needed.
>>>
>>> - Luis
>>>
>>> On Wed, Jan 25, 2017 at 7:29 PM, Jose A. Rivera <jarrpa at redhat.com> wrote:
>>>>
>>>> On Wed, Jan 25, 2017 at 2:16 PM, Luis Pabon <lpabon at gmail.com> wrote:
>>>> > Hi José,
>>>> >   This has been asked for many many times.  Heketi was designed to "rule
>>>> > them all".  Heketi was never designed for systems that have been setup
>>>> > already because the permutations of possibilities of configurations
>>>> > could be
>>>> > extensive to figure out how to manage.  It is like creating a Ceph Rados
>>>> > system by yourself, then asking the pool manager to figure out what you
>>>> > did.
>>>> > If instead Ceph is viewed as a collection of the access+pool+storage and
>>>> > not
>>>> > as individual parts, then it all works well and is predictable.  In the
>>>> > same
>>>> > way, it should not be viewed as Heketi managing GlusterFS, but
>>>> > Heketi/GlusterFS instead.  Once this view is accepted (which is what
>>>> > users
>>>> > want, but old school gluster users have a hard time with), then what
>>>> > Heketi
>>>> > currently does makes perfect sense.
>>>> >
>>>> > So, back to the question, no, Heketi does not and will never manage such
>>>> > a
>>>> > model.  Any software that manages such a configuration would be hard to
>>>> > productize and guarantee.  Can you make a hack that does it? Maybe, but
>>>> > reliability and simplicity is what Heketi is after.
>>>> >
>>>> > Hope this answers your question.
>>>>
>>>> I know this has been asked more than once and I believe this keeps
>>>> being asked for because the above is still an unsatisfactory answer.
>>>> :) Already we are seeing new users asking for maintenance features
>>>> that would be perfectly possible with Gluster but which are currently
>>>> out of reach when going with heketi. I think focusing too hard on
>>>> "simplicity" will quickly become limiting to heketi's desirability. It
>>>> would seem to make more sense to go with a mindset of a tailored
>>>> experience, with the ability to go in deeper if desired.
>>>>
>>>> There doesn't seem to be anything technically complicated about the
>>>> idea that heketi could tolerate a peer probe coming back already
>>>> satisfied, or that a node is removed without removing it from the peer
>>>> list. I don't see how this would prove to be dangerous as long as we
>>>> maintain the understanding that you are not to go in on the backend to
>>>> mess with anything heketi is actively managing. This seems like
>>>> something we could easily test, make reliable, and productize.
>>>>
>>>> --Jose
>>>>
>>>> > - Luis
>>>> >
>>>> > On Tue, Jan 24, 2017 at 12:24 PM, Jose A. Rivera <jarrpa at redhat.com>
>>>> > wrote:
>>>> >>
>>>> >> Hey Luis, et al.,
>>>> >>
>>>> >> I talked to Ashiq about $SUBJECT, and he raised some concerns.
>>>> >> Apparently heketi can not load/import nodes that are already part of a
>>>> >> Gluster cluster? E.g. if I have an existing cluster with all the nodes
>>>> >> already peer probed, heketi will try to redo the probe and then fail
>>>> >> when it comes back already in peer list? This seems odd to me, but if
>>>> >> so sounds like a relatively easy thing to change. Thoughts?
>>>> >>
>>>> >> --Jose
>>>> >
>>>> >
>>>
>>>