[heketi-devel] Working Heketi

[Jose A. Rivera]

Hey Christopher,

On Mon, Aug 28, 2017 at 12:37 AM, Christopher Schmidt
<fakod666 at gmail.com> wrote:
> Hi,
>
> sorry for the confusion.
> I opened this thread, to find an overall working configuration.
>
> My setup is as follows:
> - Container Linux (CoreOS)
> - No local installation (except Kubelet, everything (!) else runs in
> Containers)
> - I am using this deploy scripts for GlusterFS
> https://github.com/gluster/gluster-kubernetes/tree/master/deploy/kube-templates
> - I've created my own containers with
> https://github.com/gluster/gluster-containers/tree/master/CentOS (so its
> Gluster 1.10.5)
> - I am using the original container from docker hub heketi/heketi:5 (with a
> secret as DB backup, not the gk-deploy script)
> - Kubernetes v1.6.8_coreos.0 with 3 masters and 6 minions, gluster pods on
> all minions (3 x 50GB partitions for Gluster on each node, rbac enabled)
> - for network policies I use Canal (deployed as pods as well)
>
> Issues:
> - I've learned that Kubernetes 1.7.x and GlusterFS is broken, since 1.7
> requires GlusterFS > 1.10, right?

I imagine you mean GlusterFS 3.10. :) And no, the issue is because the
current GlusterFS container uses systemd, which conflicts with the new
shared PIC namespace feature in recent versions of Kube and Docker:
https://github.com/kubernetes/kubernetes/issues/48937#issuecomment-319957314

> - I was testing K8s 1.5.x with Heketi/GlusterFS and it was fine for me (but
> really only small manual tests a couple of months ago)
> - With my configuration (see above) no version of Gluster or Heketi works
> with K8s 1.6.x with dynamic provisioning.
>
> However, this is not the usual GlusterFS setup you are testing. So I am sure
> I do something wrong. What is noticeable is that creating a volume takes a
> couple of minutes (depending on the volume type) and Heketi has some issues
> with that.
> So what can I do? Where to start testing/changing my configuration?

Nothing looks immediately wrong to me. Have you tried the same
container images that we use in gluster-kubernetes? The only other
difference I note is that you're using Canal whereas out test setups
use weave-net.

--Jose

> regards Christopher
>
>
> Humble Chirammal <hchiramm at redhat.com> schrieb am Di., 22. Aug. 2017 um
> 09:51 Uhr:
>>
>> On Tue, Aug 22, 2017 at 1:14 PM, Michael Adam <obnox at samba.org> wrote:
>>>
>>> On 2017-08-22 at 07:00 +0000, Christopher Schmidt wrote:
>>> > Hi guys,
>>> >
>>> > I know it's difficult to support various installations and combinations
>>> > of
>>> > tools.
>>> > But I'll give some Kubernetes training and talks in September/October.
>>> > So what should I recommend? So far I didn't find a working GlusterFS
>>> > Heketi, neither with Kubernetes 1.6.x nor 1.7.x
>>>
>>> Hi Christopher,
>>>
>>> This statement seems surprising to me.
>>> The vagrant environment and the complex tests
>>> in the gluster-kubernetes repo
>>> (github.com/gluster/gluster-kubernetes)
>>> prove that heketi and gluster work in kube 1.6.
>>>
>>> Can you give some more details on what's going wrong?
>>
>>
>> There is another thread opened by Christopher
>> http://lists.gluster.org/pipermail/heketi-devel/2017-August/000166.html
>>
>>
>>>
>>>
>>
>>
>>
>>
>> --
>> Cheers,
>> Humble
>>
>>
>
> _______________________________________________
> heketi-devel mailing list
> heketi-devel at gluster.org
> http://lists.gluster.org/mailman/listinfo/heketi-devel
>