[heketi-devel] heketi-devel Digest, Vol 6, Issue 3
Mohamed Ashiq Liyazudeen
mliyazud at redhat.com
Fri Mar 3 14:43:44 UTC 2017
Pavel,
This is good stuff. I will definitely give it a try. So this is working as expected? are you blocked anywhere so that we can see what we can do to make it better?
It would be really good if you can do a blog or send a .md to gluster-kubernetes on what you have done so that other people can also see the what all possible ways to deploy.
--
Ashiq
----- Original Message -----
From: "Pavel Kutishchev" <pavel.kutishchev at gmail.com>
To: "Mohamed Ashiq Liyazudeen" <mliyazud at redhat.com>
Cc: heketi-devel at gluster.org
Sent: Friday, March 3, 2017 7:35:57 PM
Subject: Re: [heketi-devel] heketi-devel Digest, Vol 6, Issue 3
Ashiq,
In that case you have to have nodeselector in your deployment, and
hostname defined inside container spec.
As follows:
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: glusterfs-user-cluster-sas-108
spec:
template:
metadata:
name: glusterfs-user-cluster-sas-108
labels:
name: glusterfs-sas
app: glusterfs-user-cluster-sas-108
spec:
nodeSelector:
kubernetes.io/hostname: server37
hostname: glusterfs-sas-node-server37
subdomain: sas
and service for this deployment:
apiVersion: v1
kind: Service
metadata:
name: sas
spec:
selector:
name: glusterfs-sas
clusterIP: None
ports:
- name: fake # Actually, no port is needed.
port: 1
Moreover, you should have kubeDNS pod IP in your resolv.conf on the
host, otherwise kubelet wont be able to mount volume inside pod using
generated endpoint because kubelet pod doesn't know how to resolve A
records which contains kubeDNS.
Btw, this won't work with DaemonSet.
Thank you,
Pavel K.
3/3/17 3:46 PM, Mohamed Ashiq Liyazudeen пишет:
> I agree with the first point. Just that if the container goes down it
> should come in the same node as the glusterd config saved in the host
> will expect gluster on the node will be of same hostname.
>
> Then this will work. I don't have any questions.
>
> On Mar 3, 2017 6:31 PM, Pavel Kutishchev <pavel.kutishchev at gmail.com>
> wrote:
>
> Hey Ashiq,
> 1. hostname of container can be permanent and resolved thru service.
> https://kubernetes.io/docs/admin/dns/ - *A Records and hostname
> based on Pod’s hostname and subdomain fields paragraph*
> 2. part of the spec below should answer question regarding
> persistent folders:
> volumeMounts:
> - name: dev
> mountPath: /dev
> - name: system-etc
> mountPath: /etc/glusterd
> - name: system-config
> mountPath: /var/lib/glusterd
> - name: heketi-ssh
> mountPath: /root/.ssh
> - name: heketi-var
> mountPath: /var/lib/heketi
> - name: workaround-1
> mountPath: /run
> - name: system-sys
> mountPath: /sys/fs/cgroup
> - name: selinux
> mountPath: /etc/selinux
> - name: selinux-lib
> mountPath: /usr/lib/selinux
> volumes:
> - name: dev
> hostPath:
> path: /dev
> - name: system-etc
> hostPath:
> path: /etc/glusterd-user-sas
> - name: system-config
> hostPath:
> path: /var/lib/glusterd-user-sas
> - name: heketi-ssh
> hostPath:
> path: /etc/glusterd-user/ssh
> - name: heketi-var
> hostPath:
> path: /var/lib/heketi-user-sas
> - name: workaround-1
> hostPath:
> path: /var/sds/sas
> - name: system-sys
> hostPath:
> path: /sys/fs/cgroup
> - name: selinux
> hostPath:
> path: /etc/selinux
> - name: selinux-lib
> hostPath:
> path: /usr/lib/selinux
>
> 3. Udev work properly, there is no issues to handle udev events by
> lvm socket from different clusters.
> 4. Yes, it works correctly as you won't use the same devices at
> the same time :)
> 5. With host network it wont work, you need to use
> docker/kubernetes network
>
> and then you can manage two clusters of gluster using heketi.
>
> Please let me know in case of any questions.
>
> Thank you,
> Pavel K.
>
>
> 3/3/17 2:43 PM, Mohamed Ashiq Liyazudeen пишет:
>
> Hi Pavel,
>
> If you have configured kube to run two instance of gluster container on same node and taken care of:
>
> 1) hostname of the gluster container should not change on restarts or any docker issue. Also The container on re-spawn or restart Kube should not start in a different node.
> 2) the gluster and heketi configuration should be persisted.
> * /var/lib/heketi
> * /var/lib/glusterd
> * /var/log/glusterfs
> * /etc/glusterfs
> 3) honestly, I have not done much testing on /dev of host bind mounted on two containers and doing lvcreate. How the Udev will handle these cases I am not sure. Let us say this works properly.
> 4) Both privileged container sharing /dev has no issues.
>
> Then Yeah you can do the same now with heketi and peer probe with hostname of container. Heketi will consider as two different cluster and Hekeit will use only the devices which are mentioned in the topology file. Let me know If you need any help. I not saying this will work we have not done this. Just for the sake of Creativity, I would like to see what happens :). Good luck.
>
> --
> Ashiq
>
> ----- Original Message -----
> From: "Pavel Kutishchev"<pavel.kutishchev at gmail.com> <mailto:pavel.kutishchev at gmail.com>
> To:heketi-devel at gluster.org <mailto:heketi-devel at gluster.org>
> Sent: Friday, March 3, 2017 5:42:36 PM
> Subject: Re: [heketi-devel] heketi-devel Digest, Vol 6, Issue 3
>
> Hi Ashiq,
>
> Actually i'm doing the same as described in this issue, at one physical
> node we have two glusterFS clusters for hdd and ssd, using kubernetes.
> There is no problems to have two clusters on the same node.
> If so i can help and clarify how we're doing this in separate thread.
>
> Thank you,
> Pavel K.
>
>
> 3/3/17 2:00 PM,heketi-devel-request at gluster.org
> <mailto:heketi-devel-request at gluster.org> пишет:
>
> Send heketi-devel mailing list submissions to
> heketi-devel at gluster.org <mailto:heketi-devel at gluster.org>
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://lists.gluster.org/mailman/listinfo/heketi-devel
> or, via email, send a message with subject or body 'help' to
> heketi-devel-request at gluster.org
> <mailto:heketi-devel-request at gluster.org>
>
> You can reach the person managing the list at
> heketi-devel-owner at gluster.org
> <mailto:heketi-devel-owner at gluster.org>
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of heketi-devel digest..."
>
>
> Today's Topics:
>
> 1. Re: [heketi] Pre-existing GlusterFS cluster
> (Mohamed Ashiq Liyazudeen)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 3 Mar 2017 05:07:40 -0500 (EST)
> From: Mohamed Ashiq Liyazudeen<mliyazud at redhat.com> <mailto:mliyazud at redhat.com>
> To: Raghavendra Talur<rtalur at redhat.com> <mailto:rtalur at redhat.com>
> Cc:heketi-devel at gluster.org <mailto:heketi-devel at gluster.org>
> Subject: Re: [heketi-devel] [heketi] Pre-existing GlusterFS cluster
> Message-ID:
> <878669976.31135169.1488535660381.JavaMail.zimbra at redhat.com>
> <mailto:878669976.31135169.1488535660381.JavaMail.zimbra at redhat.com>
> Content-Type: text/plain; charset=utf-8
>
> Hi,
>
> We can do something like this. Not for now but as a support. Let me know if this helps?
>
> https://github.com/heketi/heketi/issues/700
>
> --
> Ashiq
>
> ----- Original Message -----
> From: "Raghavendra Talur"<rtalur at redhat.com> <mailto:rtalur at redhat.com>
> To: "Jose A. Rivera"<jarrpa at redhat.com> <mailto:jarrpa at redhat.com>
> Cc:heketi-devel at gluster.org <mailto:heketi-devel at gluster.org>
> Sent: Thursday, March 2, 2017 2:19:11 PM
> Subject: Re: [heketi-devel] [heketi] Pre-existing GlusterFS cluster
>
> On Tue, Feb 28, 2017 at 10:09 PM, Jose A. Rivera<jarrpa at redhat.com> <mailto:jarrpa at redhat.com> wrote:
>
> Poking at this again. This time putting forth the idea of having two
> heketi-controlled clusters overlapping in nodes (but not in devices,
> of course). This facilitates, for example, a node that has both HDDs
> and SSD, and being able to group the HDDs in come cluster and SSDs in
> another so that users could have some ability to select the underlying
> storage type if it matters to them.
>
> The same node being part of two clusters is not possible with Gluster
> architecture. We might be able to achieve that only if we figure out
> how to run Gluster containers with using host networking.
> I do want to see a solution for the problem mentioned in the subject
> and like the previous algorithm given by Jose.
>
>
> --Jose
>
> On Wed, Feb 8, 2017 at 11:51 AM, Jose A. Rivera<jarrpa at redhat.com> <mailto:jarrpa at redhat.com> wrote:
>
> Ping :)
>
> On Thu, Jan 26, 2017 at 8:59 AM, Jose A. Rivera<jarrpa at redhat.com> <mailto:jarrpa at redhat.com> wrote:
>
> Sure thing!
>
> The ask is for an OpenShift use case where I want to create a
> GlusterFS volume to store the local Docker registry before any
> containers are running. I was thinking of doing this by running
> Gluster natively on the OpenShift nodes, outside of containers,
> creating a cluster of them, then selecting a directory on each node to
> serve as bricks for the volume. The idea here is that I would still
> want to deploy heketi in a container later on, and just use these same
> nodes in the topology file. heketi would still need to be given
> dedicated storage devices on each node.
>
> As far as the algorithm, I figure it should be something like:
>
> For adding a node of a pre-existing cluster, watch for the return
> code/value from the exec of gluster peer probe and if it says peer
> already in list we return success. If for some reason the pre-existing
> cluster only overlaps on a subset of nodes with the heketi cluster,
> gluster can handle this.
>
> +1
>
> In the inverse, when you remove a node from heketi, watch for a
> message that peer cannot be detached because it has bricks and remove
> the node from heketi anyway. heketi already does its own checks to see
> if a volume is on a particular node, so we can't get to the point
> where a heketi-managed brick is still extant on a heketi-managed node
> unless something goes really wrong (and then we have to resort to the
> backend command line anyway, I'd imagine?).
>
> Feedback, concerns, or flames welcome. :)
> --Jose
>
> On Wed, Jan 25, 2017 at 10:55 PM, Luis Pabon<lpabon at gmail.com> <mailto:lpabon at gmail.com> wrote:
>
> I think that people keep asking because they want to continue what they
> know.
>
> I think it would work better if you provide a set of requirements and
> preconditions and the algorithm which satisfies what you would like to do.
> Provide it here, and let's discuss it. No code needed.
>
> - Luis
>
> On Wed, Jan 25, 2017 at 7:29 PM, Jose A. Rivera<jarrpa at redhat.com> <mailto:jarrpa at redhat.com> wrote:
>
> On Wed, Jan 25, 2017 at 2:16 PM, Luis Pabon<lpabon at gmail.com>
> <mailto:lpabon at gmail.com> wrote:
>
> Hi Jos?,
> This has been asked for many many times. Heketi was designed to "rule
> them all". Heketi was never designed for systems that have been setup
> already because the permutations of possibilities of configurations
> could be
> extensive to figure out how to manage. It is like creating a Ceph Rados
> system by yourself, then asking the pool manager to figure out what you
> did.
> If instead Ceph is viewed as a collection of the access+pool+storage and
> not
> as individual parts, then it all works well and is predictable. In the
> same
> way, it should not be viewed as Heketi managing GlusterFS, but
> Heketi/GlusterFS instead. Once this view is accepted (which is what
> users
> want, but old school gluster users have a hard time with), then what
> Heketi
> currently does makes perfect sense.
>
> So, back to the question, no, Heketi does not and will never manage such
> a
> model. Any software that manages such a configuration would be hard to
> productize and guarantee. Can you make a hack that does it? Maybe, but
> reliability and simplicity is what Heketi is after.
>
> Hope this answers your question.
>
> I know this has been asked more than once and I believe this keeps
> being asked for because the above is still an unsatisfactory answer.
> :) Already we are seeing new users asking for maintenance features
> that would be perfectly possible with Gluster but which are currently
> out of reach when going with heketi. I think focusing too hard on
> "simplicity" will quickly become limiting to heketi's desirability. It
> would seem to make more sense to go with a mindset of a tailored
> experience, with the ability to go in deeper if desired.
>
> There doesn't seem to be anything technically complicated about the
> idea that heketi could tolerate a peer probe coming back already
> satisfied, or that a node is removed without removing it from the peer
> list. I don't see how this would prove to be dangerous as long as we
> maintain the understanding that you are not to go in on the backend to
> mess with anything heketi is actively managing. This seems like
> something we could easily test, make reliable, and productize.
>
> --Jose
>
> - Luis
>
> On Tue, Jan 24, 2017 at 12:24 PM, Jose A. Rivera<jarrpa at redhat.com>
> <mailto:jarrpa at redhat.com>
> wrote:
>
> Hey Luis, et al.,
>
> I talked to Ashiq about $SUBJECT, and he raised some concerns.
> Apparently heketi can not load/import nodes that are already part of a
> Gluster cluster? E.g. if I have an existing cluster with all the nodes
> already peer probed, heketi will try to redo the probe and then fail
> when it comes back already in peer list? This seems odd to me, but if
> so sounds like a relatively easy thing to change. Thoughts?
>
> --Jose
>
> _______________________________________________
> heketi-devel mailing list
> heketi-devel at gluster.org <mailto:heketi-devel at gluster.org>
> http://lists.gluster.org/mailman/listinfo/heketi-devel
>
> _______________________________________________
> heketi-devel mailing list
> heketi-devel at gluster.org <mailto:heketi-devel at gluster.org>
> http://lists.gluster.org/mailman/listinfo/heketi-devel
>
>
>
> --
> Best regards
> Pavel Kutishchev
> DevOPS Engineer at
> Self employed.
>
>
--
Best regards
Pavel Kutishchev
DevOPS Engineer at
Self employed.
--
Regards,
Mohamed Ashiq.L
More information about the heketi-devel
mailing list