[heketi-devel] Unable to use Heketi setup to install Gluster for Kubernetes

Gaurav Chhabra varuag.chhabra at gmail.com
Tue Sep 5 16:00:54 UTC 2017


Oh...ok. I misinterpreted while thinking of HA. ​So it's persistent and HA
in the sense that even if one node in the Gluster cluster goes down, others
are available to serve request because of replication. I hope i now got it
correct.

I'm glad i posted my query here and i'm really thankful for your help. I
was struggling to get this thing work for the last 6-7 days.


Thanks again Jose! :)
Gaurav



On Tue, Sep 5, 2017 at 7:00 PM, Jose A. Rivera <jarrpa at redhat.com> wrote:

> That is expected behavior. :) GlusterFS volumes are shared storage,
> meaning the same data is presented to all accessing users. By default
> (in our configuration) it replicates data across multiple bricks, but
> all bricks are represented as a single filesystem and all replicas of
> a file are presented as a single file. Thus, when you delete a file
> you delete it and all its replicas.
>
> On Tue, Sep 5, 2017 at 6:15 AM, Gaurav Chhabra <varuag.chhabra at gmail.com>
> wrote:
> > Hi Jose,
> >
> >
> > There is some progress after using the  following container:
> > https://hub.docker.com/r/gluster/glusterfs-client/
> >
> > I created deployment with 5 replicas for Nginx in Kubernetes using your
> doc.
> > Now as described in the last section of your doc, if i create file
> > 'index.html' inside of a container running in a pod on node-a, i can see
> the
> > same file becomes available on another pod running on node-b. However,
> if i
> > delete the file on one node (say node-a), it becomes unavailable on all
> the
> > three nodes. The file should have been accessible from other nodes even
> if i
> > delete it from one pod on a given node. Correct me if i'm getting it
> wrong.
> >
> > I searched for index.html on the Gluster cluster and i could find it on
> all
> > three nodes with same content. Please check the attached log.
> >
> > If i check /etc/fstab on any of the pods, i see the following:
> >
> > root at nginx-deploy-4253768608-w645w:/# cat /etc/fstab
> > # UNCONFIGURED FSTAB FOR BASE SYSTEM
> >
> > Attached log files.
> >
> > Any idea why the file is not accessible from another pod on node-a or
> node-b
> > if i delete it from a different pod present on another node-c?
> >
> >
> > Regards,
> > Gaurav
> >
> >
> > On Tue, Sep 5, 2017 at 3:41 AM, Jose A. Rivera <jarrpa at redhat.com>
> wrote:
> >>
> >> Yup, I follow that repo and should be contributing to it in the
> >> near-future. :) I've never played with the glusterfs-client image, but
> >> here's hoping!
> >>
> >> On Mon, Sep 4, 2017 at 10:18 AM, Gaurav Chhabra
> >> <varuag.chhabra at gmail.com> wrote:
> >> > Actually the script is launching a container and i'm pretty sure that
> >> > the
> >> > command that's programmed as part of the container launch process is
> >> > actually trying to run that mount command. Now the issue is that this
> >> > again
> >> > falls under Docker area. :( Generally, Docker container providers (in
> >> > this
> >> > case, nixel) provide a Dockerfile which specifies the command that's
> >> > executed as part of the container launch process however, the problem
> in
> >> > case of this container is that it's not maintained. It's 2 years old.
> If
> >> > you
> >> > will check active projects such as gluster/gluster-containers, you
> will
> >> > see
> >> > a proper Dockerfile explaining what it does.
> >> >
> >> > https://hub.docker.com/r/gluster/glusterfs-client/~/dockerfile/
> >> >
> >> > I am now planning to use the above container and see whether i'm
> lucky.
> >> > If
> >> > you will see the above link, it's running
> >> >
> >> > dnf install -y glusterfs-fuse
> >> >
> >> > Hopefully, this should take care of this section which asks to make
> sure
> >> > that glusterfs-client is installed
> >> >
> >> >
> >> > https://github.com/gluster/gluster-kubernetes/tree/
> master/docs/examples/dynamic_provisioning_external_gluster#
> validate-communication-and-gluster-prerequisites-on-the-kubernetes-nodes
> >> >
> >> >
> >> >
> >> >
> >> > On Mon, Sep 4, 2017 at 7:33 PM, Jose A. Rivera <jarrpa at redhat.com>
> >> > wrote:
> >> >>
> >> >> Something is trying to mount a GlusterFS volume, and it's not that
> >> >> script. What's trying to mount?
> >> >>
> >> >> On Mon, Sep 4, 2017 at 8:52 AM, Gaurav Chhabra
> >> >> <varuag.chhabra at gmail.com>
> >> >> wrote:
> >> >> > I am just running this bash script which will start the
> >> >> > rancher-glusterfs-client container:
> >> >> >
> >> >> > [root at node-a ~]# cat rancher-glusterfs-client.sh
> >> >> > #!/bin/bash
> >> >> > sudo docker run --privileged \
> >> >> >         --name=gluster-client \
> >> >> >         -d \
> >> >> >         -v /sys/fs/cgroup:/sys/fs/cgroup \
> >> >> >         -v /var/log/glusterfs:/var/log/glusterfs \
> >> >> >         --env GLUSTER_PEER=10.128.0.12,10.128.0.15,10.128.0.16 \
> >> >> >         nixel/rancher-glusterfs-client
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Mon, Sep 4, 2017 at 6:26 PM, Jose A. Rivera <jarrpa at redhat.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> What is the exact command you're running?
> >> >> >>
> >> >> >> On Mon, Sep 4, 2017 at 4:26 AM, Gaurav Chhabra
> >> >> >> <varuag.chhabra at gmail.com>
> >> >> >> wrote:
> >> >> >> > Hi Jose,
> >> >> >> >
> >> >> >> >
> >> >> >> > From this link, it seems mount.glusterfs might actually be
> present
> >> >> >> > in
> >> >> >> > the
> >> >> >> > container that launched and quickly terminated.
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > https://unix.stackexchange.com/questions/312178/
> glusterfs-replicated-volume-mounting-issue
> >> >> >> >
> >> >> >> > If you check the question that the user posted, the same error
> >> >> >> > (Mount
> >> >> >> > failed) was reported that i sent you in the last email.
> >> >> >> >
> >> >> >> > After seeing the above, i checked /var/log/glusterfs on my host
> >> >> >> > (RancherOS)
> >> >> >> > but it was empty. I ran the container again but with explicit
> >> >> >> > volume
> >> >> >> > mount
> >> >> >> > as shown below:
> >> >> >> >
> >> >> >> > [root at node-a ~]# cat rancher-glusterfs-client.sh
> >> >> >> > #!/bin/bash
> >> >> >> > sudo docker run --privileged \
> >> >> >> >         --name=gluster-client \
> >> >> >> >         -d \
> >> >> >> >         -v /sys/fs/cgroup:/sys/fs/cgroup \
> >> >> >> >         -v /var/log/glusterfs:/var/log/glusterfs \
> >> >> >> >         --env GLUSTER_PEER=10.128.0.12,10.128.0.15,10.128.0.16
> \
> >> >> >> >         nixel/rancher-glusterfs-client
> >> >> >> > This time, i could see a log file
> >> >> >> > (/var/log/glusterfs/mnt-ranchervol.log)
> >> >> >> > present. I have attached the content of the same. Also attached
> >> >> >> > are
> >> >> >> > logs
> >> >> >> > from Heketi client/server (both on one node) and Gluster
> cluster.
> >> >> >> >
> >> >> >> >
> >> >> >> > Regards,
> >> >> >> > Gaurav
> >> >> >> >
> >> >> >> >
> >> >> >> > On Mon, Sep 4, 2017 at 2:29 PM, Gaurav Chhabra
> >> >> >> > <varuag.chhabra at gmail.com>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> Hi Jose,
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> I tried setting up things using the link you provided and i was
> >> >> >> >> able
> >> >> >> >> to
> >> >> >> >> get all steps working for 3 node Gluster cluster, all running
> on
> >> >> >> >> CentOS7,
> >> >> >> >> without any issue. However, as expected, when i tried
> configuring
> >> >> >> >> Kubernetes
> >> >> >> >> by installing nixel/rancher-glusterfs-client container, i got
> >> >> >> >> error:
> >> >> >> >>
> >> >> >> >> [root at node-a ~]# cat rancher-glusterfs-client.sh
> >> >> >> >> #!/bin/bash
> >> >> >> >> sudo docker run --privileged \
> >> >> >> >>         --name=gluster-client \
> >> >> >> >>         -d \
> >> >> >> >>         -v /sys/fs/cgroup:/sys/fs/cgroup \
> >> >> >> >>         --env GLUSTER_PEER=10.128.0.12,10.128.0.15,10.128.0.16
> \
> >> >> >> >>         nixel/rancher-glusterfs-client
> >> >> >> >>
> >> >> >> >> [root at node-a ~]# ./rancher-glusterfs-client.sh
> >> >> >> >> ac069caccdce147d6f423fc566166345191dbc1b11f3416c66207a1fd11f
> da6b
> >> >> >> >>
> >> >> >> >> [root at node-a ~]# docker logs gluster-client
> >> >> >> >> => Checking if I can reach GlusterFS node 10.128.0.12 ...
> >> >> >> >> => GlusterFS node 10.128.0.12 is alive
> >> >> >> >> => Mounting GlusterFS volume ranchervol from GlusterFS node
> >> >> >> >> 10.128.0.12
> >> >> >> >> ...
> >> >> >> >> Mount failed. Please check the log file for more details.
> >> >> >> >>
> >> >> >> >> If i try running the next step as described in your link, i get
> >> >> >> >> the
> >> >> >> >> following:
> >> >> >> >>
> >> >> >> >> [root at node-a ~]# modprobe fuse
> >> >> >> >> modprobe: module fuse not found in modules.dep
> >> >> >> >>
> >> >> >> >> Since the container failed to start, i could only check on the
> >> >> >> >> host
> >> >> >> >> (RancherOS) and i could only find two mount-related commands:
> >> >> >> >> mount
> >> >> >> >> &
> >> >> >> >> mountpoint
> >> >> >> >>
> >> >> >> >> Any pointers?
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> Regards,
> >> >> >> >> Gaurav
> >> >> >> >>
> >> >> >> >> On Sun, Sep 3, 2017 at 11:18 PM, Jose A. Rivera
> >> >> >> >> <jarrpa at redhat.com>
> >> >> >> >> wrote:
> >> >> >> >>>
> >> >> >> >>> Installing the glusterfs-client container should be fine. :)
> The
> >> >> >> >>> main
> >> >> >> >>> thing that's needed is that all your Kubernetes nodes need to
> >> >> >> >>> have
> >> >> >> >>> the
> >> >> >> >>> "mount.glusterfs" command available so Kube can mount the
> >> >> >> >>> GlusterFS
> >> >> >> >>> volumes and present them to the pods.
> >> >> >> >>>
> >> >> >> >>> On Sun, Sep 3, 2017 at 12:14 PM, Gaurav Chhabra
> >> >> >> >>> <varuag.chhabra at gmail.com> wrote:
> >> >> >> >>> > Thanks Jose. The link you've suggested looks good but again
> it
> >> >> >> >>> > expects
> >> >> >> >>> > me to
> >> >> >> >>> > install gluster-client on Kubernetes node and i fall into
> the
> >> >> >> >>> > same
> >> >> >> >>> > issue of
> >> >> >> >>> > installing a container for glusterfs. Only difference is
> that
> >> >> >> >>> > this
> >> >> >> >>> > time
> >> >> >> >>> > it's
> >> >> >> >>> > glusterfs-client and not glusterfs-server. :)
> >> >> >> >>> >
> >> >> >> >>> > I will try this out and let you know tomorrow.
> >> >> >> >>> >
> >> >> >> >>> >
> >> >> >> >>> > Regards,
> >> >> >> >>> > Gaurav
> >> >> >> >>> >
> >> >> >> >>> >
> >> >> >> >>> > On Sun, Sep 3, 2017 at 2:11 AM, Jose A. Rivera
> >> >> >> >>> > <jarrpa at redhat.com>
> >> >> >> >>> > wrote:
> >> >> >> >>> >>
> >> >> >> >>> >> Hey, no problem! I'm eager to learn more about different
> >> >> >> >>> >> flavors
> >> >> >> >>> >> of
> >> >> >> >>> >> Linux, I just apologize for my relative inexperience with
> >> >> >> >>> >> them.
> >> >> >> >>> >> :)
> >> >> >> >>> >>
> >> >> >> >>> >> To that end, I will also admit I'm not very experienced
> with
> >> >> >> >>> >> direct
> >> >> >> >>> >> Docker myself. I understand the basic workflow and know
> some
> >> >> >> >>> >> of
> >> >> >> >>> >> the
> >> >> >> >>> >> run options, but not having deep experience keeps me from
> >> >> >> >>> >> having
> >> >> >> >>> >> a
> >> >> >> >>> >> better understanding of the patterns and consequences.
> >> >> >> >>> >>
> >> >> >> >>> >> Thus, I'd like to guide you in a direction I'd more apt to
> >> >> >> >>> >> help
> >> >> >> >>> >> you
> >> >> >> >>> >> in
> >> >> >> >>> >> right now. I know that you can't have multiple GlusterFS
> >> >> >> >>> >> servers
> >> >> >> >>> >> running on the same nodes, and I know that we have been
> >> >> >> >>> >> successfully
> >> >> >> >>> >> running several configurations using our
> >> >> >> >>> >> gluster/gluster-centos
> >> >> >> >>> >> image.
> >> >> >> >>> >> If you follow the Kubernetes configuration on
> >> >> >> >>> >> gluster-kubernetes,
> >> >> >> >>> >> the
> >> >> >> >>> >> pod/container is run privileged and with host networking,
> and
> >> >> >> >>> >> we
> >> >> >> >>> >> require that the node has all listed ports open, not just
> >> >> >> >>> >> 2222.
> >> >> >> >>> >> The
> >> >> >> >>> >> sshd running in the container is listening on 2222, not 22,
> >> >> >> >>> >> but
> >> >> >> >>> >> it
> >> >> >> >>> >> is
> >> >> >> >>> >> also not really required if you're not doing
> geo-replication.
> >> >> >> >>> >>
> >> >> >> >>> >> Alternatively, you can indeed run GlusterFS outside of
> >> >> >> >>> >> Kubernetes
> >> >> >> >>> >> but
> >> >> >> >>> >> still have Kubernetes apps access GlusterFS storage. The
> >> >> >> >>> >> nodes
> >> >> >> >>> >> can
> >> >> >> >>> >> be
> >> >> >> >>> >> anything you want, they just need to be running GlusterFS
> and
> >> >> >> >>> >> you
> >> >> >> >>> >> need
> >> >> >> >>> >> a heketi service managing them. Here is an example of how
> to
> >> >> >> >>> >> set
> >> >> >> >>> >> this
> >> >> >> >>> >> up using CentOS:
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>> >> https://github.com/gluster/gluster-kubernetes/tree/
> master/docs/examples/dynamic_provisioning_external_gluster
> >> >> >> >>> >>
> >> >> >> >>> >> Hope this is at least leading you in a useful direction. :)
> >> >> >> >>> >>
> >> >> >> >>> >> --Jose
> >> >> >> >>> >>
> >> >> >> >>> >> On Sat, Sep 2, 2017 at 3:16 PM, Gaurav Chhabra
> >> >> >> >>> >> <varuag.chhabra at gmail.com>
> >> >> >> >>> >> wrote:
> >> >> >> >>> >> > Hi Jose,
> >> >> >> >>> >> >
> >> >> >> >>> >> > webcenter/rancher-glusterfs-server is actually a
> container
> >> >> >> >>> >> > provided
> >> >> >> >>> >> > by
> >> >> >> >>> >> > Sebastien, its maintainer. It's a Docker container which
> >> >> >> >>> >> > has
> >> >> >> >>> >> > GlusterFS
> >> >> >> >>> >> > server running within it. On the host i.e., RancherOS,
> >> >> >> >>> >> > there
> >> >> >> >>> >> > is
> >> >> >> >>> >> > no
> >> >> >> >>> >> > separate
> >> >> >> >>> >> > GlusterFS server running because we cannot install
> anything
> >> >> >> >>> >> > that
> >> >> >> >>> >> > way.
> >> >> >> >>> >> > Running using container is the only way so i started
> >> >> >> >>> >> > ancher-glusterfs-server
> >> >> >> >>> >> > container with the following parameters:
> >> >> >> >>> >> >
> >> >> >> >>> >> > [root at node-1 rancher]# cat gluster-server.sh
> >> >> >> >>> >> > #!/bin/bash
> >> >> >> >>> >> > sudo docker run --name=gluster-server -d \
> >> >> >> >>> >> >         --env 'SERVICE_NAME=gluster' \
> >> >> >> >>> >> >         --restart always \
> >> >> >> >>> >> >         --publish 2222:22 \
> >> >> >> >>> >> >         webcenter/rancher-glusterfs-server
> >> >> >> >>> >> >
> >> >> >> >>> >> > Here's the link to the dockerfile:
> >> >> >> >>> >> >
> >> >> >> >>> >> >
> >> >> >> >>> >> >
> >> >> >> >>> >> >
> >> >> >> >>> >> >
> >> >> >> >>> >> > https://hub.docker.com/r/webcenter/rancher-glusterfs-
> server/~/dockerfile/
> >> >> >> >>> >> >
> >> >> >> >>> >> > It's similar to other GlusteFS containers provided by
> other
> >> >> >> >>> >> > maintainers
> >> >> >> >>> >> > for
> >> >> >> >>> >> > different OS. For example, for CentOS, we have
> >> >> >> >>> >> >
> >> >> >> >>> >> > https://hub.docker.com/r/gluster/gluster-centos/~/
> dockerfile/
> >> >> >> >>> >> >
> >> >> >> >>> >> > From what i understand, Heketi does support container
> based
> >> >> >> >>> >> > GlusterFS
> >> >> >> >>> >> > server
> >> >> >> >>> >> > as mentioned in the prerequisite where it says:
> >> >> >> >>> >> >
> >> >> >> >>> >> > "Each node must have the following ports opened for
> >> >> >> >>> >> > GlusterFS
> >> >> >> >>> >> > communications:
> >> >> >> >>> >> >  2222 - GlusterFS pod's sshd"
> >> >> >> >>> >> >
> >> >> >> >>> >> > That's the reason i've exposed port 2222 for 22 as shown
> >> >> >> >>> >> > above.
> >> >> >> >>> >> > Please
> >> >> >> >>> >> > correct me if i misunderstood it.
> >> >> >> >>> >> >
> >> >> >> >>> >> > As soon as i run the above script (gluster-server.sh), it
> >> >> >> >>> >> > automatically
> >> >> >> >>> >> > creates the following directories on host. This should
> have
> >> >> >> >>> >> > ideally
> >> >> >> >>> >> > not
> >> >> >> >>> >> > been
> >> >> >> >>> >> > empty as you mentioned.
> >> >> >> >>> >> >
> >> >> >> >>> >> > /etc/glusterfs    /var/lib/glusterd    /var/log/glusterfs
> >> >> >> >>> >> >
> >> >> >> >>> >> > Just wanted to know in which circumstances do we get this
> >> >> >> >>> >> > specific
> >> >> >> >>> >> > error
> >> >> >> >>> >> > (Failed to get D-Bus connection: Operation not permitted)
> >> >> >> >>> >> > related
> >> >> >> >>> >> > to
> >> >> >> >>> >> > Readiness probe failing. Searching online took me to
> >> >> >> >>> >> > discussions
> >> >> >> >>> >> > around
> >> >> >> >>> >> > running container in privileged mode and some directory
> to
> >> >> >> >>> >> > be
> >> >> >> >>> >> > mounted.
> >> >> >> >>> >> > Based
> >> >> >> >>> >> > on that, i also modified my container startup script to
> the
> >> >> >> >>> >> > following:
> >> >> >> >>> >> >
> >> >> >> >>> >> > #!/bin/bash
> >> >> >> >>> >> > sudo docker run --privileged \
> >> >> >> >>> >> >         --name=gluster-server \
> >> >> >> >>> >> >         -d \
> >> >> >> >>> >> >         -v /sys/fs/cgroup:/sys/fs/cgroup \
> >> >> >> >>> >> >         -v /etc/glusterfs:/etc/glusterfs \
> >> >> >> >>> >> >         -v /var/lib/glusterd:/var/lib/glusterd \
> >> >> >> >>> >> >         -v /var/log/glusterfs:/var/log/glusterfs \
> >> >> >> >>> >> >         --env 'SERVICE_NAME=gluster' \
> >> >> >> >>> >> >         --restart always \
> >> >> >> >>> >> >         --publish 2222:22 \
> >> >> >> >>> >> >         webcenter/rancher-glusterfs-server
> >> >> >> >>> >> > Still, the issue persists.
> >> >> >> >>> >> >
> >> >> >> >>> >> > I also logged into the container and checked whether
> >> >> >> >>> >> > systemctl
> >> >> >> >>> >> > command
> >> >> >> >>> >> > is
> >> >> >> >>> >> > present. It was there but manualy running the command
> also
> >> >> >> >>> >> > doesn't
> >> >> >> >>> >> > work:
> >> >> >> >>> >> >
> >> >> >> >>> >> > [root at node-c ~]# docker exec -it gluster-server
> /bin/bash
> >> >> >> >>> >> > root at 42150f203f80:/app# systemctl status
> glusterd.service
> >> >> >> >>> >> > WARNING: terminal is not fully functional
> >> >> >> >>> >> > Failed to connect to bus: No such file or directory
> >> >> >> >>> >> >
> >> >> >> >>> >> > Under section 'ADVANCED OPTIONS - Security/Host' in this
> >> >> >> >>> >> > link,
> >> >> >> >>> >> > it
> >> >> >> >>> >> > talks
> >> >> >> >>> >> > about SYS_ADMIN setting. Any idea how i can try this?
> >> >> >> >>> >> >
> >> >> >> >>> >> > Also, there was this mentioned in the Heketi setup page:
> >> >> >> >>> >> > "If you are not able to deploy a hyper-converged
> GlusterFS
> >> >> >> >>> >> > cluster,
> >> >> >> >>> >> > you
> >> >> >> >>> >> > must
> >> >> >> >>> >> > have one running somewhere that the Kubernetes nodes can
> >> >> >> >>> >> > access"
> >> >> >> >>> >> >
> >> >> >> >>> >> >>>> Does it mean running the three node Gluster cluster
> >> >> >> >>> >> >>>> outside
> >> >> >> >>> >> >>>> Kubernetes,
> >> >> >> >>> >> >>>> may be on some VM running on RHEL/CentOS etc? If yes,
> >> >> >> >>> >> >>>> then
> >> >> >> >>> >> >>>> how
> >> >> >> >>> >> >>>> will i
> >> >> >> >>> >> >>>> be
> >> >> >> >>> >> >>>> able to tell Gluster which volume from the Kubernetes
> >> >> >> >>> >> >>>> cluster
> >> >> >> >>> >> >>>> pod
> >> >> >> >>> >> >>>> to
> >> >> >> >>> >> >>>> sync?
> >> >> >> >>> >> >>>> Any references?
> >> >> >> >>> >> >
> >> >> >> >>> >> >
> >> >> >> >>> >> > I really appreciate your responses despite the fact that
> >> >> >> >>> >> > you've
> >> >> >> >>> >> > not
> >> >> >> >>> >> > used
> >> >> >> >>> >> > RancherOS but still trying to help.
> >> >> >> >>> >> >
> >> >> >> >>> >> >
> >> >> >> >>> >> > Thanks,
> >> >> >> >>> >> > Gaurav
> >> >> >> >>> >> >
> >> >> >> >>> >> >
> >> >> >> >>> >> > On Sat, Sep 2, 2017 at 7:35 PM, Jose A. Rivera
> >> >> >> >>> >> > <jarrpa at redhat.com>
> >> >> >> >>> >> > wrote:
> >> >> >> >>> >> >>
> >> >> >> >>> >> >> I'm afraid I have no experience with RancherOS, so I may
> >> >> >> >>> >> >> be
> >> >> >> >>> >> >> missing
> >> >> >> >>> >> >> some things about how it works. My primary experience is
> >> >> >> >>> >> >> with
> >> >> >> >>> >> >> Fedora,
> >> >> >> >>> >> >> CentOS, and Ubuntu.
> >> >> >> >>> >> >>
> >> >> >> >>> >> >> What is webcenter/rancher-glusterfs-server? If it's
> >> >> >> >>> >> >> running
> >> >> >> >>> >> >> another
> >> >> >> >>> >> >> glusterd then you probably don't want to be running it
> and
> >> >> >> >>> >> >> should
> >> >> >> >>> >> >> remove it from your systems.
> >> >> >> >>> >> >>
> >> >> >> >>> >> >> The glusterfs pods mount hostpath volumes from the host
> >> >> >> >>> >> >> they're
> >> >> >> >>> >> >> running on to persist their configuration. Thus anything
> >> >> >> >>> >> >> they
> >> >> >> >>> >> >> write
> >> >> >> >>> >> >> to
> >> >> >> >>> >> >> those directories should land on the host. If that's not
> >> >> >> >>> >> >> happening
> >> >> >> >>> >> >> then that's an additional problem.
> >> >> >> >>> >> >>
> >> >> >> >>> >> >> --Jose
> >> >> >> >>> >> >>
> >> >> >> >>> >> >> On Fri, Sep 1, 2017 at 11:17 PM, Gaurav Chhabra
> >> >> >> >>> >> >> <varuag.chhabra at gmail.com> wrote:
> >> >> >> >>> >> >> > Hi Jose,
> >> >> >> >>> >> >> >
> >> >> >> >>> >> >> >
> >> >> >> >>> >> >> > I tried your suggestion but there is one confusion
> >> >> >> >>> >> >> > regarding
> >> >> >> >>> >> >> > point
> >> >> >> >>> >> >> > #3.
> >> >> >> >>> >> >> > Since
> >> >> >> >>> >> >> > RancherOS has everything running as container, i am
> >> >> >> >>> >> >> > running
> >> >> >> >>> >> >> > webcenter/rancher-glusterfs-server container on all
> >> >> >> >>> >> >> > three
> >> >> >> >>> >> >> > nodes.
> >> >> >> >>> >> >> > Now
> >> >> >> >>> >> >> > as
> >> >> >> >>> >> >> > far
> >> >> >> >>> >> >> > as removing the directories are concerned, i hope you
> >> >> >> >>> >> >> > meant
> >> >> >> >>> >> >> > removing
> >> >> >> >>> >> >> > them on
> >> >> >> >>> >> >> > the host and _not_ from within the container. After
> >> >> >> >>> >> >> > completing
> >> >> >> >>> >> >> > step 1
> >> >> >> >>> >> >> > and 2,
> >> >> >> >>> >> >> > i checked the contents of all the directories that you
> >> >> >> >>> >> >> > specified
> >> >> >> >>> >> >> > in
> >> >> >> >>> >> >> > point
> >> >> >> >>> >> >> > #3. All were empty as you can see in the attached
> >> >> >> >>> >> >> > other_logs.txt.
> >> >> >> >>> >> >> > So
> >> >> >> >>> >> >> > i
> >> >> >> >>> >> >> > got
> >> >> >> >>> >> >> > confused. I ran the deploy again but the issue
> persists.
> >> >> >> >>> >> >> > Two
> >> >> >> >>> >> >> > pods
> >> >> >> >>> >> >> > show
> >> >> >> >>> >> >> > Liveness error and the third one, Readiness error.
> >> >> >> >>> >> >> >
> >> >> >> >>> >> >> > I then tried removing those directories (Step #3) from
> >> >> >> >>> >> >> > within
> >> >> >> >>> >> >> > the
> >> >> >> >>> >> >> > container
> >> >> >> >>> >> >> > but getting following error:
> >> >> >> >>> >> >> >
> >> >> >> >>> >> >> > root at c0f8ab4d92a2:/app# rm -rf /var/lib/heketi
> >> >> >> >>> >> >> > /etc/glusterfs
> >> >> >> >>> >> >> > /var/lib/glusterd /var/log/glusterfs
> >> >> >> >>> >> >> > rm: cannot remove '/var/lib/glusterd': Device or
> >> >> >> >>> >> >> > resource
> >> >> >> >>> >> >> > busy
> >> >> >> >>> >> >> >
> >> >> >> >>> >> >> >
> >> >> >> >>> >> >> >
> >> >> >> >>> >> >> > On Fri, Sep 1, 2017 at 8:21 PM, Jose A. Rivera
> >> >> >> >>> >> >> > <jarrpa at redhat.com>
> >> >> >> >>> >> >> > wrote:
> >> >> >> >>> >> >> >>
> >> >> >> >>> >> >> >> 1. Add a line to the ssh-exec portion of heketi.json
> of
> >> >> >> >>> >> >> >> the
> >> >> >> >>> >> >> >> sort:
> >> >> >> >>> >> >> >>
> >> >> >> >>> >> >> >> "sudo": true,
> >> >> >> >>> >> >> >>
> >> >> >> >>> >> >> >> 2. Run
> >> >> >> >>> >> >> >>
> >> >> >> >>> >> >> >> gk-deploy -g --abort
> >> >> >> >>> >> >> >>
> >> >> >> >>> >> >> >> 3. On the nodes that were/will be running GlusterFS
> >> >> >> >>> >> >> >> pods,
> >> >> >> >>> >> >> >> run:
> >> >> >> >>> >> >> >>
> >> >> >> >>> >> >> >> rm -rf /var/lib/heketi /etc/glusterfs
> /var/lib/glusterd
> >> >> >> >>> >> >> >> /var/log/glusterfs
> >> >> >> >>> >> >> >>
> >> >> >> >>> >> >> >> Then try the deploy again.
> >> >> >> >>> >> >> >>
> >> >> >> >>> >> >> >> On Fri, Sep 1, 2017 at 6:05 AM, Gaurav Chhabra
> >> >> >> >>> >> >> >> <varuag.chhabra at gmail.com>
> >> >> >> >>> >> >> >> wrote:
> >> >> >> >>> >> >> >> > Hi Jose,
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > Thanks for the reply. It seems the three gluster
> pods
> >> >> >> >>> >> >> >> > might
> >> >> >> >>> >> >> >> > have
> >> >> >> >>> >> >> >> > been
> >> >> >> >>> >> >> >> > a
> >> >> >> >>> >> >> >> > copy-paste from another set of cluster where i was
> >> >> >> >>> >> >> >> > trying
> >> >> >> >>> >> >> >> > to
> >> >> >> >>> >> >> >> > setup
> >> >> >> >>> >> >> >> > the
> >> >> >> >>> >> >> >> > same
> >> >> >> >>> >> >> >> > thing using CentOS. Sorry for that. By the way, i
> did
> >> >> >> >>> >> >> >> > check
> >> >> >> >>> >> >> >> > for
> >> >> >> >>> >> >> >> > the
> >> >> >> >>> >> >> >> > kernel
> >> >> >> >>> >> >> >> > modules and it seems it's already there. Also, i am
> >> >> >> >>> >> >> >> > attaching
> >> >> >> >>> >> >> >> > fresh
> >> >> >> >>> >> >> >> > set
> >> >> >> >>> >> >> >> > of
> >> >> >> >>> >> >> >> > files because i created a new cluster and thought
> of
> >> >> >> >>> >> >> >> > giving
> >> >> >> >>> >> >> >> > it
> >> >> >> >>> >> >> >> > a
> >> >> >> >>> >> >> >> > try
> >> >> >> >>> >> >> >> > again.
> >> >> >> >>> >> >> >> > Issue still persists. :(
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > In heketi.json, there is a slight change w.r.t the
> >> >> >> >>> >> >> >> > user
> >> >> >> >>> >> >> >> > which
> >> >> >> >>> >> >> >> > connects
> >> >> >> >>> >> >> >> > to
> >> >> >> >>> >> >> >> > glusterfs node using SSH. I am not sure how Heketi
> >> >> >> >>> >> >> >> > was
> >> >> >> >>> >> >> >> > using
> >> >> >> >>> >> >> >> > root
> >> >> >> >>> >> >> >> > user
> >> >> >> >>> >> >> >> > to
> >> >> >> >>> >> >> >> > login because i wasn't able to use root and do
> manual
> >> >> >> >>> >> >> >> > SSH.
> >> >> >> >>> >> >> >> > With
> >> >> >> >>> >> >> >> > rancher
> >> >> >> >>> >> >> >> > user, i can login successfully so i think this
> should
> >> >> >> >>> >> >> >> > be
> >> >> >> >>> >> >> >> > fine.
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > /etc/heketi/heketi.json:
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >> >>> >> >> >> >     "executor": "ssh",
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >     "_sshexec_comment": "SSH username and private
> key
> >> >> >> >>> >> >> >> > file
> >> >> >> >>> >> >> >> > information",
> >> >> >> >>> >> >> >> >     "sshexec": {
> >> >> >> >>> >> >> >> >       "keyfile": "/var/lib/heketi/.ssh/id_rsa",
> >> >> >> >>> >> >> >> >       "user": "rancher",
> >> >> >> >>> >> >> >> >       "port": "22",
> >> >> >> >>> >> >> >> >       "fstab": "/etc/fstab"
> >> >> >> >>> >> >> >> >     },
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > Before running gk-deploy:
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >> >>> >> >> >> > [root at workstation deploy]# kubectl get
> >> >> >> >>> >> >> >> > nodes,pods,daemonset,deployments,services
> >> >> >> >>> >> >> >> > NAME                                     STATUS
> >> >> >> >>> >> >> >> > AGE
> >> >> >> >>> >> >> >> > VERSION
> >> >> >> >>> >> >> >> > no/node-a.c.kubernetes-174104.internal   Ready
>  3h
> >> >> >> >>> >> >> >> > v1.7.2-rancher1
> >> >> >> >>> >> >> >> > no/node-b.c.kubernetes-174104.internal   Ready
>  3h
> >> >> >> >>> >> >> >> > v1.7.2-rancher1
> >> >> >> >>> >> >> >> > no/node-c.c.kubernetes-174104.internal   Ready
>  3h
> >> >> >> >>> >> >> >> > v1.7.2-rancher1
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > NAME             CLUSTER-IP   EXTERNAL-IP   PORT(S)
> >> >> >> >>> >> >> >> > AGE
> >> >> >> >>> >> >> >> > svc/kubernetes   10.43.0.1    <none>        443/TCP
> >> >> >> >>> >> >> >> > 3h
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > After running gk-deploy:
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >> >>> >> >> >> > [root at workstation messagegc]# kubectl get
> >> >> >> >>> >> >> >> > nodes,pods,daemonset,deployments,services
> >> >> >> >>> >> >> >> > NAME                                     STATUS
> >> >> >> >>> >> >> >> > AGE
> >> >> >> >>> >> >> >> > VERSION
> >> >> >> >>> >> >> >> > no/node-a.c.kubernetes-174104.internal   Ready
>  3h
> >> >> >> >>> >> >> >> > v1.7.2-rancher1
> >> >> >> >>> >> >> >> > no/node-b.c.kubernetes-174104.internal   Ready
>  3h
> >> >> >> >>> >> >> >> > v1.7.2-rancher1
> >> >> >> >>> >> >> >> > no/node-c.c.kubernetes-174104.internal   Ready
>  3h
> >> >> >> >>> >> >> >> > v1.7.2-rancher1
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > NAME                 READY     STATUS    RESTARTS
> >> >> >> >>> >> >> >> > AGE
> >> >> >> >>> >> >> >> > po/glusterfs-0j9l5   0/1       Running   0
> >> >> >> >>> >> >> >> > 2m
> >> >> >> >>> >> >> >> > po/glusterfs-gqz4c   0/1       Running   0
> >> >> >> >>> >> >> >> > 2m
> >> >> >> >>> >> >> >> > po/glusterfs-gxvcb   0/1       Running   0
> >> >> >> >>> >> >> >> > 2m
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > NAME           DESIRED   CURRENT   READY
> >> >> >> >>> >> >> >> > UP-TO-DATE
> >> >> >> >>> >> >> >> > AVAILABLE
> >> >> >> >>> >> >> >> > NODE-SELECTOR           AGE
> >> >> >> >>> >> >> >> > ds/glusterfs   3         3         0         3
> >> >> >> >>> >> >> >> > 0
> >> >> >> >>> >> >> >> > storagenode=glusterfs   2m
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > NAME             CLUSTER-IP   EXTERNAL-IP   PORT(S)
> >> >> >> >>> >> >> >> > AGE
> >> >> >> >>> >> >> >> > svc/kubernetes   10.43.0.1    <none>        443/TCP
> >> >> >> >>> >> >> >> > 3h
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > Kernel module check on all three nodes:
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >> >>> >> >> >> > [root at node-a ~]# find /lib*/modules/$(uname -r)
> -name
> >> >> >> >>> >> >> >> > *.ko
> >> >> >> >>> >> >> >> > |
> >> >> >> >>> >> >> >> > grep
> >> >> >> >>> >> >> >> > 'thin-pool\|snapshot\|mirror' | xargs ls -ltr
> >> >> >> >>> >> >> >> > -rw-r--r--    1 root     root         92310 Jun 26
> >> >> >> >>> >> >> >> > 04:13
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > /lib64/modules/4.9.34-rancher/
> kernel/drivers/md/dm-thin-pool.ko
> >> >> >> >>> >> >> >> > -rw-r--r--    1 root     root         56982 Jun 26
> >> >> >> >>> >> >> >> > 04:13
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > /lib64/modules/4.9.34-rancher/
> kernel/drivers/md/dm-snapshot.ko
> >> >> >> >>> >> >> >> > -rw-r--r--    1 root     root         27070 Jun 26
> >> >> >> >>> >> >> >> > 04:13
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > /lib64/modules/4.9.34-rancher/
> kernel/drivers/md/dm-mirror.ko
> >> >> >> >>> >> >> >> > -rw-r--r--    1 root     root         92310 Jun 26
> >> >> >> >>> >> >> >> > 04:13
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > /lib/modules/4.9.34-rancher/
> kernel/drivers/md/dm-thin-pool.ko
> >> >> >> >>> >> >> >> > -rw-r--r--    1 root     root         56982 Jun 26
> >> >> >> >>> >> >> >> > 04:13
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > /lib/modules/4.9.34-rancher/
> kernel/drivers/md/dm-snapshot.ko
> >> >> >> >>> >> >> >> > -rw-r--r--    1 root     root         27070 Jun 26
> >> >> >> >>> >> >> >> > 04:13
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > /lib/modules/4.9.34-rancher/
> kernel/drivers/md/dm-mirror.ko
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > Error snapshot attached.
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > In my first mail, i checked that Readiness Probe
> >> >> >> >>> >> >> >> > failure
> >> >> >> >>> >> >> >> > check
> >> >> >> >>> >> >> >> > has
> >> >> >> >>> >> >> >> > this
> >> >> >> >>> >> >> >> > code
> >> >> >> >>> >> >> >> > in kube-templates/glusterfs-daemonset.yaml file:
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >> >>> >> >> >> >         readinessProbe:
> >> >> >> >>> >> >> >> >           timeoutSeconds: 3
> >> >> >> >>> >> >> >> >           initialDelaySeconds: 40
> >> >> >> >>> >> >> >> >           exec:
> >> >> >> >>> >> >> >> >             command:
> >> >> >> >>> >> >> >> >             - "/bin/bash"
> >> >> >> >>> >> >> >> >             - "-c"
> >> >> >> >>> >> >> >> >             - systemctl status glusterd.service
> >> >> >> >>> >> >> >> >           periodSeconds: 25
> >> >> >> >>> >> >> >> >           successThreshold: 1
> >> >> >> >>> >> >> >> >           failureThreshold: 15
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > I tried logging into glustefs container on one of
> the
> >> >> >> >>> >> >> >> > node
> >> >> >> >>> >> >> >> > and
> >> >> >> >>> >> >> >> > ran
> >> >> >> >>> >> >> >> > the
> >> >> >> >>> >> >> >> > above
> >> >> >> >>> >> >> >> > command:
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > [root at node-a ~]# docker exec -it
> c0f8ab4d92a23b6df2
> >> >> >> >>> >> >> >> > /bin/bash
> >> >> >> >>> >> >> >> > root at c0f8ab4d92a2:/app# systemctl status
> >> >> >> >>> >> >> >> > glusterd.service
> >> >> >> >>> >> >> >> > WARNING: terminal is not fully functional
> >> >> >> >>> >> >> >> > Failed to connect to bus: No such file or directory
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > Any check that i can do manually on nodes to debug
> >> >> >> >>> >> >> >> > further?
> >> >> >> >>> >> >> >> > Any
> >> >> >> >>> >> >> >> > suggestions?
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> > On Thu, Aug 31, 2017 at 6:53 PM, Jose A. Rivera
> >> >> >> >>> >> >> >> > <jarrpa at redhat.com>
> >> >> >> >>> >> >> >> > wrote:
> >> >> >> >>> >> >> >> >>
> >> >> >> >>> >> >> >> >> Hey Gaurav,
> >> >> >> >>> >> >> >> >>
> >> >> >> >>> >> >> >> >> The kernel modules must be loaded on all nodes
> that
> >> >> >> >>> >> >> >> >> will
> >> >> >> >>> >> >> >> >> run
> >> >> >> >>> >> >> >> >> heketi
> >> >> >> >>> >> >> >> >> pods. Additionally, you must have at least three
> >> >> >> >>> >> >> >> >> nodes
> >> >> >> >>> >> >> >> >> specified
> >> >> >> >>> >> >> >> >> in
> >> >> >> >>> >> >> >> >> your topology file. I'm not sure how you're
> getting
> >> >> >> >>> >> >> >> >> three
> >> >> >> >>> >> >> >> >> gluster
> >> >> >> >>> >> >> >> >> pods
> >> >> >> >>> >> >> >> >> when you only have two nodes defined... :)
> >> >> >> >>> >> >> >> >>
> >> >> >> >>> >> >> >> >> --Jose
> >> >> >> >>> >> >> >> >>
> >> >> >> >>> >> >> >> >> On Wed, Aug 30, 2017 at 5:27 AM, Gaurav Chhabra
> >> >> >> >>> >> >> >> >> <varuag.chhabra at gmail.com> wrote:
> >> >> >> >>> >> >> >> >> > Hi,
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > I have the following setup in place:
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > 1 node    : RancherOS having Rancher application
> >> >> >> >>> >> >> >> >> > for
> >> >> >> >>> >> >> >> >> > Kubernetes
> >> >> >> >>> >> >> >> >> > setup
> >> >> >> >>> >> >> >> >> > 2 nodes  : RancherOS having Rancher agent
> >> >> >> >>> >> >> >> >> > 1 node   : CentOS 7 workstation having kubectl
> >> >> >> >>> >> >> >> >> > installed
> >> >> >> >>> >> >> >> >> > and
> >> >> >> >>> >> >> >> >> > folder
> >> >> >> >>> >> >> >> >> > cloned/downloaded from
> >> >> >> >>> >> >> >> >> > https://github.com/gluster/gluster-kubernetes
> >> >> >> >>> >> >> >> >> > using
> >> >> >> >>> >> >> >> >> > which i run Heketi setup (gk-deploy -g)
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > I also have rancher-glusterfs-server container
> >> >> >> >>> >> >> >> >> > running
> >> >> >> >>> >> >> >> >> > with
> >> >> >> >>> >> >> >> >> > the
> >> >> >> >>> >> >> >> >> > following
> >> >> >> >>> >> >> >> >> > configuration:
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >> >>> >> >> >> >> > [root at node-1 rancher]# cat gluster-server.sh
> >> >> >> >>> >> >> >> >> > #!/bin/bash
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > sudo docker run --name=gluster-server -d \
> >> >> >> >>> >> >> >> >> >         --env 'SERVICE_NAME=gluster' \
> >> >> >> >>> >> >> >> >> >         --restart always \
> >> >> >> >>> >> >> >> >> >         --env 'GLUSTER_DATA=/srv/docker/gitlab'
> \
> >> >> >> >>> >> >> >> >> >         --publish 2222:22 \
> >> >> >> >>> >> >> >> >> >         webcenter/rancher-glusterfs-server
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > In /etc/heketi/heketi.json, following is the
> only
> >> >> >> >>> >> >> >> >> > modified
> >> >> >> >>> >> >> >> >> > portion:
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >> >>> >> >> >> >> >     "executor": "ssh",
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >     "_sshexec_comment": "SSH username and
> private
> >> >> >> >>> >> >> >> >> > key
> >> >> >> >>> >> >> >> >> > file
> >> >> >> >>> >> >> >> >> > information",
> >> >> >> >>> >> >> >> >> >     "sshexec": {
> >> >> >> >>> >> >> >> >> >       "keyfile": "/var/lib/heketi/.ssh/id_rsa",
> >> >> >> >>> >> >> >> >> >       "user": "root",
> >> >> >> >>> >> >> >> >> >       "port": "22",
> >> >> >> >>> >> >> >> >> >       "fstab": "/etc/fstab"
> >> >> >> >>> >> >> >> >> >     },
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > Status before running gk-deploy:
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > [root at workstation deploy]# kubectl get
> >> >> >> >>> >> >> >> >> > nodes,pods,services,deployments
> >> >> >> >>> >> >> >> >> > NAME                                     STATUS
> >> >> >> >>> >> >> >> >> > AGE
> >> >> >> >>> >> >> >> >> > VERSION
> >> >> >> >>> >> >> >> >> > no/node-1.c.kubernetes-174104.internal   Ready
> >> >> >> >>> >> >> >> >> > 2d
> >> >> >> >>> >> >> >> >> > v1.7.2-rancher1
> >> >> >> >>> >> >> >> >> > no/node-2.c.kubernetes-174104.internal   Ready
> >> >> >> >>> >> >> >> >> > 2d
> >> >> >> >>> >> >> >> >> > v1.7.2-rancher1
> >> >> >> >>> >> >> >> >> > no/node-3.c.kubernetes-174104.internal   Ready
> >> >> >> >>> >> >> >> >> > 2d
> >> >> >> >>> >> >> >> >> > v1.7.2-rancher1
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > NAME             CLUSTER-IP   EXTERNAL-IP
> >> >> >> >>> >> >> >> >> > PORT(S)
> >> >> >> >>> >> >> >> >> > AGE
> >> >> >> >>> >> >> >> >> > svc/kubernetes   10.43.0.1    <none>
> >> >> >> >>> >> >> >> >> > 443/TCP
> >> >> >> >>> >> >> >> >> > 2d
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > Now when i run 'gk-deploy -g', in the Rancher
> >> >> >> >>> >> >> >> >> > console, i
> >> >> >> >>> >> >> >> >> > see
> >> >> >> >>> >> >> >> >> > the
> >> >> >> >>> >> >> >> >> > following
> >> >> >> >>> >> >> >> >> > error:
> >> >> >> >>> >> >> >> >> > Readiness probe failed: Failed to get D-Bus
> >> >> >> >>> >> >> >> >> > connection:
> >> >> >> >>> >> >> >> >> > Operation
> >> >> >> >>> >> >> >> >> > not
> >> >> >> >>> >> >> >> >> > permitted
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > From the attached gk-deploy_log i see that it
> >> >> >> >>> >> >> >> >> > failed
> >> >> >> >>> >> >> >> >> > at:
> >> >> >> >>> >> >> >> >> > Waiting for GlusterFS pods to start ... pods not
> >> >> >> >>> >> >> >> >> > found.
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > In the kube-templates/glusterfs-daemonset.yaml
> >> >> >> >>> >> >> >> >> > file,
> >> >> >> >>> >> >> >> >> > i
> >> >> >> >>> >> >> >> >> > see
> >> >> >> >>> >> >> >> >> > this
> >> >> >> >>> >> >> >> >> > for
> >> >> >> >>> >> >> >> >> > Readiness probe section:
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >> >>> >> >> >> >> >         readinessProbe:
> >> >> >> >>> >> >> >> >> >           timeoutSeconds: 3
> >> >> >> >>> >> >> >> >> >           initialDelaySeconds: 40
> >> >> >> >>> >> >> >> >> >           exec:
> >> >> >> >>> >> >> >> >> >             command:
> >> >> >> >>> >> >> >> >> >             - "/bin/bash"
> >> >> >> >>> >> >> >> >> >             - "-c"
> >> >> >> >>> >> >> >> >> >             - systemctl status glusterd.service
> >> >> >> >>> >> >> >> >> >           periodSeconds: 25
> >> >> >> >>> >> >> >> >> >           successThreshold: 1
> >> >> >> >>> >> >> >> >> >           failureThreshold: 15
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > Status after running gk-deploy:
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > [root at workstation deploy]# kubectl get
> >> >> >> >>> >> >> >> >> > nodes,pods,deployments,services
> >> >> >> >>> >> >> >> >> > NAME                                     STATUS
> >> >> >> >>> >> >> >> >> > AGE
> >> >> >> >>> >> >> >> >> > VERSION
> >> >> >> >>> >> >> >> >> > no/node-1.c.kubernetes-174104.internal   Ready
> >> >> >> >>> >> >> >> >> > 2d
> >> >> >> >>> >> >> >> >> > v1.7.2-rancher1
> >> >> >> >>> >> >> >> >> > no/node-2.c.kubernetes-174104.internal   Ready
> >> >> >> >>> >> >> >> >> > 2d
> >> >> >> >>> >> >> >> >> > v1.7.2-rancher1
> >> >> >> >>> >> >> >> >> > no/node-3.c.kubernetes-174104.internal   Ready
> >> >> >> >>> >> >> >> >> > 2d
> >> >> >> >>> >> >> >> >> > v1.7.2-rancher1
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > NAME                 READY     STATUS
> RESTARTS
> >> >> >> >>> >> >> >> >> > AGE
> >> >> >> >>> >> >> >> >> > po/glusterfs-0s440   0/1       Running   0
> >> >> >> >>> >> >> >> >> > 1m
> >> >> >> >>> >> >> >> >> > po/glusterfs-j7dgr   0/1       Running   0
> >> >> >> >>> >> >> >> >> > 1m
> >> >> >> >>> >> >> >> >> > po/glusterfs-p6jl3   0/1       Running   0
> >> >> >> >>> >> >> >> >> > 1m
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > NAME             CLUSTER-IP   EXTERNAL-IP
> >> >> >> >>> >> >> >> >> > PORT(S)
> >> >> >> >>> >> >> >> >> > AGE
> >> >> >> >>> >> >> >> >> > svc/kubernetes   10.43.0.1    <none>
> >> >> >> >>> >> >> >> >> > 443/TCP
> >> >> >> >>> >> >> >> >> > 2d
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > Also, from prerequisite perspective, i was also
> >> >> >> >>> >> >> >> >> > seeing
> >> >> >> >>> >> >> >> >> > this
> >> >> >> >>> >> >> >> >> > mentioned:
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > The following kernel modules must be loaded:
> >> >> >> >>> >> >> >> >> >  * dm_snapshot
> >> >> >> >>> >> >> >> >> >  * dm_mirror
> >> >> >> >>> >> >> >> >> >  * dm_thin_pool
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > Where exactly is this to be checked? On all
> >> >> >> >>> >> >> >> >> > Gluster
> >> >> >> >>> >> >> >> >> > server
> >> >> >> >>> >> >> >> >> > nodes?
> >> >> >> >>> >> >> >> >> > How
> >> >> >> >>> >> >> >> >> > can i
> >> >> >> >>> >> >> >> >> > check whether it's there?
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > I have attached topology.json and gk-deploy log
> >> >> >> >>> >> >> >> >> > for
> >> >> >> >>> >> >> >> >> > reference.
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > Does this issue has anything to do with the host
> >> >> >> >>> >> >> >> >> > OS
> >> >> >> >>> >> >> >> >> > (RancherOS)
> >> >> >> >>> >> >> >> >> > that
> >> >> >> >>> >> >> >> >> > i
> >> >> >> >>> >> >> >> >> > am
> >> >> >> >>> >> >> >> >> > using for Gluster nodes? Any idea how i can fix
> >> >> >> >>> >> >> >> >> > this?
> >> >> >> >>> >> >> >> >> > Any
> >> >> >> >>> >> >> >> >> > help
> >> >> >> >>> >> >> >> >> > will
> >> >> >> >>> >> >> >> >> > really
> >> >> >> >>> >> >> >> >> > be appreciated.
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > Thanks.
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > _______________________________________________
> >> >> >> >>> >> >> >> >> > heketi-devel mailing list
> >> >> >> >>> >> >> >> >> > heketi-devel at gluster.org
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >> > http://lists.gluster.org/
> mailman/listinfo/heketi-devel
> >> >> >> >>> >> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >> >
> >> >> >> >>> >> >> >
> >> >> >> >>> >> >> >
> >> >> >> >>> >> >
> >> >> >> >>> >> >
> >> >> >> >>> >
> >> >> >> >>> >
> >> >> >> >>
> >> >> >> >>
> >> >> >> >
> >> >> >
> >> >> >
> >> >
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/heketi-devel/attachments/20170905/d20eb5cb/attachment-0001.html>


More information about the heketi-devel mailing list