[heketi-devel] Unable to use Heketi setup to install Gluster for Kubernetes

Jose A. Rivera jarrpa at redhat.com
Tue Sep 5 13:30:37 UTC 2017


That is expected behavior. :) GlusterFS volumes are shared storage,
meaning the same data is presented to all accessing users. By default
(in our configuration) it replicates data across multiple bricks, but
all bricks are represented as a single filesystem and all replicas of
a file are presented as a single file. Thus, when you delete a file
you delete it and all its replicas.

On Tue, Sep 5, 2017 at 6:15 AM, Gaurav Chhabra <varuag.chhabra at gmail.com> wrote:
> Hi Jose,
>
>
> There is some progress after using the  following container:
> https://hub.docker.com/r/gluster/glusterfs-client/
>
> I created deployment with 5 replicas for Nginx in Kubernetes using your doc.
> Now as described in the last section of your doc, if i create file
> 'index.html' inside of a container running in a pod on node-a, i can see the
> same file becomes available on another pod running on node-b. However, if i
> delete the file on one node (say node-a), it becomes unavailable on all the
> three nodes. The file should have been accessible from other nodes even if i
> delete it from one pod on a given node. Correct me if i'm getting it wrong.
>
> I searched for index.html on the Gluster cluster and i could find it on all
> three nodes with same content. Please check the attached log.
>
> If i check /etc/fstab on any of the pods, i see the following:
>
> root at nginx-deploy-4253768608-w645w:/# cat /etc/fstab
> # UNCONFIGURED FSTAB FOR BASE SYSTEM
>
> Attached log files.
>
> Any idea why the file is not accessible from another pod on node-a or node-b
> if i delete it from a different pod present on another node-c?
>
>
> Regards,
> Gaurav
>
>
> On Tue, Sep 5, 2017 at 3:41 AM, Jose A. Rivera <jarrpa at redhat.com> wrote:
>>
>> Yup, I follow that repo and should be contributing to it in the
>> near-future. :) I've never played with the glusterfs-client image, but
>> here's hoping!
>>
>> On Mon, Sep 4, 2017 at 10:18 AM, Gaurav Chhabra
>> <varuag.chhabra at gmail.com> wrote:
>> > Actually the script is launching a container and i'm pretty sure that
>> > the
>> > command that's programmed as part of the container launch process is
>> > actually trying to run that mount command. Now the issue is that this
>> > again
>> > falls under Docker area. :( Generally, Docker container providers (in
>> > this
>> > case, nixel) provide a Dockerfile which specifies the command that's
>> > executed as part of the container launch process however, the problem in
>> > case of this container is that it's not maintained. It's 2 years old. If
>> > you
>> > will check active projects such as gluster/gluster-containers, you will
>> > see
>> > a proper Dockerfile explaining what it does.
>> >
>> > https://hub.docker.com/r/gluster/glusterfs-client/~/dockerfile/
>> >
>> > I am now planning to use the above container and see whether i'm lucky.
>> > If
>> > you will see the above link, it's running
>> >
>> > dnf install -y glusterfs-fuse
>> >
>> > Hopefully, this should take care of this section which asks to make sure
>> > that glusterfs-client is installed
>> >
>> >
>> > https://github.com/gluster/gluster-kubernetes/tree/master/docs/examples/dynamic_provisioning_external_gluster#validate-communication-and-gluster-prerequisites-on-the-kubernetes-nodes
>> >
>> >
>> >
>> >
>> > On Mon, Sep 4, 2017 at 7:33 PM, Jose A. Rivera <jarrpa at redhat.com>
>> > wrote:
>> >>
>> >> Something is trying to mount a GlusterFS volume, and it's not that
>> >> script. What's trying to mount?
>> >>
>> >> On Mon, Sep 4, 2017 at 8:52 AM, Gaurav Chhabra
>> >> <varuag.chhabra at gmail.com>
>> >> wrote:
>> >> > I am just running this bash script which will start the
>> >> > rancher-glusterfs-client container:
>> >> >
>> >> > [root at node-a ~]# cat rancher-glusterfs-client.sh
>> >> > #!/bin/bash
>> >> > sudo docker run --privileged \
>> >> >         --name=gluster-client \
>> >> >         -d \
>> >> >         -v /sys/fs/cgroup:/sys/fs/cgroup \
>> >> >         -v /var/log/glusterfs:/var/log/glusterfs \
>> >> >         --env GLUSTER_PEER=10.128.0.12,10.128.0.15,10.128.0.16 \
>> >> >         nixel/rancher-glusterfs-client
>> >> >
>> >> >
>> >> >
>> >> > On Mon, Sep 4, 2017 at 6:26 PM, Jose A. Rivera <jarrpa at redhat.com>
>> >> > wrote:
>> >> >>
>> >> >> What is the exact command you're running?
>> >> >>
>> >> >> On Mon, Sep 4, 2017 at 4:26 AM, Gaurav Chhabra
>> >> >> <varuag.chhabra at gmail.com>
>> >> >> wrote:
>> >> >> > Hi Jose,
>> >> >> >
>> >> >> >
>> >> >> > From this link, it seems mount.glusterfs might actually be present
>> >> >> > in
>> >> >> > the
>> >> >> > container that launched and quickly terminated.
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > https://unix.stackexchange.com/questions/312178/glusterfs-replicated-volume-mounting-issue
>> >> >> >
>> >> >> > If you check the question that the user posted, the same error
>> >> >> > (Mount
>> >> >> > failed) was reported that i sent you in the last email.
>> >> >> >
>> >> >> > After seeing the above, i checked /var/log/glusterfs on my host
>> >> >> > (RancherOS)
>> >> >> > but it was empty. I ran the container again but with explicit
>> >> >> > volume
>> >> >> > mount
>> >> >> > as shown below:
>> >> >> >
>> >> >> > [root at node-a ~]# cat rancher-glusterfs-client.sh
>> >> >> > #!/bin/bash
>> >> >> > sudo docker run --privileged \
>> >> >> >         --name=gluster-client \
>> >> >> >         -d \
>> >> >> >         -v /sys/fs/cgroup:/sys/fs/cgroup \
>> >> >> >         -v /var/log/glusterfs:/var/log/glusterfs \
>> >> >> >         --env GLUSTER_PEER=10.128.0.12,10.128.0.15,10.128.0.16 \
>> >> >> >         nixel/rancher-glusterfs-client
>> >> >> > This time, i could see a log file
>> >> >> > (/var/log/glusterfs/mnt-ranchervol.log)
>> >> >> > present. I have attached the content of the same. Also attached
>> >> >> > are
>> >> >> > logs
>> >> >> > from Heketi client/server (both on one node) and Gluster cluster.
>> >> >> >
>> >> >> >
>> >> >> > Regards,
>> >> >> > Gaurav
>> >> >> >
>> >> >> >
>> >> >> > On Mon, Sep 4, 2017 at 2:29 PM, Gaurav Chhabra
>> >> >> > <varuag.chhabra at gmail.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> Hi Jose,
>> >> >> >>
>> >> >> >>
>> >> >> >> I tried setting up things using the link you provided and i was
>> >> >> >> able
>> >> >> >> to
>> >> >> >> get all steps working for 3 node Gluster cluster, all running on
>> >> >> >> CentOS7,
>> >> >> >> without any issue. However, as expected, when i tried configuring
>> >> >> >> Kubernetes
>> >> >> >> by installing nixel/rancher-glusterfs-client container, i got
>> >> >> >> error:
>> >> >> >>
>> >> >> >> [root at node-a ~]# cat rancher-glusterfs-client.sh
>> >> >> >> #!/bin/bash
>> >> >> >> sudo docker run --privileged \
>> >> >> >>         --name=gluster-client \
>> >> >> >>         -d \
>> >> >> >>         -v /sys/fs/cgroup:/sys/fs/cgroup \
>> >> >> >>         --env GLUSTER_PEER=10.128.0.12,10.128.0.15,10.128.0.16 \
>> >> >> >>         nixel/rancher-glusterfs-client
>> >> >> >>
>> >> >> >> [root at node-a ~]# ./rancher-glusterfs-client.sh
>> >> >> >> ac069caccdce147d6f423fc566166345191dbc1b11f3416c66207a1fd11fda6b
>> >> >> >>
>> >> >> >> [root at node-a ~]# docker logs gluster-client
>> >> >> >> => Checking if I can reach GlusterFS node 10.128.0.12 ...
>> >> >> >> => GlusterFS node 10.128.0.12 is alive
>> >> >> >> => Mounting GlusterFS volume ranchervol from GlusterFS node
>> >> >> >> 10.128.0.12
>> >> >> >> ...
>> >> >> >> Mount failed. Please check the log file for more details.
>> >> >> >>
>> >> >> >> If i try running the next step as described in your link, i get
>> >> >> >> the
>> >> >> >> following:
>> >> >> >>
>> >> >> >> [root at node-a ~]# modprobe fuse
>> >> >> >> modprobe: module fuse not found in modules.dep
>> >> >> >>
>> >> >> >> Since the container failed to start, i could only check on the
>> >> >> >> host
>> >> >> >> (RancherOS) and i could only find two mount-related commands:
>> >> >> >> mount
>> >> >> >> &
>> >> >> >> mountpoint
>> >> >> >>
>> >> >> >> Any pointers?
>> >> >> >>
>> >> >> >>
>> >> >> >> Regards,
>> >> >> >> Gaurav
>> >> >> >>
>> >> >> >> On Sun, Sep 3, 2017 at 11:18 PM, Jose A. Rivera
>> >> >> >> <jarrpa at redhat.com>
>> >> >> >> wrote:
>> >> >> >>>
>> >> >> >>> Installing the glusterfs-client container should be fine. :) The
>> >> >> >>> main
>> >> >> >>> thing that's needed is that all your Kubernetes nodes need to
>> >> >> >>> have
>> >> >> >>> the
>> >> >> >>> "mount.glusterfs" command available so Kube can mount the
>> >> >> >>> GlusterFS
>> >> >> >>> volumes and present them to the pods.
>> >> >> >>>
>> >> >> >>> On Sun, Sep 3, 2017 at 12:14 PM, Gaurav Chhabra
>> >> >> >>> <varuag.chhabra at gmail.com> wrote:
>> >> >> >>> > Thanks Jose. The link you've suggested looks good but again it
>> >> >> >>> > expects
>> >> >> >>> > me to
>> >> >> >>> > install gluster-client on Kubernetes node and i fall into the
>> >> >> >>> > same
>> >> >> >>> > issue of
>> >> >> >>> > installing a container for glusterfs. Only difference is that
>> >> >> >>> > this
>> >> >> >>> > time
>> >> >> >>> > it's
>> >> >> >>> > glusterfs-client and not glusterfs-server. :)
>> >> >> >>> >
>> >> >> >>> > I will try this out and let you know tomorrow.
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>> > Regards,
>> >> >> >>> > Gaurav
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>> > On Sun, Sep 3, 2017 at 2:11 AM, Jose A. Rivera
>> >> >> >>> > <jarrpa at redhat.com>
>> >> >> >>> > wrote:
>> >> >> >>> >>
>> >> >> >>> >> Hey, no problem! I'm eager to learn more about different
>> >> >> >>> >> flavors
>> >> >> >>> >> of
>> >> >> >>> >> Linux, I just apologize for my relative inexperience with
>> >> >> >>> >> them.
>> >> >> >>> >> :)
>> >> >> >>> >>
>> >> >> >>> >> To that end, I will also admit I'm not very experienced with
>> >> >> >>> >> direct
>> >> >> >>> >> Docker myself. I understand the basic workflow and know some
>> >> >> >>> >> of
>> >> >> >>> >> the
>> >> >> >>> >> run options, but not having deep experience keeps me from
>> >> >> >>> >> having
>> >> >> >>> >> a
>> >> >> >>> >> better understanding of the patterns and consequences.
>> >> >> >>> >>
>> >> >> >>> >> Thus, I'd like to guide you in a direction I'd more apt to
>> >> >> >>> >> help
>> >> >> >>> >> you
>> >> >> >>> >> in
>> >> >> >>> >> right now. I know that you can't have multiple GlusterFS
>> >> >> >>> >> servers
>> >> >> >>> >> running on the same nodes, and I know that we have been
>> >> >> >>> >> successfully
>> >> >> >>> >> running several configurations using our
>> >> >> >>> >> gluster/gluster-centos
>> >> >> >>> >> image.
>> >> >> >>> >> If you follow the Kubernetes configuration on
>> >> >> >>> >> gluster-kubernetes,
>> >> >> >>> >> the
>> >> >> >>> >> pod/container is run privileged and with host networking, and
>> >> >> >>> >> we
>> >> >> >>> >> require that the node has all listed ports open, not just
>> >> >> >>> >> 2222.
>> >> >> >>> >> The
>> >> >> >>> >> sshd running in the container is listening on 2222, not 22,
>> >> >> >>> >> but
>> >> >> >>> >> it
>> >> >> >>> >> is
>> >> >> >>> >> also not really required if you're not doing geo-replication.
>> >> >> >>> >>
>> >> >> >>> >> Alternatively, you can indeed run GlusterFS outside of
>> >> >> >>> >> Kubernetes
>> >> >> >>> >> but
>> >> >> >>> >> still have Kubernetes apps access GlusterFS storage. The
>> >> >> >>> >> nodes
>> >> >> >>> >> can
>> >> >> >>> >> be
>> >> >> >>> >> anything you want, they just need to be running GlusterFS and
>> >> >> >>> >> you
>> >> >> >>> >> need
>> >> >> >>> >> a heketi service managing them. Here is an example of how to
>> >> >> >>> >> set
>> >> >> >>> >> this
>> >> >> >>> >> up using CentOS:
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >> https://github.com/gluster/gluster-kubernetes/tree/master/docs/examples/dynamic_provisioning_external_gluster
>> >> >> >>> >>
>> >> >> >>> >> Hope this is at least leading you in a useful direction. :)
>> >> >> >>> >>
>> >> >> >>> >> --Jose
>> >> >> >>> >>
>> >> >> >>> >> On Sat, Sep 2, 2017 at 3:16 PM, Gaurav Chhabra
>> >> >> >>> >> <varuag.chhabra at gmail.com>
>> >> >> >>> >> wrote:
>> >> >> >>> >> > Hi Jose,
>> >> >> >>> >> >
>> >> >> >>> >> > webcenter/rancher-glusterfs-server is actually a container
>> >> >> >>> >> > provided
>> >> >> >>> >> > by
>> >> >> >>> >> > Sebastien, its maintainer. It's a Docker container which
>> >> >> >>> >> > has
>> >> >> >>> >> > GlusterFS
>> >> >> >>> >> > server running within it. On the host i.e., RancherOS,
>> >> >> >>> >> > there
>> >> >> >>> >> > is
>> >> >> >>> >> > no
>> >> >> >>> >> > separate
>> >> >> >>> >> > GlusterFS server running because we cannot install anything
>> >> >> >>> >> > that
>> >> >> >>> >> > way.
>> >> >> >>> >> > Running using container is the only way so i started
>> >> >> >>> >> > ancher-glusterfs-server
>> >> >> >>> >> > container with the following parameters:
>> >> >> >>> >> >
>> >> >> >>> >> > [root at node-1 rancher]# cat gluster-server.sh
>> >> >> >>> >> > #!/bin/bash
>> >> >> >>> >> > sudo docker run --name=gluster-server -d \
>> >> >> >>> >> >         --env 'SERVICE_NAME=gluster' \
>> >> >> >>> >> >         --restart always \
>> >> >> >>> >> >         --publish 2222:22 \
>> >> >> >>> >> >         webcenter/rancher-glusterfs-server
>> >> >> >>> >> >
>> >> >> >>> >> > Here's the link to the dockerfile:
>> >> >> >>> >> >
>> >> >> >>> >> >
>> >> >> >>> >> >
>> >> >> >>> >> >
>> >> >> >>> >> >
>> >> >> >>> >> > https://hub.docker.com/r/webcenter/rancher-glusterfs-server/~/dockerfile/
>> >> >> >>> >> >
>> >> >> >>> >> > It's similar to other GlusteFS containers provided by other
>> >> >> >>> >> > maintainers
>> >> >> >>> >> > for
>> >> >> >>> >> > different OS. For example, for CentOS, we have
>> >> >> >>> >> >
>> >> >> >>> >> > https://hub.docker.com/r/gluster/gluster-centos/~/dockerfile/
>> >> >> >>> >> >
>> >> >> >>> >> > From what i understand, Heketi does support container based
>> >> >> >>> >> > GlusterFS
>> >> >> >>> >> > server
>> >> >> >>> >> > as mentioned in the prerequisite where it says:
>> >> >> >>> >> >
>> >> >> >>> >> > "Each node must have the following ports opened for
>> >> >> >>> >> > GlusterFS
>> >> >> >>> >> > communications:
>> >> >> >>> >> >  2222 - GlusterFS pod's sshd"
>> >> >> >>> >> >
>> >> >> >>> >> > That's the reason i've exposed port 2222 for 22 as shown
>> >> >> >>> >> > above.
>> >> >> >>> >> > Please
>> >> >> >>> >> > correct me if i misunderstood it.
>> >> >> >>> >> >
>> >> >> >>> >> > As soon as i run the above script (gluster-server.sh), it
>> >> >> >>> >> > automatically
>> >> >> >>> >> > creates the following directories on host. This should have
>> >> >> >>> >> > ideally
>> >> >> >>> >> > not
>> >> >> >>> >> > been
>> >> >> >>> >> > empty as you mentioned.
>> >> >> >>> >> >
>> >> >> >>> >> > /etc/glusterfs    /var/lib/glusterd    /var/log/glusterfs
>> >> >> >>> >> >
>> >> >> >>> >> > Just wanted to know in which circumstances do we get this
>> >> >> >>> >> > specific
>> >> >> >>> >> > error
>> >> >> >>> >> > (Failed to get D-Bus connection: Operation not permitted)
>> >> >> >>> >> > related
>> >> >> >>> >> > to
>> >> >> >>> >> > Readiness probe failing. Searching online took me to
>> >> >> >>> >> > discussions
>> >> >> >>> >> > around
>> >> >> >>> >> > running container in privileged mode and some directory to
>> >> >> >>> >> > be
>> >> >> >>> >> > mounted.
>> >> >> >>> >> > Based
>> >> >> >>> >> > on that, i also modified my container startup script to the
>> >> >> >>> >> > following:
>> >> >> >>> >> >
>> >> >> >>> >> > #!/bin/bash
>> >> >> >>> >> > sudo docker run --privileged \
>> >> >> >>> >> >         --name=gluster-server \
>> >> >> >>> >> >         -d \
>> >> >> >>> >> >         -v /sys/fs/cgroup:/sys/fs/cgroup \
>> >> >> >>> >> >         -v /etc/glusterfs:/etc/glusterfs \
>> >> >> >>> >> >         -v /var/lib/glusterd:/var/lib/glusterd \
>> >> >> >>> >> >         -v /var/log/glusterfs:/var/log/glusterfs \
>> >> >> >>> >> >         --env 'SERVICE_NAME=gluster' \
>> >> >> >>> >> >         --restart always \
>> >> >> >>> >> >         --publish 2222:22 \
>> >> >> >>> >> >         webcenter/rancher-glusterfs-server
>> >> >> >>> >> > Still, the issue persists.
>> >> >> >>> >> >
>> >> >> >>> >> > I also logged into the container and checked whether
>> >> >> >>> >> > systemctl
>> >> >> >>> >> > command
>> >> >> >>> >> > is
>> >> >> >>> >> > present. It was there but manualy running the command also
>> >> >> >>> >> > doesn't
>> >> >> >>> >> > work:
>> >> >> >>> >> >
>> >> >> >>> >> > [root at node-c ~]# docker exec -it gluster-server /bin/bash
>> >> >> >>> >> > root at 42150f203f80:/app# systemctl status glusterd.service
>> >> >> >>> >> > WARNING: terminal is not fully functional
>> >> >> >>> >> > Failed to connect to bus: No such file or directory
>> >> >> >>> >> >
>> >> >> >>> >> > Under section 'ADVANCED OPTIONS - Security/Host' in this
>> >> >> >>> >> > link,
>> >> >> >>> >> > it
>> >> >> >>> >> > talks
>> >> >> >>> >> > about SYS_ADMIN setting. Any idea how i can try this?
>> >> >> >>> >> >
>> >> >> >>> >> > Also, there was this mentioned in the Heketi setup page:
>> >> >> >>> >> > "If you are not able to deploy a hyper-converged GlusterFS
>> >> >> >>> >> > cluster,
>> >> >> >>> >> > you
>> >> >> >>> >> > must
>> >> >> >>> >> > have one running somewhere that the Kubernetes nodes can
>> >> >> >>> >> > access"
>> >> >> >>> >> >
>> >> >> >>> >> >>>> Does it mean running the three node Gluster cluster
>> >> >> >>> >> >>>> outside
>> >> >> >>> >> >>>> Kubernetes,
>> >> >> >>> >> >>>> may be on some VM running on RHEL/CentOS etc? If yes,
>> >> >> >>> >> >>>> then
>> >> >> >>> >> >>>> how
>> >> >> >>> >> >>>> will i
>> >> >> >>> >> >>>> be
>> >> >> >>> >> >>>> able to tell Gluster which volume from the Kubernetes
>> >> >> >>> >> >>>> cluster
>> >> >> >>> >> >>>> pod
>> >> >> >>> >> >>>> to
>> >> >> >>> >> >>>> sync?
>> >> >> >>> >> >>>> Any references?
>> >> >> >>> >> >
>> >> >> >>> >> >
>> >> >> >>> >> > I really appreciate your responses despite the fact that
>> >> >> >>> >> > you've
>> >> >> >>> >> > not
>> >> >> >>> >> > used
>> >> >> >>> >> > RancherOS but still trying to help.
>> >> >> >>> >> >
>> >> >> >>> >> >
>> >> >> >>> >> > Thanks,
>> >> >> >>> >> > Gaurav
>> >> >> >>> >> >
>> >> >> >>> >> >
>> >> >> >>> >> > On Sat, Sep 2, 2017 at 7:35 PM, Jose A. Rivera
>> >> >> >>> >> > <jarrpa at redhat.com>
>> >> >> >>> >> > wrote:
>> >> >> >>> >> >>
>> >> >> >>> >> >> I'm afraid I have no experience with RancherOS, so I may
>> >> >> >>> >> >> be
>> >> >> >>> >> >> missing
>> >> >> >>> >> >> some things about how it works. My primary experience is
>> >> >> >>> >> >> with
>> >> >> >>> >> >> Fedora,
>> >> >> >>> >> >> CentOS, and Ubuntu.
>> >> >> >>> >> >>
>> >> >> >>> >> >> What is webcenter/rancher-glusterfs-server? If it's
>> >> >> >>> >> >> running
>> >> >> >>> >> >> another
>> >> >> >>> >> >> glusterd then you probably don't want to be running it and
>> >> >> >>> >> >> should
>> >> >> >>> >> >> remove it from your systems.
>> >> >> >>> >> >>
>> >> >> >>> >> >> The glusterfs pods mount hostpath volumes from the host
>> >> >> >>> >> >> they're
>> >> >> >>> >> >> running on to persist their configuration. Thus anything
>> >> >> >>> >> >> they
>> >> >> >>> >> >> write
>> >> >> >>> >> >> to
>> >> >> >>> >> >> those directories should land on the host. If that's not
>> >> >> >>> >> >> happening
>> >> >> >>> >> >> then that's an additional problem.
>> >> >> >>> >> >>
>> >> >> >>> >> >> --Jose
>> >> >> >>> >> >>
>> >> >> >>> >> >> On Fri, Sep 1, 2017 at 11:17 PM, Gaurav Chhabra
>> >> >> >>> >> >> <varuag.chhabra at gmail.com> wrote:
>> >> >> >>> >> >> > Hi Jose,
>> >> >> >>> >> >> >
>> >> >> >>> >> >> >
>> >> >> >>> >> >> > I tried your suggestion but there is one confusion
>> >> >> >>> >> >> > regarding
>> >> >> >>> >> >> > point
>> >> >> >>> >> >> > #3.
>> >> >> >>> >> >> > Since
>> >> >> >>> >> >> > RancherOS has everything running as container, i am
>> >> >> >>> >> >> > running
>> >> >> >>> >> >> > webcenter/rancher-glusterfs-server container on all
>> >> >> >>> >> >> > three
>> >> >> >>> >> >> > nodes.
>> >> >> >>> >> >> > Now
>> >> >> >>> >> >> > as
>> >> >> >>> >> >> > far
>> >> >> >>> >> >> > as removing the directories are concerned, i hope you
>> >> >> >>> >> >> > meant
>> >> >> >>> >> >> > removing
>> >> >> >>> >> >> > them on
>> >> >> >>> >> >> > the host and _not_ from within the container. After
>> >> >> >>> >> >> > completing
>> >> >> >>> >> >> > step 1
>> >> >> >>> >> >> > and 2,
>> >> >> >>> >> >> > i checked the contents of all the directories that you
>> >> >> >>> >> >> > specified
>> >> >> >>> >> >> > in
>> >> >> >>> >> >> > point
>> >> >> >>> >> >> > #3. All were empty as you can see in the attached
>> >> >> >>> >> >> > other_logs.txt.
>> >> >> >>> >> >> > So
>> >> >> >>> >> >> > i
>> >> >> >>> >> >> > got
>> >> >> >>> >> >> > confused. I ran the deploy again but the issue persists.
>> >> >> >>> >> >> > Two
>> >> >> >>> >> >> > pods
>> >> >> >>> >> >> > show
>> >> >> >>> >> >> > Liveness error and the third one, Readiness error.
>> >> >> >>> >> >> >
>> >> >> >>> >> >> > I then tried removing those directories (Step #3) from
>> >> >> >>> >> >> > within
>> >> >> >>> >> >> > the
>> >> >> >>> >> >> > container
>> >> >> >>> >> >> > but getting following error:
>> >> >> >>> >> >> >
>> >> >> >>> >> >> > root at c0f8ab4d92a2:/app# rm -rf /var/lib/heketi
>> >> >> >>> >> >> > /etc/glusterfs
>> >> >> >>> >> >> > /var/lib/glusterd /var/log/glusterfs
>> >> >> >>> >> >> > rm: cannot remove '/var/lib/glusterd': Device or
>> >> >> >>> >> >> > resource
>> >> >> >>> >> >> > busy
>> >> >> >>> >> >> >
>> >> >> >>> >> >> >
>> >> >> >>> >> >> >
>> >> >> >>> >> >> > On Fri, Sep 1, 2017 at 8:21 PM, Jose A. Rivera
>> >> >> >>> >> >> > <jarrpa at redhat.com>
>> >> >> >>> >> >> > wrote:
>> >> >> >>> >> >> >>
>> >> >> >>> >> >> >> 1. Add a line to the ssh-exec portion of heketi.json of
>> >> >> >>> >> >> >> the
>> >> >> >>> >> >> >> sort:
>> >> >> >>> >> >> >>
>> >> >> >>> >> >> >> "sudo": true,
>> >> >> >>> >> >> >>
>> >> >> >>> >> >> >> 2. Run
>> >> >> >>> >> >> >>
>> >> >> >>> >> >> >> gk-deploy -g --abort
>> >> >> >>> >> >> >>
>> >> >> >>> >> >> >> 3. On the nodes that were/will be running GlusterFS
>> >> >> >>> >> >> >> pods,
>> >> >> >>> >> >> >> run:
>> >> >> >>> >> >> >>
>> >> >> >>> >> >> >> rm -rf /var/lib/heketi /etc/glusterfs /var/lib/glusterd
>> >> >> >>> >> >> >> /var/log/glusterfs
>> >> >> >>> >> >> >>
>> >> >> >>> >> >> >> Then try the deploy again.
>> >> >> >>> >> >> >>
>> >> >> >>> >> >> >> On Fri, Sep 1, 2017 at 6:05 AM, Gaurav Chhabra
>> >> >> >>> >> >> >> <varuag.chhabra at gmail.com>
>> >> >> >>> >> >> >> wrote:
>> >> >> >>> >> >> >> > Hi Jose,
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > Thanks for the reply. It seems the three gluster pods
>> >> >> >>> >> >> >> > might
>> >> >> >>> >> >> >> > have
>> >> >> >>> >> >> >> > been
>> >> >> >>> >> >> >> > a
>> >> >> >>> >> >> >> > copy-paste from another set of cluster where i was
>> >> >> >>> >> >> >> > trying
>> >> >> >>> >> >> >> > to
>> >> >> >>> >> >> >> > setup
>> >> >> >>> >> >> >> > the
>> >> >> >>> >> >> >> > same
>> >> >> >>> >> >> >> > thing using CentOS. Sorry for that. By the way, i did
>> >> >> >>> >> >> >> > check
>> >> >> >>> >> >> >> > for
>> >> >> >>> >> >> >> > the
>> >> >> >>> >> >> >> > kernel
>> >> >> >>> >> >> >> > modules and it seems it's already there. Also, i am
>> >> >> >>> >> >> >> > attaching
>> >> >> >>> >> >> >> > fresh
>> >> >> >>> >> >> >> > set
>> >> >> >>> >> >> >> > of
>> >> >> >>> >> >> >> > files because i created a new cluster and thought of
>> >> >> >>> >> >> >> > giving
>> >> >> >>> >> >> >> > it
>> >> >> >>> >> >> >> > a
>> >> >> >>> >> >> >> > try
>> >> >> >>> >> >> >> > again.
>> >> >> >>> >> >> >> > Issue still persists. :(
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > In heketi.json, there is a slight change w.r.t the
>> >> >> >>> >> >> >> > user
>> >> >> >>> >> >> >> > which
>> >> >> >>> >> >> >> > connects
>> >> >> >>> >> >> >> > to
>> >> >> >>> >> >> >> > glusterfs node using SSH. I am not sure how Heketi
>> >> >> >>> >> >> >> > was
>> >> >> >>> >> >> >> > using
>> >> >> >>> >> >> >> > root
>> >> >> >>> >> >> >> > user
>> >> >> >>> >> >> >> > to
>> >> >> >>> >> >> >> > login because i wasn't able to use root and do manual
>> >> >> >>> >> >> >> > SSH.
>> >> >> >>> >> >> >> > With
>> >> >> >>> >> >> >> > rancher
>> >> >> >>> >> >> >> > user, i can login successfully so i think this should
>> >> >> >>> >> >> >> > be
>> >> >> >>> >> >> >> > fine.
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > /etc/heketi/heketi.json:
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >>> >> >> >> >     "executor": "ssh",
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >     "_sshexec_comment": "SSH username and private key
>> >> >> >>> >> >> >> > file
>> >> >> >>> >> >> >> > information",
>> >> >> >>> >> >> >> >     "sshexec": {
>> >> >> >>> >> >> >> >       "keyfile": "/var/lib/heketi/.ssh/id_rsa",
>> >> >> >>> >> >> >> >       "user": "rancher",
>> >> >> >>> >> >> >> >       "port": "22",
>> >> >> >>> >> >> >> >       "fstab": "/etc/fstab"
>> >> >> >>> >> >> >> >     },
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > Before running gk-deploy:
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >>> >> >> >> > [root at workstation deploy]# kubectl get
>> >> >> >>> >> >> >> > nodes,pods,daemonset,deployments,services
>> >> >> >>> >> >> >> > NAME                                     STATUS
>> >> >> >>> >> >> >> > AGE
>> >> >> >>> >> >> >> > VERSION
>> >> >> >>> >> >> >> > no/node-a.c.kubernetes-174104.internal   Ready     3h
>> >> >> >>> >> >> >> > v1.7.2-rancher1
>> >> >> >>> >> >> >> > no/node-b.c.kubernetes-174104.internal   Ready     3h
>> >> >> >>> >> >> >> > v1.7.2-rancher1
>> >> >> >>> >> >> >> > no/node-c.c.kubernetes-174104.internal   Ready     3h
>> >> >> >>> >> >> >> > v1.7.2-rancher1
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > NAME             CLUSTER-IP   EXTERNAL-IP   PORT(S)
>> >> >> >>> >> >> >> > AGE
>> >> >> >>> >> >> >> > svc/kubernetes   10.43.0.1    <none>        443/TCP
>> >> >> >>> >> >> >> > 3h
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > After running gk-deploy:
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >>> >> >> >> > [root at workstation messagegc]# kubectl get
>> >> >> >>> >> >> >> > nodes,pods,daemonset,deployments,services
>> >> >> >>> >> >> >> > NAME                                     STATUS
>> >> >> >>> >> >> >> > AGE
>> >> >> >>> >> >> >> > VERSION
>> >> >> >>> >> >> >> > no/node-a.c.kubernetes-174104.internal   Ready     3h
>> >> >> >>> >> >> >> > v1.7.2-rancher1
>> >> >> >>> >> >> >> > no/node-b.c.kubernetes-174104.internal   Ready     3h
>> >> >> >>> >> >> >> > v1.7.2-rancher1
>> >> >> >>> >> >> >> > no/node-c.c.kubernetes-174104.internal   Ready     3h
>> >> >> >>> >> >> >> > v1.7.2-rancher1
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > NAME                 READY     STATUS    RESTARTS
>> >> >> >>> >> >> >> > AGE
>> >> >> >>> >> >> >> > po/glusterfs-0j9l5   0/1       Running   0
>> >> >> >>> >> >> >> > 2m
>> >> >> >>> >> >> >> > po/glusterfs-gqz4c   0/1       Running   0
>> >> >> >>> >> >> >> > 2m
>> >> >> >>> >> >> >> > po/glusterfs-gxvcb   0/1       Running   0
>> >> >> >>> >> >> >> > 2m
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > NAME           DESIRED   CURRENT   READY
>> >> >> >>> >> >> >> > UP-TO-DATE
>> >> >> >>> >> >> >> > AVAILABLE
>> >> >> >>> >> >> >> > NODE-SELECTOR           AGE
>> >> >> >>> >> >> >> > ds/glusterfs   3         3         0         3
>> >> >> >>> >> >> >> > 0
>> >> >> >>> >> >> >> > storagenode=glusterfs   2m
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > NAME             CLUSTER-IP   EXTERNAL-IP   PORT(S)
>> >> >> >>> >> >> >> > AGE
>> >> >> >>> >> >> >> > svc/kubernetes   10.43.0.1    <none>        443/TCP
>> >> >> >>> >> >> >> > 3h
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > Kernel module check on all three nodes:
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >>> >> >> >> > [root at node-a ~]# find /lib*/modules/$(uname -r) -name
>> >> >> >>> >> >> >> > *.ko
>> >> >> >>> >> >> >> > |
>> >> >> >>> >> >> >> > grep
>> >> >> >>> >> >> >> > 'thin-pool\|snapshot\|mirror' | xargs ls -ltr
>> >> >> >>> >> >> >> > -rw-r--r--    1 root     root         92310 Jun 26
>> >> >> >>> >> >> >> > 04:13
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > /lib64/modules/4.9.34-rancher/kernel/drivers/md/dm-thin-pool.ko
>> >> >> >>> >> >> >> > -rw-r--r--    1 root     root         56982 Jun 26
>> >> >> >>> >> >> >> > 04:13
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > /lib64/modules/4.9.34-rancher/kernel/drivers/md/dm-snapshot.ko
>> >> >> >>> >> >> >> > -rw-r--r--    1 root     root         27070 Jun 26
>> >> >> >>> >> >> >> > 04:13
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > /lib64/modules/4.9.34-rancher/kernel/drivers/md/dm-mirror.ko
>> >> >> >>> >> >> >> > -rw-r--r--    1 root     root         92310 Jun 26
>> >> >> >>> >> >> >> > 04:13
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > /lib/modules/4.9.34-rancher/kernel/drivers/md/dm-thin-pool.ko
>> >> >> >>> >> >> >> > -rw-r--r--    1 root     root         56982 Jun 26
>> >> >> >>> >> >> >> > 04:13
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > /lib/modules/4.9.34-rancher/kernel/drivers/md/dm-snapshot.ko
>> >> >> >>> >> >> >> > -rw-r--r--    1 root     root         27070 Jun 26
>> >> >> >>> >> >> >> > 04:13
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > /lib/modules/4.9.34-rancher/kernel/drivers/md/dm-mirror.ko
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > Error snapshot attached.
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > In my first mail, i checked that Readiness Probe
>> >> >> >>> >> >> >> > failure
>> >> >> >>> >> >> >> > check
>> >> >> >>> >> >> >> > has
>> >> >> >>> >> >> >> > this
>> >> >> >>> >> >> >> > code
>> >> >> >>> >> >> >> > in kube-templates/glusterfs-daemonset.yaml file:
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >>> >> >> >> >         readinessProbe:
>> >> >> >>> >> >> >> >           timeoutSeconds: 3
>> >> >> >>> >> >> >> >           initialDelaySeconds: 40
>> >> >> >>> >> >> >> >           exec:
>> >> >> >>> >> >> >> >             command:
>> >> >> >>> >> >> >> >             - "/bin/bash"
>> >> >> >>> >> >> >> >             - "-c"
>> >> >> >>> >> >> >> >             - systemctl status glusterd.service
>> >> >> >>> >> >> >> >           periodSeconds: 25
>> >> >> >>> >> >> >> >           successThreshold: 1
>> >> >> >>> >> >> >> >           failureThreshold: 15
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > I tried logging into glustefs container on one of the
>> >> >> >>> >> >> >> > node
>> >> >> >>> >> >> >> > and
>> >> >> >>> >> >> >> > ran
>> >> >> >>> >> >> >> > the
>> >> >> >>> >> >> >> > above
>> >> >> >>> >> >> >> > command:
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > [root at node-a ~]# docker exec -it c0f8ab4d92a23b6df2
>> >> >> >>> >> >> >> > /bin/bash
>> >> >> >>> >> >> >> > root at c0f8ab4d92a2:/app# systemctl status
>> >> >> >>> >> >> >> > glusterd.service
>> >> >> >>> >> >> >> > WARNING: terminal is not fully functional
>> >> >> >>> >> >> >> > Failed to connect to bus: No such file or directory
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > Any check that i can do manually on nodes to debug
>> >> >> >>> >> >> >> > further?
>> >> >> >>> >> >> >> > Any
>> >> >> >>> >> >> >> > suggestions?
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> > On Thu, Aug 31, 2017 at 6:53 PM, Jose A. Rivera
>> >> >> >>> >> >> >> > <jarrpa at redhat.com>
>> >> >> >>> >> >> >> > wrote:
>> >> >> >>> >> >> >> >>
>> >> >> >>> >> >> >> >> Hey Gaurav,
>> >> >> >>> >> >> >> >>
>> >> >> >>> >> >> >> >> The kernel modules must be loaded on all nodes that
>> >> >> >>> >> >> >> >> will
>> >> >> >>> >> >> >> >> run
>> >> >> >>> >> >> >> >> heketi
>> >> >> >>> >> >> >> >> pods. Additionally, you must have at least three
>> >> >> >>> >> >> >> >> nodes
>> >> >> >>> >> >> >> >> specified
>> >> >> >>> >> >> >> >> in
>> >> >> >>> >> >> >> >> your topology file. I'm not sure how you're getting
>> >> >> >>> >> >> >> >> three
>> >> >> >>> >> >> >> >> gluster
>> >> >> >>> >> >> >> >> pods
>> >> >> >>> >> >> >> >> when you only have two nodes defined... :)
>> >> >> >>> >> >> >> >>
>> >> >> >>> >> >> >> >> --Jose
>> >> >> >>> >> >> >> >>
>> >> >> >>> >> >> >> >> On Wed, Aug 30, 2017 at 5:27 AM, Gaurav Chhabra
>> >> >> >>> >> >> >> >> <varuag.chhabra at gmail.com> wrote:
>> >> >> >>> >> >> >> >> > Hi,
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > I have the following setup in place:
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > 1 node    : RancherOS having Rancher application
>> >> >> >>> >> >> >> >> > for
>> >> >> >>> >> >> >> >> > Kubernetes
>> >> >> >>> >> >> >> >> > setup
>> >> >> >>> >> >> >> >> > 2 nodes  : RancherOS having Rancher agent
>> >> >> >>> >> >> >> >> > 1 node   : CentOS 7 workstation having kubectl
>> >> >> >>> >> >> >> >> > installed
>> >> >> >>> >> >> >> >> > and
>> >> >> >>> >> >> >> >> > folder
>> >> >> >>> >> >> >> >> > cloned/downloaded from
>> >> >> >>> >> >> >> >> > https://github.com/gluster/gluster-kubernetes
>> >> >> >>> >> >> >> >> > using
>> >> >> >>> >> >> >> >> > which i run Heketi setup (gk-deploy -g)
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > I also have rancher-glusterfs-server container
>> >> >> >>> >> >> >> >> > running
>> >> >> >>> >> >> >> >> > with
>> >> >> >>> >> >> >> >> > the
>> >> >> >>> >> >> >> >> > following
>> >> >> >>> >> >> >> >> > configuration:
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > ------------------------------------------------------------------
>> >> >> >>> >> >> >> >> > [root at node-1 rancher]# cat gluster-server.sh
>> >> >> >>> >> >> >> >> > #!/bin/bash
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > sudo docker run --name=gluster-server -d \
>> >> >> >>> >> >> >> >> >         --env 'SERVICE_NAME=gluster' \
>> >> >> >>> >> >> >> >> >         --restart always \
>> >> >> >>> >> >> >> >> >         --env 'GLUSTER_DATA=/srv/docker/gitlab' \
>> >> >> >>> >> >> >> >> >         --publish 2222:22 \
>> >> >> >>> >> >> >> >> >         webcenter/rancher-glusterfs-server
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > ------------------------------------------------------------------
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > In /etc/heketi/heketi.json, following is the only
>> >> >> >>> >> >> >> >> > modified
>> >> >> >>> >> >> >> >> > portion:
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > ------------------------------------------------------------------
>> >> >> >>> >> >> >> >> >     "executor": "ssh",
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >     "_sshexec_comment": "SSH username and private
>> >> >> >>> >> >> >> >> > key
>> >> >> >>> >> >> >> >> > file
>> >> >> >>> >> >> >> >> > information",
>> >> >> >>> >> >> >> >> >     "sshexec": {
>> >> >> >>> >> >> >> >> >       "keyfile": "/var/lib/heketi/.ssh/id_rsa",
>> >> >> >>> >> >> >> >> >       "user": "root",
>> >> >> >>> >> >> >> >> >       "port": "22",
>> >> >> >>> >> >> >> >> >       "fstab": "/etc/fstab"
>> >> >> >>> >> >> >> >> >     },
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > ------------------------------------------------------------------
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > Status before running gk-deploy:
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > [root at workstation deploy]# kubectl get
>> >> >> >>> >> >> >> >> > nodes,pods,services,deployments
>> >> >> >>> >> >> >> >> > NAME                                     STATUS
>> >> >> >>> >> >> >> >> > AGE
>> >> >> >>> >> >> >> >> > VERSION
>> >> >> >>> >> >> >> >> > no/node-1.c.kubernetes-174104.internal   Ready
>> >> >> >>> >> >> >> >> > 2d
>> >> >> >>> >> >> >> >> > v1.7.2-rancher1
>> >> >> >>> >> >> >> >> > no/node-2.c.kubernetes-174104.internal   Ready
>> >> >> >>> >> >> >> >> > 2d
>> >> >> >>> >> >> >> >> > v1.7.2-rancher1
>> >> >> >>> >> >> >> >> > no/node-3.c.kubernetes-174104.internal   Ready
>> >> >> >>> >> >> >> >> > 2d
>> >> >> >>> >> >> >> >> > v1.7.2-rancher1
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > NAME             CLUSTER-IP   EXTERNAL-IP
>> >> >> >>> >> >> >> >> > PORT(S)
>> >> >> >>> >> >> >> >> > AGE
>> >> >> >>> >> >> >> >> > svc/kubernetes   10.43.0.1    <none>
>> >> >> >>> >> >> >> >> > 443/TCP
>> >> >> >>> >> >> >> >> > 2d
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > Now when i run 'gk-deploy -g', in the Rancher
>> >> >> >>> >> >> >> >> > console, i
>> >> >> >>> >> >> >> >> > see
>> >> >> >>> >> >> >> >> > the
>> >> >> >>> >> >> >> >> > following
>> >> >> >>> >> >> >> >> > error:
>> >> >> >>> >> >> >> >> > Readiness probe failed: Failed to get D-Bus
>> >> >> >>> >> >> >> >> > connection:
>> >> >> >>> >> >> >> >> > Operation
>> >> >> >>> >> >> >> >> > not
>> >> >> >>> >> >> >> >> > permitted
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > From the attached gk-deploy_log i see that it
>> >> >> >>> >> >> >> >> > failed
>> >> >> >>> >> >> >> >> > at:
>> >> >> >>> >> >> >> >> > Waiting for GlusterFS pods to start ... pods not
>> >> >> >>> >> >> >> >> > found.
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > In the kube-templates/glusterfs-daemonset.yaml
>> >> >> >>> >> >> >> >> > file,
>> >> >> >>> >> >> >> >> > i
>> >> >> >>> >> >> >> >> > see
>> >> >> >>> >> >> >> >> > this
>> >> >> >>> >> >> >> >> > for
>> >> >> >>> >> >> >> >> > Readiness probe section:
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > ------------------------------------------------------------------
>> >> >> >>> >> >> >> >> >         readinessProbe:
>> >> >> >>> >> >> >> >> >           timeoutSeconds: 3
>> >> >> >>> >> >> >> >> >           initialDelaySeconds: 40
>> >> >> >>> >> >> >> >> >           exec:
>> >> >> >>> >> >> >> >> >             command:
>> >> >> >>> >> >> >> >> >             - "/bin/bash"
>> >> >> >>> >> >> >> >> >             - "-c"
>> >> >> >>> >> >> >> >> >             - systemctl status glusterd.service
>> >> >> >>> >> >> >> >> >           periodSeconds: 25
>> >> >> >>> >> >> >> >> >           successThreshold: 1
>> >> >> >>> >> >> >> >> >           failureThreshold: 15
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > ------------------------------------------------------------------
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > Status after running gk-deploy:
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > [root at workstation deploy]# kubectl get
>> >> >> >>> >> >> >> >> > nodes,pods,deployments,services
>> >> >> >>> >> >> >> >> > NAME                                     STATUS
>> >> >> >>> >> >> >> >> > AGE
>> >> >> >>> >> >> >> >> > VERSION
>> >> >> >>> >> >> >> >> > no/node-1.c.kubernetes-174104.internal   Ready
>> >> >> >>> >> >> >> >> > 2d
>> >> >> >>> >> >> >> >> > v1.7.2-rancher1
>> >> >> >>> >> >> >> >> > no/node-2.c.kubernetes-174104.internal   Ready
>> >> >> >>> >> >> >> >> > 2d
>> >> >> >>> >> >> >> >> > v1.7.2-rancher1
>> >> >> >>> >> >> >> >> > no/node-3.c.kubernetes-174104.internal   Ready
>> >> >> >>> >> >> >> >> > 2d
>> >> >> >>> >> >> >> >> > v1.7.2-rancher1
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > NAME                 READY     STATUS    RESTARTS
>> >> >> >>> >> >> >> >> > AGE
>> >> >> >>> >> >> >> >> > po/glusterfs-0s440   0/1       Running   0
>> >> >> >>> >> >> >> >> > 1m
>> >> >> >>> >> >> >> >> > po/glusterfs-j7dgr   0/1       Running   0
>> >> >> >>> >> >> >> >> > 1m
>> >> >> >>> >> >> >> >> > po/glusterfs-p6jl3   0/1       Running   0
>> >> >> >>> >> >> >> >> > 1m
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > NAME             CLUSTER-IP   EXTERNAL-IP
>> >> >> >>> >> >> >> >> > PORT(S)
>> >> >> >>> >> >> >> >> > AGE
>> >> >> >>> >> >> >> >> > svc/kubernetes   10.43.0.1    <none>
>> >> >> >>> >> >> >> >> > 443/TCP
>> >> >> >>> >> >> >> >> > 2d
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > Also, from prerequisite perspective, i was also
>> >> >> >>> >> >> >> >> > seeing
>> >> >> >>> >> >> >> >> > this
>> >> >> >>> >> >> >> >> > mentioned:
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > The following kernel modules must be loaded:
>> >> >> >>> >> >> >> >> >  * dm_snapshot
>> >> >> >>> >> >> >> >> >  * dm_mirror
>> >> >> >>> >> >> >> >> >  * dm_thin_pool
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > Where exactly is this to be checked? On all
>> >> >> >>> >> >> >> >> > Gluster
>> >> >> >>> >> >> >> >> > server
>> >> >> >>> >> >> >> >> > nodes?
>> >> >> >>> >> >> >> >> > How
>> >> >> >>> >> >> >> >> > can i
>> >> >> >>> >> >> >> >> > check whether it's there?
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > I have attached topology.json and gk-deploy log
>> >> >> >>> >> >> >> >> > for
>> >> >> >>> >> >> >> >> > reference.
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > Does this issue has anything to do with the host
>> >> >> >>> >> >> >> >> > OS
>> >> >> >>> >> >> >> >> > (RancherOS)
>> >> >> >>> >> >> >> >> > that
>> >> >> >>> >> >> >> >> > i
>> >> >> >>> >> >> >> >> > am
>> >> >> >>> >> >> >> >> > using for Gluster nodes? Any idea how i can fix
>> >> >> >>> >> >> >> >> > this?
>> >> >> >>> >> >> >> >> > Any
>> >> >> >>> >> >> >> >> > help
>> >> >> >>> >> >> >> >> > will
>> >> >> >>> >> >> >> >> > really
>> >> >> >>> >> >> >> >> > be appreciated.
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > Thanks.
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > _______________________________________________
>> >> >> >>> >> >> >> >> > heketi-devel mailing list
>> >> >> >>> >> >> >> >> > heketi-devel at gluster.org
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >> > http://lists.gluster.org/mailman/listinfo/heketi-devel
>> >> >> >>> >> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >> >
>> >> >> >>> >> >> >
>> >> >> >>> >> >> >
>> >> >> >>> >> >
>> >> >> >>> >> >
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>
>> >> >> >>
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>
>


More information about the heketi-devel mailing list