[heketi-devel] Unable to use Heketi setup to install Gluster for Kubernetes

Gaurav Chhabra varuag.chhabra at gmail.com
Mon Sep 4 15:18:21 UTC 2017


Actually the script is launching a container and i'm pretty sure that the
command that's programmed as part of the container launch process is
actually trying to run that mount command. Now the issue is that this again
falls under Docker area. :( Generally, Docker container providers (in this
case, nixel <https://hub.docker.com/r/nixel/rancher-glusterfs-client/>)
provide a Dockerfile which specifies the command that's executed as part of
the container launch process however, the problem in case of this container
is that it's not maintained. It's 2 years old. If you will check active
projects such as gluster/gluster-containers
<https://github.com/gluster/gluster-containers>, you will see a proper
Dockerfile explaining what it does.

https://hub.docker.com/r/gluster/glusterfs-client/~/dockerfile/

I am now planning to use the above container and see whether i'm lucky. If
you will see the above link, it's running

dnf install -y glusterfs-fuse

Hopefully, this should take care of this section which asks to make sure
that glusterfs-client is installed

https://github.com/gluster/gluster-kubernetes/tree/master/docs/examples/dynamic_provisioning_external_gluster#validate-communication-and-gluster-prerequisites-on-the-kubernetes-nodes




On Mon, Sep 4, 2017 at 7:33 PM, Jose A. Rivera <jarrpa at redhat.com> wrote:

> Something is trying to mount a GlusterFS volume, and it's not that
> script. What's trying to mount?
>
> On Mon, Sep 4, 2017 at 8:52 AM, Gaurav Chhabra <varuag.chhabra at gmail.com>
> wrote:
> > I am just running this bash script which will start the
> > rancher-glusterfs-client container:
> >
> > [root at node-a ~]# cat rancher-glusterfs-client.sh
> > #!/bin/bash
> > sudo docker run --privileged \
> >         --name=gluster-client \
> >         -d \
> >         -v /sys/fs/cgroup:/sys/fs/cgroup \
> >         -v /var/log/glusterfs:/var/log/glusterfs \
> >         --env GLUSTER_PEER=10.128.0.12,10.128.0.15,10.128.0.16 \
> >         nixel/rancher-glusterfs-client
> >
> >
> >
> > On Mon, Sep 4, 2017 at 6:26 PM, Jose A. Rivera <jarrpa at redhat.com>
> wrote:
> >>
> >> What is the exact command you're running?
> >>
> >> On Mon, Sep 4, 2017 at 4:26 AM, Gaurav Chhabra <
> varuag.chhabra at gmail.com>
> >> wrote:
> >> > Hi Jose,
> >> >
> >> >
> >> > From this link, it seems mount.glusterfs might actually be present in
> >> > the
> >> > container that launched and quickly terminated.
> >> >
> >> >
> >> > https://unix.stackexchange.com/questions/312178/
> glusterfs-replicated-volume-mounting-issue
> >> >
> >> > If you check the question that the user posted, the same error (Mount
> >> > failed) was reported that i sent you in the last email.
> >> >
> >> > After seeing the above, i checked /var/log/glusterfs on my host
> >> > (RancherOS)
> >> > but it was empty. I ran the container again but with explicit volume
> >> > mount
> >> > as shown below:
> >> >
> >> > [root at node-a ~]# cat rancher-glusterfs-client.sh
> >> > #!/bin/bash
> >> > sudo docker run --privileged \
> >> >         --name=gluster-client \
> >> >         -d \
> >> >         -v /sys/fs/cgroup:/sys/fs/cgroup \
> >> >         -v /var/log/glusterfs:/var/log/glusterfs \
> >> >         --env GLUSTER_PEER=10.128.0.12,10.128.0.15,10.128.0.16 \
> >> >         nixel/rancher-glusterfs-client
> >> > This time, i could see a log file
> >> > (/var/log/glusterfs/mnt-ranchervol.log)
> >> > present. I have attached the content of the same. Also attached are
> logs
> >> > from Heketi client/server (both on one node) and Gluster cluster.
> >> >
> >> >
> >> > Regards,
> >> > Gaurav
> >> >
> >> >
> >> > On Mon, Sep 4, 2017 at 2:29 PM, Gaurav Chhabra
> >> > <varuag.chhabra at gmail.com>
> >> > wrote:
> >> >>
> >> >> Hi Jose,
> >> >>
> >> >>
> >> >> I tried setting up things using the link you provided and i was able
> to
> >> >> get all steps working for 3 node Gluster cluster, all running on
> >> >> CentOS7,
> >> >> without any issue. However, as expected, when i tried configuring
> >> >> Kubernetes
> >> >> by installing nixel/rancher-glusterfs-client container, i got error:
> >> >>
> >> >> [root at node-a ~]# cat rancher-glusterfs-client.sh
> >> >> #!/bin/bash
> >> >> sudo docker run --privileged \
> >> >>         --name=gluster-client \
> >> >>         -d \
> >> >>         -v /sys/fs/cgroup:/sys/fs/cgroup \
> >> >>         --env GLUSTER_PEER=10.128.0.12,10.128.0.15,10.128.0.16 \
> >> >>         nixel/rancher-glusterfs-client
> >> >>
> >> >> [root at node-a ~]# ./rancher-glusterfs-client.sh
> >> >> ac069caccdce147d6f423fc566166345191dbc1b11f3416c66207a1fd11fda6b
> >> >>
> >> >> [root at node-a ~]# docker logs gluster-client
> >> >> => Checking if I can reach GlusterFS node 10.128.0.12 ...
> >> >> => GlusterFS node 10.128.0.12 is alive
> >> >> => Mounting GlusterFS volume ranchervol from GlusterFS node
> 10.128.0.12
> >> >> ...
> >> >> Mount failed. Please check the log file for more details.
> >> >>
> >> >> If i try running the next step as described in your link, i get the
> >> >> following:
> >> >>
> >> >> [root at node-a ~]# modprobe fuse
> >> >> modprobe: module fuse not found in modules.dep
> >> >>
> >> >> Since the container failed to start, i could only check on the host
> >> >> (RancherOS) and i could only find two mount-related commands: mount &
> >> >> mountpoint
> >> >>
> >> >> Any pointers?
> >> >>
> >> >>
> >> >> Regards,
> >> >> Gaurav
> >> >>
> >> >> On Sun, Sep 3, 2017 at 11:18 PM, Jose A. Rivera <jarrpa at redhat.com>
> >> >> wrote:
> >> >>>
> >> >>> Installing the glusterfs-client container should be fine. :) The
> main
> >> >>> thing that's needed is that all your Kubernetes nodes need to have
> the
> >> >>> "mount.glusterfs" command available so Kube can mount the GlusterFS
> >> >>> volumes and present them to the pods.
> >> >>>
> >> >>> On Sun, Sep 3, 2017 at 12:14 PM, Gaurav Chhabra
> >> >>> <varuag.chhabra at gmail.com> wrote:
> >> >>> > Thanks Jose. The link you've suggested looks good but again it
> >> >>> > expects
> >> >>> > me to
> >> >>> > install gluster-client on Kubernetes node and i fall into the same
> >> >>> > issue of
> >> >>> > installing a container for glusterfs. Only difference is that this
> >> >>> > time
> >> >>> > it's
> >> >>> > glusterfs-client and not glusterfs-server. :)
> >> >>> >
> >> >>> > I will try this out and let you know tomorrow.
> >> >>> >
> >> >>> >
> >> >>> > Regards,
> >> >>> > Gaurav
> >> >>> >
> >> >>> >
> >> >>> > On Sun, Sep 3, 2017 at 2:11 AM, Jose A. Rivera <jarrpa at redhat.com
> >
> >> >>> > wrote:
> >> >>> >>
> >> >>> >> Hey, no problem! I'm eager to learn more about different flavors
> of
> >> >>> >> Linux, I just apologize for my relative inexperience with them.
> :)
> >> >>> >>
> >> >>> >> To that end, I will also admit I'm not very experienced with
> direct
> >> >>> >> Docker myself. I understand the basic workflow and know some of
> the
> >> >>> >> run options, but not having deep experience keeps me from having
> a
> >> >>> >> better understanding of the patterns and consequences.
> >> >>> >>
> >> >>> >> Thus, I'd like to guide you in a direction I'd more apt to help
> you
> >> >>> >> in
> >> >>> >> right now. I know that you can't have multiple GlusterFS servers
> >> >>> >> running on the same nodes, and I know that we have been
> >> >>> >> successfully
> >> >>> >> running several configurations using our gluster/gluster-centos
> >> >>> >> image.
> >> >>> >> If you follow the Kubernetes configuration on gluster-kubernetes,
> >> >>> >> the
> >> >>> >> pod/container is run privileged and with host networking, and we
> >> >>> >> require that the node has all listed ports open, not just 2222.
> The
> >> >>> >> sshd running in the container is listening on 2222, not 22, but
> it
> >> >>> >> is
> >> >>> >> also not really required if you're not doing geo-replication.
> >> >>> >>
> >> >>> >> Alternatively, you can indeed run GlusterFS outside of Kubernetes
> >> >>> >> but
> >> >>> >> still have Kubernetes apps access GlusterFS storage. The nodes
> can
> >> >>> >> be
> >> >>> >> anything you want, they just need to be running GlusterFS and you
> >> >>> >> need
> >> >>> >> a heketi service managing them. Here is an example of how to set
> >> >>> >> this
> >> >>> >> up using CentOS:
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> https://github.com/gluster/gluster-kubernetes/tree/
> master/docs/examples/dynamic_provisioning_external_gluster
> >> >>> >>
> >> >>> >> Hope this is at least leading you in a useful direction. :)
> >> >>> >>
> >> >>> >> --Jose
> >> >>> >>
> >> >>> >> On Sat, Sep 2, 2017 at 3:16 PM, Gaurav Chhabra
> >> >>> >> <varuag.chhabra at gmail.com>
> >> >>> >> wrote:
> >> >>> >> > Hi Jose,
> >> >>> >> >
> >> >>> >> > webcenter/rancher-glusterfs-server is actually a container
> >> >>> >> > provided
> >> >>> >> > by
> >> >>> >> > Sebastien, its maintainer. It's a Docker container which has
> >> >>> >> > GlusterFS
> >> >>> >> > server running within it. On the host i.e., RancherOS, there is
> >> >>> >> > no
> >> >>> >> > separate
> >> >>> >> > GlusterFS server running because we cannot install anything
> that
> >> >>> >> > way.
> >> >>> >> > Running using container is the only way so i started
> >> >>> >> > ancher-glusterfs-server
> >> >>> >> > container with the following parameters:
> >> >>> >> >
> >> >>> >> > [root at node-1 rancher]# cat gluster-server.sh
> >> >>> >> > #!/bin/bash
> >> >>> >> > sudo docker run --name=gluster-server -d \
> >> >>> >> >         --env 'SERVICE_NAME=gluster' \
> >> >>> >> >         --restart always \
> >> >>> >> >         --publish 2222:22 \
> >> >>> >> >         webcenter/rancher-glusterfs-server
> >> >>> >> >
> >> >>> >> > Here's the link to the dockerfile:
> >> >>> >> >
> >> >>> >> >
> >> >>> >> >
> >> >>> >> > https://hub.docker.com/r/webcenter/rancher-glusterfs-
> server/~/dockerfile/
> >> >>> >> >
> >> >>> >> > It's similar to other GlusteFS containers provided by other
> >> >>> >> > maintainers
> >> >>> >> > for
> >> >>> >> > different OS. For example, for CentOS, we have
> >> >>> >> > https://hub.docker.com/r/gluster/gluster-centos/~/dockerfile/
> >> >>> >> >
> >> >>> >> > From what i understand, Heketi does support container based
> >> >>> >> > GlusterFS
> >> >>> >> > server
> >> >>> >> > as mentioned in the prerequisite where it says:
> >> >>> >> >
> >> >>> >> > "Each node must have the following ports opened for GlusterFS
> >> >>> >> > communications:
> >> >>> >> >  2222 - GlusterFS pod's sshd"
> >> >>> >> >
> >> >>> >> > That's the reason i've exposed port 2222 for 22 as shown above.
> >> >>> >> > Please
> >> >>> >> > correct me if i misunderstood it.
> >> >>> >> >
> >> >>> >> > As soon as i run the above script (gluster-server.sh), it
> >> >>> >> > automatically
> >> >>> >> > creates the following directories on host. This should have
> >> >>> >> > ideally
> >> >>> >> > not
> >> >>> >> > been
> >> >>> >> > empty as you mentioned.
> >> >>> >> >
> >> >>> >> > /etc/glusterfs    /var/lib/glusterd    /var/log/glusterfs
> >> >>> >> >
> >> >>> >> > Just wanted to know in which circumstances do we get this
> >> >>> >> > specific
> >> >>> >> > error
> >> >>> >> > (Failed to get D-Bus connection: Operation not permitted)
> related
> >> >>> >> > to
> >> >>> >> > Readiness probe failing. Searching online took me to
> discussions
> >> >>> >> > around
> >> >>> >> > running container in privileged mode and some directory to be
> >> >>> >> > mounted.
> >> >>> >> > Based
> >> >>> >> > on that, i also modified my container startup script to the
> >> >>> >> > following:
> >> >>> >> >
> >> >>> >> > #!/bin/bash
> >> >>> >> > sudo docker run --privileged \
> >> >>> >> >         --name=gluster-server \
> >> >>> >> >         -d \
> >> >>> >> >         -v /sys/fs/cgroup:/sys/fs/cgroup \
> >> >>> >> >         -v /etc/glusterfs:/etc/glusterfs \
> >> >>> >> >         -v /var/lib/glusterd:/var/lib/glusterd \
> >> >>> >> >         -v /var/log/glusterfs:/var/log/glusterfs \
> >> >>> >> >         --env 'SERVICE_NAME=gluster' \
> >> >>> >> >         --restart always \
> >> >>> >> >         --publish 2222:22 \
> >> >>> >> >         webcenter/rancher-glusterfs-server
> >> >>> >> > Still, the issue persists.
> >> >>> >> >
> >> >>> >> > I also logged into the container and checked whether systemctl
> >> >>> >> > command
> >> >>> >> > is
> >> >>> >> > present. It was there but manualy running the command also
> >> >>> >> > doesn't
> >> >>> >> > work:
> >> >>> >> >
> >> >>> >> > [root at node-c ~]# docker exec -it gluster-server /bin/bash
> >> >>> >> > root at 42150f203f80:/app# systemctl status glusterd.service
> >> >>> >> > WARNING: terminal is not fully functional
> >> >>> >> > Failed to connect to bus: No such file or directory
> >> >>> >> >
> >> >>> >> > Under section 'ADVANCED OPTIONS - Security/Host' in this link,
> it
> >> >>> >> > talks
> >> >>> >> > about SYS_ADMIN setting. Any idea how i can try this?
> >> >>> >> >
> >> >>> >> > Also, there was this mentioned in the Heketi setup page:
> >> >>> >> > "If you are not able to deploy a hyper-converged GlusterFS
> >> >>> >> > cluster,
> >> >>> >> > you
> >> >>> >> > must
> >> >>> >> > have one running somewhere that the Kubernetes nodes can
> access"
> >> >>> >> >
> >> >>> >> >>>> Does it mean running the three node Gluster cluster outside
> >> >>> >> >>>> Kubernetes,
> >> >>> >> >>>> may be on some VM running on RHEL/CentOS etc? If yes, then
> how
> >> >>> >> >>>> will i
> >> >>> >> >>>> be
> >> >>> >> >>>> able to tell Gluster which volume from the Kubernetes
> cluster
> >> >>> >> >>>> pod
> >> >>> >> >>>> to
> >> >>> >> >>>> sync?
> >> >>> >> >>>> Any references?
> >> >>> >> >
> >> >>> >> >
> >> >>> >> > I really appreciate your responses despite the fact that you've
> >> >>> >> > not
> >> >>> >> > used
> >> >>> >> > RancherOS but still trying to help.
> >> >>> >> >
> >> >>> >> >
> >> >>> >> > Thanks,
> >> >>> >> > Gaurav
> >> >>> >> >
> >> >>> >> >
> >> >>> >> > On Sat, Sep 2, 2017 at 7:35 PM, Jose A. Rivera
> >> >>> >> > <jarrpa at redhat.com>
> >> >>> >> > wrote:
> >> >>> >> >>
> >> >>> >> >> I'm afraid I have no experience with RancherOS, so I may be
> >> >>> >> >> missing
> >> >>> >> >> some things about how it works. My primary experience is with
> >> >>> >> >> Fedora,
> >> >>> >> >> CentOS, and Ubuntu.
> >> >>> >> >>
> >> >>> >> >> What is webcenter/rancher-glusterfs-server? If it's running
> >> >>> >> >> another
> >> >>> >> >> glusterd then you probably don't want to be running it and
> >> >>> >> >> should
> >> >>> >> >> remove it from your systems.
> >> >>> >> >>
> >> >>> >> >> The glusterfs pods mount hostpath volumes from the host
> they're
> >> >>> >> >> running on to persist their configuration. Thus anything they
> >> >>> >> >> write
> >> >>> >> >> to
> >> >>> >> >> those directories should land on the host. If that's not
> >> >>> >> >> happening
> >> >>> >> >> then that's an additional problem.
> >> >>> >> >>
> >> >>> >> >> --Jose
> >> >>> >> >>
> >> >>> >> >> On Fri, Sep 1, 2017 at 11:17 PM, Gaurav Chhabra
> >> >>> >> >> <varuag.chhabra at gmail.com> wrote:
> >> >>> >> >> > Hi Jose,
> >> >>> >> >> >
> >> >>> >> >> >
> >> >>> >> >> > I tried your suggestion but there is one confusion regarding
> >> >>> >> >> > point
> >> >>> >> >> > #3.
> >> >>> >> >> > Since
> >> >>> >> >> > RancherOS has everything running as container, i am running
> >> >>> >> >> > webcenter/rancher-glusterfs-server container on all three
> >> >>> >> >> > nodes.
> >> >>> >> >> > Now
> >> >>> >> >> > as
> >> >>> >> >> > far
> >> >>> >> >> > as removing the directories are concerned, i hope you meant
> >> >>> >> >> > removing
> >> >>> >> >> > them on
> >> >>> >> >> > the host and _not_ from within the container. After
> completing
> >> >>> >> >> > step 1
> >> >>> >> >> > and 2,
> >> >>> >> >> > i checked the contents of all the directories that you
> >> >>> >> >> > specified
> >> >>> >> >> > in
> >> >>> >> >> > point
> >> >>> >> >> > #3. All were empty as you can see in the attached
> >> >>> >> >> > other_logs.txt.
> >> >>> >> >> > So
> >> >>> >> >> > i
> >> >>> >> >> > got
> >> >>> >> >> > confused. I ran the deploy again but the issue persists. Two
> >> >>> >> >> > pods
> >> >>> >> >> > show
> >> >>> >> >> > Liveness error and the third one, Readiness error.
> >> >>> >> >> >
> >> >>> >> >> > I then tried removing those directories (Step #3) from
> within
> >> >>> >> >> > the
> >> >>> >> >> > container
> >> >>> >> >> > but getting following error:
> >> >>> >> >> >
> >> >>> >> >> > root at c0f8ab4d92a2:/app# rm -rf /var/lib/heketi
> /etc/glusterfs
> >> >>> >> >> > /var/lib/glusterd /var/log/glusterfs
> >> >>> >> >> > rm: cannot remove '/var/lib/glusterd': Device or resource
> busy
> >> >>> >> >> >
> >> >>> >> >> >
> >> >>> >> >> >
> >> >>> >> >> > On Fri, Sep 1, 2017 at 8:21 PM, Jose A. Rivera
> >> >>> >> >> > <jarrpa at redhat.com>
> >> >>> >> >> > wrote:
> >> >>> >> >> >>
> >> >>> >> >> >> 1. Add a line to the ssh-exec portion of heketi.json of the
> >> >>> >> >> >> sort:
> >> >>> >> >> >>
> >> >>> >> >> >> "sudo": true,
> >> >>> >> >> >>
> >> >>> >> >> >> 2. Run
> >> >>> >> >> >>
> >> >>> >> >> >> gk-deploy -g --abort
> >> >>> >> >> >>
> >> >>> >> >> >> 3. On the nodes that were/will be running GlusterFS pods,
> >> >>> >> >> >> run:
> >> >>> >> >> >>
> >> >>> >> >> >> rm -rf /var/lib/heketi /etc/glusterfs /var/lib/glusterd
> >> >>> >> >> >> /var/log/glusterfs
> >> >>> >> >> >>
> >> >>> >> >> >> Then try the deploy again.
> >> >>> >> >> >>
> >> >>> >> >> >> On Fri, Sep 1, 2017 at 6:05 AM, Gaurav Chhabra
> >> >>> >> >> >> <varuag.chhabra at gmail.com>
> >> >>> >> >> >> wrote:
> >> >>> >> >> >> > Hi Jose,
> >> >>> >> >> >> >
> >> >>> >> >> >> >
> >> >>> >> >> >> > Thanks for the reply. It seems the three gluster pods
> might
> >> >>> >> >> >> > have
> >> >>> >> >> >> > been
> >> >>> >> >> >> > a
> >> >>> >> >> >> > copy-paste from another set of cluster where i was trying
> >> >>> >> >> >> > to
> >> >>> >> >> >> > setup
> >> >>> >> >> >> > the
> >> >>> >> >> >> > same
> >> >>> >> >> >> > thing using CentOS. Sorry for that. By the way, i did
> check
> >> >>> >> >> >> > for
> >> >>> >> >> >> > the
> >> >>> >> >> >> > kernel
> >> >>> >> >> >> > modules and it seems it's already there. Also, i am
> >> >>> >> >> >> > attaching
> >> >>> >> >> >> > fresh
> >> >>> >> >> >> > set
> >> >>> >> >> >> > of
> >> >>> >> >> >> > files because i created a new cluster and thought of
> giving
> >> >>> >> >> >> > it
> >> >>> >> >> >> > a
> >> >>> >> >> >> > try
> >> >>> >> >> >> > again.
> >> >>> >> >> >> > Issue still persists. :(
> >> >>> >> >> >> >
> >> >>> >> >> >> > In heketi.json, there is a slight change w.r.t the user
> >> >>> >> >> >> > which
> >> >>> >> >> >> > connects
> >> >>> >> >> >> > to
> >> >>> >> >> >> > glusterfs node using SSH. I am not sure how Heketi was
> >> >>> >> >> >> > using
> >> >>> >> >> >> > root
> >> >>> >> >> >> > user
> >> >>> >> >> >> > to
> >> >>> >> >> >> > login because i wasn't able to use root and do manual
> SSH.
> >> >>> >> >> >> > With
> >> >>> >> >> >> > rancher
> >> >>> >> >> >> > user, i can login successfully so i think this should be
> >> >>> >> >> >> > fine.
> >> >>> >> >> >> >
> >> >>> >> >> >> > /etc/heketi/heketi.json:
> >> >>> >> >> >> >
> >> >>> >> >> >> >
> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >>> >> >> >> >     "executor": "ssh",
> >> >>> >> >> >> >
> >> >>> >> >> >> >     "_sshexec_comment": "SSH username and private key
> file
> >> >>> >> >> >> > information",
> >> >>> >> >> >> >     "sshexec": {
> >> >>> >> >> >> >       "keyfile": "/var/lib/heketi/.ssh/id_rsa",
> >> >>> >> >> >> >       "user": "rancher",
> >> >>> >> >> >> >       "port": "22",
> >> >>> >> >> >> >       "fstab": "/etc/fstab"
> >> >>> >> >> >> >     },
> >> >>> >> >> >> >
> >> >>> >> >> >> >
> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >>> >> >> >> >
> >> >>> >> >> >> > Before running gk-deploy:
> >> >>> >> >> >> >
> >> >>> >> >> >> >
> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >>> >> >> >> > [root at workstation deploy]# kubectl get
> >> >>> >> >> >> > nodes,pods,daemonset,deployments,services
> >> >>> >> >> >> > NAME                                     STATUS    AGE
> >> >>> >> >> >> > VERSION
> >> >>> >> >> >> > no/node-a.c.kubernetes-174104.internal   Ready     3h
> >> >>> >> >> >> > v1.7.2-rancher1
> >> >>> >> >> >> > no/node-b.c.kubernetes-174104.internal   Ready     3h
> >> >>> >> >> >> > v1.7.2-rancher1
> >> >>> >> >> >> > no/node-c.c.kubernetes-174104.internal   Ready     3h
> >> >>> >> >> >> > v1.7.2-rancher1
> >> >>> >> >> >> >
> >> >>> >> >> >> > NAME             CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
> >> >>> >> >> >> > svc/kubernetes   10.43.0.1    <none>        443/TCP   3h
> >> >>> >> >> >> >
> >> >>> >> >> >> >
> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >>> >> >> >> >
> >> >>> >> >> >> > After running gk-deploy:
> >> >>> >> >> >> >
> >> >>> >> >> >> >
> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >>> >> >> >> > [root at workstation messagegc]# kubectl get
> >> >>> >> >> >> > nodes,pods,daemonset,deployments,services
> >> >>> >> >> >> > NAME                                     STATUS    AGE
> >> >>> >> >> >> > VERSION
> >> >>> >> >> >> > no/node-a.c.kubernetes-174104.internal   Ready     3h
> >> >>> >> >> >> > v1.7.2-rancher1
> >> >>> >> >> >> > no/node-b.c.kubernetes-174104.internal   Ready     3h
> >> >>> >> >> >> > v1.7.2-rancher1
> >> >>> >> >> >> > no/node-c.c.kubernetes-174104.internal   Ready     3h
> >> >>> >> >> >> > v1.7.2-rancher1
> >> >>> >> >> >> >
> >> >>> >> >> >> > NAME                 READY     STATUS    RESTARTS   AGE
> >> >>> >> >> >> > po/glusterfs-0j9l5   0/1       Running   0          2m
> >> >>> >> >> >> > po/glusterfs-gqz4c   0/1       Running   0          2m
> >> >>> >> >> >> > po/glusterfs-gxvcb   0/1       Running   0          2m
> >> >>> >> >> >> >
> >> >>> >> >> >> > NAME           DESIRED   CURRENT   READY     UP-TO-DATE
> >> >>> >> >> >> > AVAILABLE
> >> >>> >> >> >> > NODE-SELECTOR           AGE
> >> >>> >> >> >> > ds/glusterfs   3         3         0         3
>   0
> >> >>> >> >> >> > storagenode=glusterfs   2m
> >> >>> >> >> >> >
> >> >>> >> >> >> > NAME             CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
> >> >>> >> >> >> > svc/kubernetes   10.43.0.1    <none>        443/TCP   3h
> >> >>> >> >> >> >
> >> >>> >> >> >> >
> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >>> >> >> >> >
> >> >>> >> >> >> > Kernel module check on all three nodes:
> >> >>> >> >> >> >
> >> >>> >> >> >> >
> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >>> >> >> >> > [root at node-a ~]# find /lib*/modules/$(uname -r) -name
> *.ko
> >> >>> >> >> >> > |
> >> >>> >> >> >> > grep
> >> >>> >> >> >> > 'thin-pool\|snapshot\|mirror' | xargs ls -ltr
> >> >>> >> >> >> > -rw-r--r--    1 root     root         92310 Jun 26 04:13
> >> >>> >> >> >> >
> >> >>> >> >> >> >
> >> >>> >> >> >> > /lib64/modules/4.9.34-rancher/kernel/drivers/md/dm-thin-
> pool.ko
> >> >>> >> >> >> > -rw-r--r--    1 root     root         56982 Jun 26 04:13
> >> >>> >> >> >> >
> >> >>> >> >> >> > /lib64/modules/4.9.34-rancher/
> kernel/drivers/md/dm-snapshot.ko
> >> >>> >> >> >> > -rw-r--r--    1 root     root         27070 Jun 26 04:13
> >> >>> >> >> >> >
> >> >>> >> >> >> > /lib64/modules/4.9.34-rancher/
> kernel/drivers/md/dm-mirror.ko
> >> >>> >> >> >> > -rw-r--r--    1 root     root         92310 Jun 26 04:13
> >> >>> >> >> >> >
> >> >>> >> >> >> > /lib/modules/4.9.34-rancher/kernel/drivers/md/dm-thin-
> pool.ko
> >> >>> >> >> >> > -rw-r--r--    1 root     root         56982 Jun 26 04:13
> >> >>> >> >> >> >
> >> >>> >> >> >> > /lib/modules/4.9.34-rancher/
> kernel/drivers/md/dm-snapshot.ko
> >> >>> >> >> >> > -rw-r--r--    1 root     root         27070 Jun 26 04:13
> >> >>> >> >> >> > /lib/modules/4.9.34-rancher/
> kernel/drivers/md/dm-mirror.ko
> >> >>> >> >> >> >
> >> >>> >> >> >> >
> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >>> >> >> >> >
> >> >>> >> >> >> > Error snapshot attached.
> >> >>> >> >> >> >
> >> >>> >> >> >> > In my first mail, i checked that Readiness Probe failure
> >> >>> >> >> >> > check
> >> >>> >> >> >> > has
> >> >>> >> >> >> > this
> >> >>> >> >> >> > code
> >> >>> >> >> >> > in kube-templates/glusterfs-daemonset.yaml file:
> >> >>> >> >> >> >
> >> >>> >> >> >> >
> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >>> >> >> >> >         readinessProbe:
> >> >>> >> >> >> >           timeoutSeconds: 3
> >> >>> >> >> >> >           initialDelaySeconds: 40
> >> >>> >> >> >> >           exec:
> >> >>> >> >> >> >             command:
> >> >>> >> >> >> >             - "/bin/bash"
> >> >>> >> >> >> >             - "-c"
> >> >>> >> >> >> >             - systemctl status glusterd.service
> >> >>> >> >> >> >           periodSeconds: 25
> >> >>> >> >> >> >           successThreshold: 1
> >> >>> >> >> >> >           failureThreshold: 15
> >> >>> >> >> >> >
> >> >>> >> >> >> >
> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >>> >> >> >> >
> >> >>> >> >> >> > I tried logging into glustefs container on one of the
> node
> >> >>> >> >> >> > and
> >> >>> >> >> >> > ran
> >> >>> >> >> >> > the
> >> >>> >> >> >> > above
> >> >>> >> >> >> > command:
> >> >>> >> >> >> >
> >> >>> >> >> >> > [root at node-a ~]# docker exec -it c0f8ab4d92a23b6df2
> >> >>> >> >> >> > /bin/bash
> >> >>> >> >> >> > root at c0f8ab4d92a2:/app# systemctl status
> glusterd.service
> >> >>> >> >> >> > WARNING: terminal is not fully functional
> >> >>> >> >> >> > Failed to connect to bus: No such file or directory
> >> >>> >> >> >> >
> >> >>> >> >> >> >
> >> >>> >> >> >> > Any check that i can do manually on nodes to debug
> further?
> >> >>> >> >> >> > Any
> >> >>> >> >> >> > suggestions?
> >> >>> >> >> >> >
> >> >>> >> >> >> >
> >> >>> >> >> >> > On Thu, Aug 31, 2017 at 6:53 PM, Jose A. Rivera
> >> >>> >> >> >> > <jarrpa at redhat.com>
> >> >>> >> >> >> > wrote:
> >> >>> >> >> >> >>
> >> >>> >> >> >> >> Hey Gaurav,
> >> >>> >> >> >> >>
> >> >>> >> >> >> >> The kernel modules must be loaded on all nodes that will
> >> >>> >> >> >> >> run
> >> >>> >> >> >> >> heketi
> >> >>> >> >> >> >> pods. Additionally, you must have at least three nodes
> >> >>> >> >> >> >> specified
> >> >>> >> >> >> >> in
> >> >>> >> >> >> >> your topology file. I'm not sure how you're getting
> three
> >> >>> >> >> >> >> gluster
> >> >>> >> >> >> >> pods
> >> >>> >> >> >> >> when you only have two nodes defined... :)
> >> >>> >> >> >> >>
> >> >>> >> >> >> >> --Jose
> >> >>> >> >> >> >>
> >> >>> >> >> >> >> On Wed, Aug 30, 2017 at 5:27 AM, Gaurav Chhabra
> >> >>> >> >> >> >> <varuag.chhabra at gmail.com> wrote:
> >> >>> >> >> >> >> > Hi,
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > I have the following setup in place:
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > 1 node    : RancherOS having Rancher application for
> >> >>> >> >> >> >> > Kubernetes
> >> >>> >> >> >> >> > setup
> >> >>> >> >> >> >> > 2 nodes  : RancherOS having Rancher agent
> >> >>> >> >> >> >> > 1 node   : CentOS 7 workstation having kubectl
> installed
> >> >>> >> >> >> >> > and
> >> >>> >> >> >> >> > folder
> >> >>> >> >> >> >> > cloned/downloaded from
> >> >>> >> >> >> >> > https://github.com/gluster/gluster-kubernetes
> >> >>> >> >> >> >> > using
> >> >>> >> >> >> >> > which i run Heketi setup (gk-deploy -g)
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > I also have rancher-glusterfs-server container running
> >> >>> >> >> >> >> > with
> >> >>> >> >> >> >> > the
> >> >>> >> >> >> >> > following
> >> >>> >> >> >> >> > configuration:
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > ------------------------------
> ------------------------------------
> >> >>> >> >> >> >> > [root at node-1 rancher]# cat gluster-server.sh
> >> >>> >> >> >> >> > #!/bin/bash
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > sudo docker run --name=gluster-server -d \
> >> >>> >> >> >> >> >         --env 'SERVICE_NAME=gluster' \
> >> >>> >> >> >> >> >         --restart always \
> >> >>> >> >> >> >> >         --env 'GLUSTER_DATA=/srv/docker/gitlab' \
> >> >>> >> >> >> >> >         --publish 2222:22 \
> >> >>> >> >> >> >> >         webcenter/rancher-glusterfs-server
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > ------------------------------
> ------------------------------------
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > In /etc/heketi/heketi.json, following is the only
> >> >>> >> >> >> >> > modified
> >> >>> >> >> >> >> > portion:
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > ------------------------------
> ------------------------------------
> >> >>> >> >> >> >> >     "executor": "ssh",
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >     "_sshexec_comment": "SSH username and private key
> >> >>> >> >> >> >> > file
> >> >>> >> >> >> >> > information",
> >> >>> >> >> >> >> >     "sshexec": {
> >> >>> >> >> >> >> >       "keyfile": "/var/lib/heketi/.ssh/id_rsa",
> >> >>> >> >> >> >> >       "user": "root",
> >> >>> >> >> >> >> >       "port": "22",
> >> >>> >> >> >> >> >       "fstab": "/etc/fstab"
> >> >>> >> >> >> >> >     },
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > ------------------------------
> ------------------------------------
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > Status before running gk-deploy:
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > [root at workstation deploy]# kubectl get
> >> >>> >> >> >> >> > nodes,pods,services,deployments
> >> >>> >> >> >> >> > NAME                                     STATUS    AGE
> >> >>> >> >> >> >> > VERSION
> >> >>> >> >> >> >> > no/node-1.c.kubernetes-174104.internal   Ready     2d
> >> >>> >> >> >> >> > v1.7.2-rancher1
> >> >>> >> >> >> >> > no/node-2.c.kubernetes-174104.internal   Ready     2d
> >> >>> >> >> >> >> > v1.7.2-rancher1
> >> >>> >> >> >> >> > no/node-3.c.kubernetes-174104.internal   Ready     2d
> >> >>> >> >> >> >> > v1.7.2-rancher1
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > NAME             CLUSTER-IP   EXTERNAL-IP   PORT(S)
> >> >>> >> >> >> >> > AGE
> >> >>> >> >> >> >> > svc/kubernetes   10.43.0.1    <none>        443/TCP
>  2d
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > Now when i run 'gk-deploy -g', in the Rancher
> console, i
> >> >>> >> >> >> >> > see
> >> >>> >> >> >> >> > the
> >> >>> >> >> >> >> > following
> >> >>> >> >> >> >> > error:
> >> >>> >> >> >> >> > Readiness probe failed: Failed to get D-Bus
> connection:
> >> >>> >> >> >> >> > Operation
> >> >>> >> >> >> >> > not
> >> >>> >> >> >> >> > permitted
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > From the attached gk-deploy_log i see that it failed
> at:
> >> >>> >> >> >> >> > Waiting for GlusterFS pods to start ... pods not
> found.
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > In the kube-templates/glusterfs-daemonset.yaml file,
> i
> >> >>> >> >> >> >> > see
> >> >>> >> >> >> >> > this
> >> >>> >> >> >> >> > for
> >> >>> >> >> >> >> > Readiness probe section:
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > ------------------------------
> ------------------------------------
> >> >>> >> >> >> >> >         readinessProbe:
> >> >>> >> >> >> >> >           timeoutSeconds: 3
> >> >>> >> >> >> >> >           initialDelaySeconds: 40
> >> >>> >> >> >> >> >           exec:
> >> >>> >> >> >> >> >             command:
> >> >>> >> >> >> >> >             - "/bin/bash"
> >> >>> >> >> >> >> >             - "-c"
> >> >>> >> >> >> >> >             - systemctl status glusterd.service
> >> >>> >> >> >> >> >           periodSeconds: 25
> >> >>> >> >> >> >> >           successThreshold: 1
> >> >>> >> >> >> >> >           failureThreshold: 15
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > ------------------------------
> ------------------------------------
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > Status after running gk-deploy:
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > [root at workstation deploy]# kubectl get
> >> >>> >> >> >> >> > nodes,pods,deployments,services
> >> >>> >> >> >> >> > NAME                                     STATUS    AGE
> >> >>> >> >> >> >> > VERSION
> >> >>> >> >> >> >> > no/node-1.c.kubernetes-174104.internal   Ready     2d
> >> >>> >> >> >> >> > v1.7.2-rancher1
> >> >>> >> >> >> >> > no/node-2.c.kubernetes-174104.internal   Ready     2d
> >> >>> >> >> >> >> > v1.7.2-rancher1
> >> >>> >> >> >> >> > no/node-3.c.kubernetes-174104.internal   Ready     2d
> >> >>> >> >> >> >> > v1.7.2-rancher1
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > NAME                 READY     STATUS    RESTARTS
>  AGE
> >> >>> >> >> >> >> > po/glusterfs-0s440   0/1       Running   0          1m
> >> >>> >> >> >> >> > po/glusterfs-j7dgr   0/1       Running   0          1m
> >> >>> >> >> >> >> > po/glusterfs-p6jl3   0/1       Running   0          1m
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > NAME             CLUSTER-IP   EXTERNAL-IP   PORT(S)
> >> >>> >> >> >> >> > AGE
> >> >>> >> >> >> >> > svc/kubernetes   10.43.0.1    <none>        443/TCP
>  2d
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > Also, from prerequisite perspective, i was also seeing
> >> >>> >> >> >> >> > this
> >> >>> >> >> >> >> > mentioned:
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > The following kernel modules must be loaded:
> >> >>> >> >> >> >> >  * dm_snapshot
> >> >>> >> >> >> >> >  * dm_mirror
> >> >>> >> >> >> >> >  * dm_thin_pool
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > Where exactly is this to be checked? On all Gluster
> >> >>> >> >> >> >> > server
> >> >>> >> >> >> >> > nodes?
> >> >>> >> >> >> >> > How
> >> >>> >> >> >> >> > can i
> >> >>> >> >> >> >> > check whether it's there?
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > I have attached topology.json and gk-deploy log for
> >> >>> >> >> >> >> > reference.
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > Does this issue has anything to do with the host OS
> >> >>> >> >> >> >> > (RancherOS)
> >> >>> >> >> >> >> > that
> >> >>> >> >> >> >> > i
> >> >>> >> >> >> >> > am
> >> >>> >> >> >> >> > using for Gluster nodes? Any idea how i can fix this?
> >> >>> >> >> >> >> > Any
> >> >>> >> >> >> >> > help
> >> >>> >> >> >> >> > will
> >> >>> >> >> >> >> > really
> >> >>> >> >> >> >> > be appreciated.
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > Thanks.
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >> > _______________________________________________
> >> >>> >> >> >> >> > heketi-devel mailing list
> >> >>> >> >> >> >> > heketi-devel at gluster.org
> >> >>> >> >> >> >> > http://lists.gluster.org/
> mailman/listinfo/heketi-devel
> >> >>> >> >> >> >> >
> >> >>> >> >> >> >
> >> >>> >> >> >> >
> >> >>> >> >> >
> >> >>> >> >> >
> >> >>> >> >
> >> >>> >> >
> >> >>> >
> >> >>> >
> >> >>
> >> >>
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/heketi-devel/attachments/20170904/6a7d0adb/attachment-0001.html>


More information about the heketi-devel mailing list