[heketi-devel] Unable to use Heketi setup to install Gluster for Kubernetes
Jose A. Rivera
jarrpa at redhat.com
Mon Sep 4 22:11:07 UTC 2017
Yup, I follow that repo and should be contributing to it in the
near-future. :) I've never played with the glusterfs-client image, but
here's hoping!
On Mon, Sep 4, 2017 at 10:18 AM, Gaurav Chhabra
<varuag.chhabra at gmail.com> wrote:
> Actually the script is launching a container and i'm pretty sure that the
> command that's programmed as part of the container launch process is
> actually trying to run that mount command. Now the issue is that this again
> falls under Docker area. :( Generally, Docker container providers (in this
> case, nixel) provide a Dockerfile which specifies the command that's
> executed as part of the container launch process however, the problem in
> case of this container is that it's not maintained. It's 2 years old. If you
> will check active projects such as gluster/gluster-containers, you will see
> a proper Dockerfile explaining what it does.
>
> https://hub.docker.com/r/gluster/glusterfs-client/~/dockerfile/
>
> I am now planning to use the above container and see whether i'm lucky. If
> you will see the above link, it's running
>
> dnf install -y glusterfs-fuse
>
> Hopefully, this should take care of this section which asks to make sure
> that glusterfs-client is installed
>
> https://github.com/gluster/gluster-kubernetes/tree/master/docs/examples/dynamic_provisioning_external_gluster#validate-communication-and-gluster-prerequisites-on-the-kubernetes-nodes
>
>
>
>
> On Mon, Sep 4, 2017 at 7:33 PM, Jose A. Rivera <jarrpa at redhat.com> wrote:
>>
>> Something is trying to mount a GlusterFS volume, and it's not that
>> script. What's trying to mount?
>>
>> On Mon, Sep 4, 2017 at 8:52 AM, Gaurav Chhabra <varuag.chhabra at gmail.com>
>> wrote:
>> > I am just running this bash script which will start the
>> > rancher-glusterfs-client container:
>> >
>> > [root at node-a ~]# cat rancher-glusterfs-client.sh
>> > #!/bin/bash
>> > sudo docker run --privileged \
>> > --name=gluster-client \
>> > -d \
>> > -v /sys/fs/cgroup:/sys/fs/cgroup \
>> > -v /var/log/glusterfs:/var/log/glusterfs \
>> > --env GLUSTER_PEER=10.128.0.12,10.128.0.15,10.128.0.16 \
>> > nixel/rancher-glusterfs-client
>> >
>> >
>> >
>> > On Mon, Sep 4, 2017 at 6:26 PM, Jose A. Rivera <jarrpa at redhat.com>
>> > wrote:
>> >>
>> >> What is the exact command you're running?
>> >>
>> >> On Mon, Sep 4, 2017 at 4:26 AM, Gaurav Chhabra
>> >> <varuag.chhabra at gmail.com>
>> >> wrote:
>> >> > Hi Jose,
>> >> >
>> >> >
>> >> > From this link, it seems mount.glusterfs might actually be present in
>> >> > the
>> >> > container that launched and quickly terminated.
>> >> >
>> >> >
>> >> >
>> >> > https://unix.stackexchange.com/questions/312178/glusterfs-replicated-volume-mounting-issue
>> >> >
>> >> > If you check the question that the user posted, the same error (Mount
>> >> > failed) was reported that i sent you in the last email.
>> >> >
>> >> > After seeing the above, i checked /var/log/glusterfs on my host
>> >> > (RancherOS)
>> >> > but it was empty. I ran the container again but with explicit volume
>> >> > mount
>> >> > as shown below:
>> >> >
>> >> > [root at node-a ~]# cat rancher-glusterfs-client.sh
>> >> > #!/bin/bash
>> >> > sudo docker run --privileged \
>> >> > --name=gluster-client \
>> >> > -d \
>> >> > -v /sys/fs/cgroup:/sys/fs/cgroup \
>> >> > -v /var/log/glusterfs:/var/log/glusterfs \
>> >> > --env GLUSTER_PEER=10.128.0.12,10.128.0.15,10.128.0.16 \
>> >> > nixel/rancher-glusterfs-client
>> >> > This time, i could see a log file
>> >> > (/var/log/glusterfs/mnt-ranchervol.log)
>> >> > present. I have attached the content of the same. Also attached are
>> >> > logs
>> >> > from Heketi client/server (both on one node) and Gluster cluster.
>> >> >
>> >> >
>> >> > Regards,
>> >> > Gaurav
>> >> >
>> >> >
>> >> > On Mon, Sep 4, 2017 at 2:29 PM, Gaurav Chhabra
>> >> > <varuag.chhabra at gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi Jose,
>> >> >>
>> >> >>
>> >> >> I tried setting up things using the link you provided and i was able
>> >> >> to
>> >> >> get all steps working for 3 node Gluster cluster, all running on
>> >> >> CentOS7,
>> >> >> without any issue. However, as expected, when i tried configuring
>> >> >> Kubernetes
>> >> >> by installing nixel/rancher-glusterfs-client container, i got error:
>> >> >>
>> >> >> [root at node-a ~]# cat rancher-glusterfs-client.sh
>> >> >> #!/bin/bash
>> >> >> sudo docker run --privileged \
>> >> >> --name=gluster-client \
>> >> >> -d \
>> >> >> -v /sys/fs/cgroup:/sys/fs/cgroup \
>> >> >> --env GLUSTER_PEER=10.128.0.12,10.128.0.15,10.128.0.16 \
>> >> >> nixel/rancher-glusterfs-client
>> >> >>
>> >> >> [root at node-a ~]# ./rancher-glusterfs-client.sh
>> >> >> ac069caccdce147d6f423fc566166345191dbc1b11f3416c66207a1fd11fda6b
>> >> >>
>> >> >> [root at node-a ~]# docker logs gluster-client
>> >> >> => Checking if I can reach GlusterFS node 10.128.0.12 ...
>> >> >> => GlusterFS node 10.128.0.12 is alive
>> >> >> => Mounting GlusterFS volume ranchervol from GlusterFS node
>> >> >> 10.128.0.12
>> >> >> ...
>> >> >> Mount failed. Please check the log file for more details.
>> >> >>
>> >> >> If i try running the next step as described in your link, i get the
>> >> >> following:
>> >> >>
>> >> >> [root at node-a ~]# modprobe fuse
>> >> >> modprobe: module fuse not found in modules.dep
>> >> >>
>> >> >> Since the container failed to start, i could only check on the host
>> >> >> (RancherOS) and i could only find two mount-related commands: mount
>> >> >> &
>> >> >> mountpoint
>> >> >>
>> >> >> Any pointers?
>> >> >>
>> >> >>
>> >> >> Regards,
>> >> >> Gaurav
>> >> >>
>> >> >> On Sun, Sep 3, 2017 at 11:18 PM, Jose A. Rivera <jarrpa at redhat.com>
>> >> >> wrote:
>> >> >>>
>> >> >>> Installing the glusterfs-client container should be fine. :) The
>> >> >>> main
>> >> >>> thing that's needed is that all your Kubernetes nodes need to have
>> >> >>> the
>> >> >>> "mount.glusterfs" command available so Kube can mount the GlusterFS
>> >> >>> volumes and present them to the pods.
>> >> >>>
>> >> >>> On Sun, Sep 3, 2017 at 12:14 PM, Gaurav Chhabra
>> >> >>> <varuag.chhabra at gmail.com> wrote:
>> >> >>> > Thanks Jose. The link you've suggested looks good but again it
>> >> >>> > expects
>> >> >>> > me to
>> >> >>> > install gluster-client on Kubernetes node and i fall into the
>> >> >>> > same
>> >> >>> > issue of
>> >> >>> > installing a container for glusterfs. Only difference is that
>> >> >>> > this
>> >> >>> > time
>> >> >>> > it's
>> >> >>> > glusterfs-client and not glusterfs-server. :)
>> >> >>> >
>> >> >>> > I will try this out and let you know tomorrow.
>> >> >>> >
>> >> >>> >
>> >> >>> > Regards,
>> >> >>> > Gaurav
>> >> >>> >
>> >> >>> >
>> >> >>> > On Sun, Sep 3, 2017 at 2:11 AM, Jose A. Rivera
>> >> >>> > <jarrpa at redhat.com>
>> >> >>> > wrote:
>> >> >>> >>
>> >> >>> >> Hey, no problem! I'm eager to learn more about different flavors
>> >> >>> >> of
>> >> >>> >> Linux, I just apologize for my relative inexperience with them.
>> >> >>> >> :)
>> >> >>> >>
>> >> >>> >> To that end, I will also admit I'm not very experienced with
>> >> >>> >> direct
>> >> >>> >> Docker myself. I understand the basic workflow and know some of
>> >> >>> >> the
>> >> >>> >> run options, but not having deep experience keeps me from having
>> >> >>> >> a
>> >> >>> >> better understanding of the patterns and consequences.
>> >> >>> >>
>> >> >>> >> Thus, I'd like to guide you in a direction I'd more apt to help
>> >> >>> >> you
>> >> >>> >> in
>> >> >>> >> right now. I know that you can't have multiple GlusterFS servers
>> >> >>> >> running on the same nodes, and I know that we have been
>> >> >>> >> successfully
>> >> >>> >> running several configurations using our gluster/gluster-centos
>> >> >>> >> image.
>> >> >>> >> If you follow the Kubernetes configuration on
>> >> >>> >> gluster-kubernetes,
>> >> >>> >> the
>> >> >>> >> pod/container is run privileged and with host networking, and we
>> >> >>> >> require that the node has all listed ports open, not just 2222.
>> >> >>> >> The
>> >> >>> >> sshd running in the container is listening on 2222, not 22, but
>> >> >>> >> it
>> >> >>> >> is
>> >> >>> >> also not really required if you're not doing geo-replication.
>> >> >>> >>
>> >> >>> >> Alternatively, you can indeed run GlusterFS outside of
>> >> >>> >> Kubernetes
>> >> >>> >> but
>> >> >>> >> still have Kubernetes apps access GlusterFS storage. The nodes
>> >> >>> >> can
>> >> >>> >> be
>> >> >>> >> anything you want, they just need to be running GlusterFS and
>> >> >>> >> you
>> >> >>> >> need
>> >> >>> >> a heketi service managing them. Here is an example of how to set
>> >> >>> >> this
>> >> >>> >> up using CentOS:
>> >> >>> >>
>> >> >>> >>
>> >> >>> >>
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> https://github.com/gluster/gluster-kubernetes/tree/master/docs/examples/dynamic_provisioning_external_gluster
>> >> >>> >>
>> >> >>> >> Hope this is at least leading you in a useful direction. :)
>> >> >>> >>
>> >> >>> >> --Jose
>> >> >>> >>
>> >> >>> >> On Sat, Sep 2, 2017 at 3:16 PM, Gaurav Chhabra
>> >> >>> >> <varuag.chhabra at gmail.com>
>> >> >>> >> wrote:
>> >> >>> >> > Hi Jose,
>> >> >>> >> >
>> >> >>> >> > webcenter/rancher-glusterfs-server is actually a container
>> >> >>> >> > provided
>> >> >>> >> > by
>> >> >>> >> > Sebastien, its maintainer. It's a Docker container which has
>> >> >>> >> > GlusterFS
>> >> >>> >> > server running within it. On the host i.e., RancherOS, there
>> >> >>> >> > is
>> >> >>> >> > no
>> >> >>> >> > separate
>> >> >>> >> > GlusterFS server running because we cannot install anything
>> >> >>> >> > that
>> >> >>> >> > way.
>> >> >>> >> > Running using container is the only way so i started
>> >> >>> >> > ancher-glusterfs-server
>> >> >>> >> > container with the following parameters:
>> >> >>> >> >
>> >> >>> >> > [root at node-1 rancher]# cat gluster-server.sh
>> >> >>> >> > #!/bin/bash
>> >> >>> >> > sudo docker run --name=gluster-server -d \
>> >> >>> >> > --env 'SERVICE_NAME=gluster' \
>> >> >>> >> > --restart always \
>> >> >>> >> > --publish 2222:22 \
>> >> >>> >> > webcenter/rancher-glusterfs-server
>> >> >>> >> >
>> >> >>> >> > Here's the link to the dockerfile:
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >> > https://hub.docker.com/r/webcenter/rancher-glusterfs-server/~/dockerfile/
>> >> >>> >> >
>> >> >>> >> > It's similar to other GlusteFS containers provided by other
>> >> >>> >> > maintainers
>> >> >>> >> > for
>> >> >>> >> > different OS. For example, for CentOS, we have
>> >> >>> >> > https://hub.docker.com/r/gluster/gluster-centos/~/dockerfile/
>> >> >>> >> >
>> >> >>> >> > From what i understand, Heketi does support container based
>> >> >>> >> > GlusterFS
>> >> >>> >> > server
>> >> >>> >> > as mentioned in the prerequisite where it says:
>> >> >>> >> >
>> >> >>> >> > "Each node must have the following ports opened for GlusterFS
>> >> >>> >> > communications:
>> >> >>> >> > 2222 - GlusterFS pod's sshd"
>> >> >>> >> >
>> >> >>> >> > That's the reason i've exposed port 2222 for 22 as shown
>> >> >>> >> > above.
>> >> >>> >> > Please
>> >> >>> >> > correct me if i misunderstood it.
>> >> >>> >> >
>> >> >>> >> > As soon as i run the above script (gluster-server.sh), it
>> >> >>> >> > automatically
>> >> >>> >> > creates the following directories on host. This should have
>> >> >>> >> > ideally
>> >> >>> >> > not
>> >> >>> >> > been
>> >> >>> >> > empty as you mentioned.
>> >> >>> >> >
>> >> >>> >> > /etc/glusterfs /var/lib/glusterd /var/log/glusterfs
>> >> >>> >> >
>> >> >>> >> > Just wanted to know in which circumstances do we get this
>> >> >>> >> > specific
>> >> >>> >> > error
>> >> >>> >> > (Failed to get D-Bus connection: Operation not permitted)
>> >> >>> >> > related
>> >> >>> >> > to
>> >> >>> >> > Readiness probe failing. Searching online took me to
>> >> >>> >> > discussions
>> >> >>> >> > around
>> >> >>> >> > running container in privileged mode and some directory to be
>> >> >>> >> > mounted.
>> >> >>> >> > Based
>> >> >>> >> > on that, i also modified my container startup script to the
>> >> >>> >> > following:
>> >> >>> >> >
>> >> >>> >> > #!/bin/bash
>> >> >>> >> > sudo docker run --privileged \
>> >> >>> >> > --name=gluster-server \
>> >> >>> >> > -d \
>> >> >>> >> > -v /sys/fs/cgroup:/sys/fs/cgroup \
>> >> >>> >> > -v /etc/glusterfs:/etc/glusterfs \
>> >> >>> >> > -v /var/lib/glusterd:/var/lib/glusterd \
>> >> >>> >> > -v /var/log/glusterfs:/var/log/glusterfs \
>> >> >>> >> > --env 'SERVICE_NAME=gluster' \
>> >> >>> >> > --restart always \
>> >> >>> >> > --publish 2222:22 \
>> >> >>> >> > webcenter/rancher-glusterfs-server
>> >> >>> >> > Still, the issue persists.
>> >> >>> >> >
>> >> >>> >> > I also logged into the container and checked whether systemctl
>> >> >>> >> > command
>> >> >>> >> > is
>> >> >>> >> > present. It was there but manualy running the command also
>> >> >>> >> > doesn't
>> >> >>> >> > work:
>> >> >>> >> >
>> >> >>> >> > [root at node-c ~]# docker exec -it gluster-server /bin/bash
>> >> >>> >> > root at 42150f203f80:/app# systemctl status glusterd.service
>> >> >>> >> > WARNING: terminal is not fully functional
>> >> >>> >> > Failed to connect to bus: No such file or directory
>> >> >>> >> >
>> >> >>> >> > Under section 'ADVANCED OPTIONS - Security/Host' in this link,
>> >> >>> >> > it
>> >> >>> >> > talks
>> >> >>> >> > about SYS_ADMIN setting. Any idea how i can try this?
>> >> >>> >> >
>> >> >>> >> > Also, there was this mentioned in the Heketi setup page:
>> >> >>> >> > "If you are not able to deploy a hyper-converged GlusterFS
>> >> >>> >> > cluster,
>> >> >>> >> > you
>> >> >>> >> > must
>> >> >>> >> > have one running somewhere that the Kubernetes nodes can
>> >> >>> >> > access"
>> >> >>> >> >
>> >> >>> >> >>>> Does it mean running the three node Gluster cluster outside
>> >> >>> >> >>>> Kubernetes,
>> >> >>> >> >>>> may be on some VM running on RHEL/CentOS etc? If yes, then
>> >> >>> >> >>>> how
>> >> >>> >> >>>> will i
>> >> >>> >> >>>> be
>> >> >>> >> >>>> able to tell Gluster which volume from the Kubernetes
>> >> >>> >> >>>> cluster
>> >> >>> >> >>>> pod
>> >> >>> >> >>>> to
>> >> >>> >> >>>> sync?
>> >> >>> >> >>>> Any references?
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >> > I really appreciate your responses despite the fact that
>> >> >>> >> > you've
>> >> >>> >> > not
>> >> >>> >> > used
>> >> >>> >> > RancherOS but still trying to help.
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >> > Thanks,
>> >> >>> >> > Gaurav
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >> > On Sat, Sep 2, 2017 at 7:35 PM, Jose A. Rivera
>> >> >>> >> > <jarrpa at redhat.com>
>> >> >>> >> > wrote:
>> >> >>> >> >>
>> >> >>> >> >> I'm afraid I have no experience with RancherOS, so I may be
>> >> >>> >> >> missing
>> >> >>> >> >> some things about how it works. My primary experience is with
>> >> >>> >> >> Fedora,
>> >> >>> >> >> CentOS, and Ubuntu.
>> >> >>> >> >>
>> >> >>> >> >> What is webcenter/rancher-glusterfs-server? If it's running
>> >> >>> >> >> another
>> >> >>> >> >> glusterd then you probably don't want to be running it and
>> >> >>> >> >> should
>> >> >>> >> >> remove it from your systems.
>> >> >>> >> >>
>> >> >>> >> >> The glusterfs pods mount hostpath volumes from the host
>> >> >>> >> >> they're
>> >> >>> >> >> running on to persist their configuration. Thus anything they
>> >> >>> >> >> write
>> >> >>> >> >> to
>> >> >>> >> >> those directories should land on the host. If that's not
>> >> >>> >> >> happening
>> >> >>> >> >> then that's an additional problem.
>> >> >>> >> >>
>> >> >>> >> >> --Jose
>> >> >>> >> >>
>> >> >>> >> >> On Fri, Sep 1, 2017 at 11:17 PM, Gaurav Chhabra
>> >> >>> >> >> <varuag.chhabra at gmail.com> wrote:
>> >> >>> >> >> > Hi Jose,
>> >> >>> >> >> >
>> >> >>> >> >> >
>> >> >>> >> >> > I tried your suggestion but there is one confusion
>> >> >>> >> >> > regarding
>> >> >>> >> >> > point
>> >> >>> >> >> > #3.
>> >> >>> >> >> > Since
>> >> >>> >> >> > RancherOS has everything running as container, i am running
>> >> >>> >> >> > webcenter/rancher-glusterfs-server container on all three
>> >> >>> >> >> > nodes.
>> >> >>> >> >> > Now
>> >> >>> >> >> > as
>> >> >>> >> >> > far
>> >> >>> >> >> > as removing the directories are concerned, i hope you meant
>> >> >>> >> >> > removing
>> >> >>> >> >> > them on
>> >> >>> >> >> > the host and _not_ from within the container. After
>> >> >>> >> >> > completing
>> >> >>> >> >> > step 1
>> >> >>> >> >> > and 2,
>> >> >>> >> >> > i checked the contents of all the directories that you
>> >> >>> >> >> > specified
>> >> >>> >> >> > in
>> >> >>> >> >> > point
>> >> >>> >> >> > #3. All were empty as you can see in the attached
>> >> >>> >> >> > other_logs.txt.
>> >> >>> >> >> > So
>> >> >>> >> >> > i
>> >> >>> >> >> > got
>> >> >>> >> >> > confused. I ran the deploy again but the issue persists.
>> >> >>> >> >> > Two
>> >> >>> >> >> > pods
>> >> >>> >> >> > show
>> >> >>> >> >> > Liveness error and the third one, Readiness error.
>> >> >>> >> >> >
>> >> >>> >> >> > I then tried removing those directories (Step #3) from
>> >> >>> >> >> > within
>> >> >>> >> >> > the
>> >> >>> >> >> > container
>> >> >>> >> >> > but getting following error:
>> >> >>> >> >> >
>> >> >>> >> >> > root at c0f8ab4d92a2:/app# rm -rf /var/lib/heketi
>> >> >>> >> >> > /etc/glusterfs
>> >> >>> >> >> > /var/lib/glusterd /var/log/glusterfs
>> >> >>> >> >> > rm: cannot remove '/var/lib/glusterd': Device or resource
>> >> >>> >> >> > busy
>> >> >>> >> >> >
>> >> >>> >> >> >
>> >> >>> >> >> >
>> >> >>> >> >> > On Fri, Sep 1, 2017 at 8:21 PM, Jose A. Rivera
>> >> >>> >> >> > <jarrpa at redhat.com>
>> >> >>> >> >> > wrote:
>> >> >>> >> >> >>
>> >> >>> >> >> >> 1. Add a line to the ssh-exec portion of heketi.json of
>> >> >>> >> >> >> the
>> >> >>> >> >> >> sort:
>> >> >>> >> >> >>
>> >> >>> >> >> >> "sudo": true,
>> >> >>> >> >> >>
>> >> >>> >> >> >> 2. Run
>> >> >>> >> >> >>
>> >> >>> >> >> >> gk-deploy -g --abort
>> >> >>> >> >> >>
>> >> >>> >> >> >> 3. On the nodes that were/will be running GlusterFS pods,
>> >> >>> >> >> >> run:
>> >> >>> >> >> >>
>> >> >>> >> >> >> rm -rf /var/lib/heketi /etc/glusterfs /var/lib/glusterd
>> >> >>> >> >> >> /var/log/glusterfs
>> >> >>> >> >> >>
>> >> >>> >> >> >> Then try the deploy again.
>> >> >>> >> >> >>
>> >> >>> >> >> >> On Fri, Sep 1, 2017 at 6:05 AM, Gaurav Chhabra
>> >> >>> >> >> >> <varuag.chhabra at gmail.com>
>> >> >>> >> >> >> wrote:
>> >> >>> >> >> >> > Hi Jose,
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > Thanks for the reply. It seems the three gluster pods
>> >> >>> >> >> >> > might
>> >> >>> >> >> >> > have
>> >> >>> >> >> >> > been
>> >> >>> >> >> >> > a
>> >> >>> >> >> >> > copy-paste from another set of cluster where i was
>> >> >>> >> >> >> > trying
>> >> >>> >> >> >> > to
>> >> >>> >> >> >> > setup
>> >> >>> >> >> >> > the
>> >> >>> >> >> >> > same
>> >> >>> >> >> >> > thing using CentOS. Sorry for that. By the way, i did
>> >> >>> >> >> >> > check
>> >> >>> >> >> >> > for
>> >> >>> >> >> >> > the
>> >> >>> >> >> >> > kernel
>> >> >>> >> >> >> > modules and it seems it's already there. Also, i am
>> >> >>> >> >> >> > attaching
>> >> >>> >> >> >> > fresh
>> >> >>> >> >> >> > set
>> >> >>> >> >> >> > of
>> >> >>> >> >> >> > files because i created a new cluster and thought of
>> >> >>> >> >> >> > giving
>> >> >>> >> >> >> > it
>> >> >>> >> >> >> > a
>> >> >>> >> >> >> > try
>> >> >>> >> >> >> > again.
>> >> >>> >> >> >> > Issue still persists. :(
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > In heketi.json, there is a slight change w.r.t the user
>> >> >>> >> >> >> > which
>> >> >>> >> >> >> > connects
>> >> >>> >> >> >> > to
>> >> >>> >> >> >> > glusterfs node using SSH. I am not sure how Heketi was
>> >> >>> >> >> >> > using
>> >> >>> >> >> >> > root
>> >> >>> >> >> >> > user
>> >> >>> >> >> >> > to
>> >> >>> >> >> >> > login because i wasn't able to use root and do manual
>> >> >>> >> >> >> > SSH.
>> >> >>> >> >> >> > With
>> >> >>> >> >> >> > rancher
>> >> >>> >> >> >> > user, i can login successfully so i think this should be
>> >> >>> >> >> >> > fine.
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > /etc/heketi/heketi.json:
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >>> >> >> >> > "executor": "ssh",
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > "_sshexec_comment": "SSH username and private key
>> >> >>> >> >> >> > file
>> >> >>> >> >> >> > information",
>> >> >>> >> >> >> > "sshexec": {
>> >> >>> >> >> >> > "keyfile": "/var/lib/heketi/.ssh/id_rsa",
>> >> >>> >> >> >> > "user": "rancher",
>> >> >>> >> >> >> > "port": "22",
>> >> >>> >> >> >> > "fstab": "/etc/fstab"
>> >> >>> >> >> >> > },
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > Before running gk-deploy:
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >>> >> >> >> > [root at workstation deploy]# kubectl get
>> >> >>> >> >> >> > nodes,pods,daemonset,deployments,services
>> >> >>> >> >> >> > NAME STATUS AGE
>> >> >>> >> >> >> > VERSION
>> >> >>> >> >> >> > no/node-a.c.kubernetes-174104.internal Ready 3h
>> >> >>> >> >> >> > v1.7.2-rancher1
>> >> >>> >> >> >> > no/node-b.c.kubernetes-174104.internal Ready 3h
>> >> >>> >> >> >> > v1.7.2-rancher1
>> >> >>> >> >> >> > no/node-c.c.kubernetes-174104.internal Ready 3h
>> >> >>> >> >> >> > v1.7.2-rancher1
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > NAME CLUSTER-IP EXTERNAL-IP PORT(S)
>> >> >>> >> >> >> > AGE
>> >> >>> >> >> >> > svc/kubernetes 10.43.0.1 <none> 443/TCP 3h
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > After running gk-deploy:
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >>> >> >> >> > [root at workstation messagegc]# kubectl get
>> >> >>> >> >> >> > nodes,pods,daemonset,deployments,services
>> >> >>> >> >> >> > NAME STATUS AGE
>> >> >>> >> >> >> > VERSION
>> >> >>> >> >> >> > no/node-a.c.kubernetes-174104.internal Ready 3h
>> >> >>> >> >> >> > v1.7.2-rancher1
>> >> >>> >> >> >> > no/node-b.c.kubernetes-174104.internal Ready 3h
>> >> >>> >> >> >> > v1.7.2-rancher1
>> >> >>> >> >> >> > no/node-c.c.kubernetes-174104.internal Ready 3h
>> >> >>> >> >> >> > v1.7.2-rancher1
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > NAME READY STATUS RESTARTS AGE
>> >> >>> >> >> >> > po/glusterfs-0j9l5 0/1 Running 0 2m
>> >> >>> >> >> >> > po/glusterfs-gqz4c 0/1 Running 0 2m
>> >> >>> >> >> >> > po/glusterfs-gxvcb 0/1 Running 0 2m
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > NAME DESIRED CURRENT READY UP-TO-DATE
>> >> >>> >> >> >> > AVAILABLE
>> >> >>> >> >> >> > NODE-SELECTOR AGE
>> >> >>> >> >> >> > ds/glusterfs 3 3 0 3
>> >> >>> >> >> >> > 0
>> >> >>> >> >> >> > storagenode=glusterfs 2m
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > NAME CLUSTER-IP EXTERNAL-IP PORT(S)
>> >> >>> >> >> >> > AGE
>> >> >>> >> >> >> > svc/kubernetes 10.43.0.1 <none> 443/TCP 3h
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > Kernel module check on all three nodes:
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >>> >> >> >> > [root at node-a ~]# find /lib*/modules/$(uname -r) -name
>> >> >>> >> >> >> > *.ko
>> >> >>> >> >> >> > |
>> >> >>> >> >> >> > grep
>> >> >>> >> >> >> > 'thin-pool\|snapshot\|mirror' | xargs ls -ltr
>> >> >>> >> >> >> > -rw-r--r-- 1 root root 92310 Jun 26 04:13
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > /lib64/modules/4.9.34-rancher/kernel/drivers/md/dm-thin-pool.ko
>> >> >>> >> >> >> > -rw-r--r-- 1 root root 56982 Jun 26 04:13
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > /lib64/modules/4.9.34-rancher/kernel/drivers/md/dm-snapshot.ko
>> >> >>> >> >> >> > -rw-r--r-- 1 root root 27070 Jun 26 04:13
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > /lib64/modules/4.9.34-rancher/kernel/drivers/md/dm-mirror.ko
>> >> >>> >> >> >> > -rw-r--r-- 1 root root 92310 Jun 26 04:13
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > /lib/modules/4.9.34-rancher/kernel/drivers/md/dm-thin-pool.ko
>> >> >>> >> >> >> > -rw-r--r-- 1 root root 56982 Jun 26 04:13
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > /lib/modules/4.9.34-rancher/kernel/drivers/md/dm-snapshot.ko
>> >> >>> >> >> >> > -rw-r--r-- 1 root root 27070 Jun 26 04:13
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > /lib/modules/4.9.34-rancher/kernel/drivers/md/dm-mirror.ko
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > Error snapshot attached.
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > In my first mail, i checked that Readiness Probe failure
>> >> >>> >> >> >> > check
>> >> >>> >> >> >> > has
>> >> >>> >> >> >> > this
>> >> >>> >> >> >> > code
>> >> >>> >> >> >> > in kube-templates/glusterfs-daemonset.yaml file:
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >>> >> >> >> > readinessProbe:
>> >> >>> >> >> >> > timeoutSeconds: 3
>> >> >>> >> >> >> > initialDelaySeconds: 40
>> >> >>> >> >> >> > exec:
>> >> >>> >> >> >> > command:
>> >> >>> >> >> >> > - "/bin/bash"
>> >> >>> >> >> >> > - "-c"
>> >> >>> >> >> >> > - systemctl status glusterd.service
>> >> >>> >> >> >> > periodSeconds: 25
>> >> >>> >> >> >> > successThreshold: 1
>> >> >>> >> >> >> > failureThreshold: 15
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > I tried logging into glustefs container on one of the
>> >> >>> >> >> >> > node
>> >> >>> >> >> >> > and
>> >> >>> >> >> >> > ran
>> >> >>> >> >> >> > the
>> >> >>> >> >> >> > above
>> >> >>> >> >> >> > command:
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > [root at node-a ~]# docker exec -it c0f8ab4d92a23b6df2
>> >> >>> >> >> >> > /bin/bash
>> >> >>> >> >> >> > root at c0f8ab4d92a2:/app# systemctl status
>> >> >>> >> >> >> > glusterd.service
>> >> >>> >> >> >> > WARNING: terminal is not fully functional
>> >> >>> >> >> >> > Failed to connect to bus: No such file or directory
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > Any check that i can do manually on nodes to debug
>> >> >>> >> >> >> > further?
>> >> >>> >> >> >> > Any
>> >> >>> >> >> >> > suggestions?
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> > On Thu, Aug 31, 2017 at 6:53 PM, Jose A. Rivera
>> >> >>> >> >> >> > <jarrpa at redhat.com>
>> >> >>> >> >> >> > wrote:
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >> Hey Gaurav,
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >> The kernel modules must be loaded on all nodes that
>> >> >>> >> >> >> >> will
>> >> >>> >> >> >> >> run
>> >> >>> >> >> >> >> heketi
>> >> >>> >> >> >> >> pods. Additionally, you must have at least three nodes
>> >> >>> >> >> >> >> specified
>> >> >>> >> >> >> >> in
>> >> >>> >> >> >> >> your topology file. I'm not sure how you're getting
>> >> >>> >> >> >> >> three
>> >> >>> >> >> >> >> gluster
>> >> >>> >> >> >> >> pods
>> >> >>> >> >> >> >> when you only have two nodes defined... :)
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >> --Jose
>> >> >>> >> >> >> >>
>> >> >>> >> >> >> >> On Wed, Aug 30, 2017 at 5:27 AM, Gaurav Chhabra
>> >> >>> >> >> >> >> <varuag.chhabra at gmail.com> wrote:
>> >> >>> >> >> >> >> > Hi,
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > I have the following setup in place:
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > 1 node : RancherOS having Rancher application for
>> >> >>> >> >> >> >> > Kubernetes
>> >> >>> >> >> >> >> > setup
>> >> >>> >> >> >> >> > 2 nodes : RancherOS having Rancher agent
>> >> >>> >> >> >> >> > 1 node : CentOS 7 workstation having kubectl
>> >> >>> >> >> >> >> > installed
>> >> >>> >> >> >> >> > and
>> >> >>> >> >> >> >> > folder
>> >> >>> >> >> >> >> > cloned/downloaded from
>> >> >>> >> >> >> >> > https://github.com/gluster/gluster-kubernetes
>> >> >>> >> >> >> >> > using
>> >> >>> >> >> >> >> > which i run Heketi setup (gk-deploy -g)
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > I also have rancher-glusterfs-server container
>> >> >>> >> >> >> >> > running
>> >> >>> >> >> >> >> > with
>> >> >>> >> >> >> >> > the
>> >> >>> >> >> >> >> > following
>> >> >>> >> >> >> >> > configuration:
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > ------------------------------------------------------------------
>> >> >>> >> >> >> >> > [root at node-1 rancher]# cat gluster-server.sh
>> >> >>> >> >> >> >> > #!/bin/bash
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > sudo docker run --name=gluster-server -d \
>> >> >>> >> >> >> >> > --env 'SERVICE_NAME=gluster' \
>> >> >>> >> >> >> >> > --restart always \
>> >> >>> >> >> >> >> > --env 'GLUSTER_DATA=/srv/docker/gitlab' \
>> >> >>> >> >> >> >> > --publish 2222:22 \
>> >> >>> >> >> >> >> > webcenter/rancher-glusterfs-server
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > ------------------------------------------------------------------
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > In /etc/heketi/heketi.json, following is the only
>> >> >>> >> >> >> >> > modified
>> >> >>> >> >> >> >> > portion:
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > ------------------------------------------------------------------
>> >> >>> >> >> >> >> > "executor": "ssh",
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > "_sshexec_comment": "SSH username and private key
>> >> >>> >> >> >> >> > file
>> >> >>> >> >> >> >> > information",
>> >> >>> >> >> >> >> > "sshexec": {
>> >> >>> >> >> >> >> > "keyfile": "/var/lib/heketi/.ssh/id_rsa",
>> >> >>> >> >> >> >> > "user": "root",
>> >> >>> >> >> >> >> > "port": "22",
>> >> >>> >> >> >> >> > "fstab": "/etc/fstab"
>> >> >>> >> >> >> >> > },
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > ------------------------------------------------------------------
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > Status before running gk-deploy:
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > [root at workstation deploy]# kubectl get
>> >> >>> >> >> >> >> > nodes,pods,services,deployments
>> >> >>> >> >> >> >> > NAME STATUS
>> >> >>> >> >> >> >> > AGE
>> >> >>> >> >> >> >> > VERSION
>> >> >>> >> >> >> >> > no/node-1.c.kubernetes-174104.internal Ready 2d
>> >> >>> >> >> >> >> > v1.7.2-rancher1
>> >> >>> >> >> >> >> > no/node-2.c.kubernetes-174104.internal Ready 2d
>> >> >>> >> >> >> >> > v1.7.2-rancher1
>> >> >>> >> >> >> >> > no/node-3.c.kubernetes-174104.internal Ready 2d
>> >> >>> >> >> >> >> > v1.7.2-rancher1
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > NAME CLUSTER-IP EXTERNAL-IP PORT(S)
>> >> >>> >> >> >> >> > AGE
>> >> >>> >> >> >> >> > svc/kubernetes 10.43.0.1 <none> 443/TCP
>> >> >>> >> >> >> >> > 2d
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > Now when i run 'gk-deploy -g', in the Rancher
>> >> >>> >> >> >> >> > console, i
>> >> >>> >> >> >> >> > see
>> >> >>> >> >> >> >> > the
>> >> >>> >> >> >> >> > following
>> >> >>> >> >> >> >> > error:
>> >> >>> >> >> >> >> > Readiness probe failed: Failed to get D-Bus
>> >> >>> >> >> >> >> > connection:
>> >> >>> >> >> >> >> > Operation
>> >> >>> >> >> >> >> > not
>> >> >>> >> >> >> >> > permitted
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > From the attached gk-deploy_log i see that it failed
>> >> >>> >> >> >> >> > at:
>> >> >>> >> >> >> >> > Waiting for GlusterFS pods to start ... pods not
>> >> >>> >> >> >> >> > found.
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > In the kube-templates/glusterfs-daemonset.yaml file,
>> >> >>> >> >> >> >> > i
>> >> >>> >> >> >> >> > see
>> >> >>> >> >> >> >> > this
>> >> >>> >> >> >> >> > for
>> >> >>> >> >> >> >> > Readiness probe section:
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > ------------------------------------------------------------------
>> >> >>> >> >> >> >> > readinessProbe:
>> >> >>> >> >> >> >> > timeoutSeconds: 3
>> >> >>> >> >> >> >> > initialDelaySeconds: 40
>> >> >>> >> >> >> >> > exec:
>> >> >>> >> >> >> >> > command:
>> >> >>> >> >> >> >> > - "/bin/bash"
>> >> >>> >> >> >> >> > - "-c"
>> >> >>> >> >> >> >> > - systemctl status glusterd.service
>> >> >>> >> >> >> >> > periodSeconds: 25
>> >> >>> >> >> >> >> > successThreshold: 1
>> >> >>> >> >> >> >> > failureThreshold: 15
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > ------------------------------------------------------------------
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > Status after running gk-deploy:
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > [root at workstation deploy]# kubectl get
>> >> >>> >> >> >> >> > nodes,pods,deployments,services
>> >> >>> >> >> >> >> > NAME STATUS
>> >> >>> >> >> >> >> > AGE
>> >> >>> >> >> >> >> > VERSION
>> >> >>> >> >> >> >> > no/node-1.c.kubernetes-174104.internal Ready 2d
>> >> >>> >> >> >> >> > v1.7.2-rancher1
>> >> >>> >> >> >> >> > no/node-2.c.kubernetes-174104.internal Ready 2d
>> >> >>> >> >> >> >> > v1.7.2-rancher1
>> >> >>> >> >> >> >> > no/node-3.c.kubernetes-174104.internal Ready 2d
>> >> >>> >> >> >> >> > v1.7.2-rancher1
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > NAME READY STATUS RESTARTS
>> >> >>> >> >> >> >> > AGE
>> >> >>> >> >> >> >> > po/glusterfs-0s440 0/1 Running 0
>> >> >>> >> >> >> >> > 1m
>> >> >>> >> >> >> >> > po/glusterfs-j7dgr 0/1 Running 0
>> >> >>> >> >> >> >> > 1m
>> >> >>> >> >> >> >> > po/glusterfs-p6jl3 0/1 Running 0
>> >> >>> >> >> >> >> > 1m
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > NAME CLUSTER-IP EXTERNAL-IP PORT(S)
>> >> >>> >> >> >> >> > AGE
>> >> >>> >> >> >> >> > svc/kubernetes 10.43.0.1 <none> 443/TCP
>> >> >>> >> >> >> >> > 2d
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > Also, from prerequisite perspective, i was also
>> >> >>> >> >> >> >> > seeing
>> >> >>> >> >> >> >> > this
>> >> >>> >> >> >> >> > mentioned:
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > The following kernel modules must be loaded:
>> >> >>> >> >> >> >> > * dm_snapshot
>> >> >>> >> >> >> >> > * dm_mirror
>> >> >>> >> >> >> >> > * dm_thin_pool
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > Where exactly is this to be checked? On all Gluster
>> >> >>> >> >> >> >> > server
>> >> >>> >> >> >> >> > nodes?
>> >> >>> >> >> >> >> > How
>> >> >>> >> >> >> >> > can i
>> >> >>> >> >> >> >> > check whether it's there?
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > I have attached topology.json and gk-deploy log for
>> >> >>> >> >> >> >> > reference.
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > Does this issue has anything to do with the host OS
>> >> >>> >> >> >> >> > (RancherOS)
>> >> >>> >> >> >> >> > that
>> >> >>> >> >> >> >> > i
>> >> >>> >> >> >> >> > am
>> >> >>> >> >> >> >> > using for Gluster nodes? Any idea how i can fix this?
>> >> >>> >> >> >> >> > Any
>> >> >>> >> >> >> >> > help
>> >> >>> >> >> >> >> > will
>> >> >>> >> >> >> >> > really
>> >> >>> >> >> >> >> > be appreciated.
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > Thanks.
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > _______________________________________________
>> >> >>> >> >> >> >> > heketi-devel mailing list
>> >> >>> >> >> >> >> > heketi-devel at gluster.org
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >> > http://lists.gluster.org/mailman/listinfo/heketi-devel
>> >> >>> >> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >> >
>> >> >>> >> >> >
>> >> >>> >> >> >
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >
>> >> >>> >
>> >> >>
>> >> >>
>> >> >
>> >
>> >
>
>
More information about the heketi-devel
mailing list