[heketi-devel] Unable to use Heketi setup to install Gluster for Kubernetes
Gaurav Chhabra
varuag.chhabra at gmail.com
Tue Sep 5 11:15:26 UTC 2017
Hi Jose,
There is some progress after using the following container:
https://hub.docker.com/r/gluster/glusterfs-client/
I created deployment with 5 replicas for Nginx in Kubernetes using your doc
<https://github.com/gluster/gluster-kubernetes/tree/master/docs/examples/dynamic_provisioning_external_gluster>.
Now as described in the last section of your doc, if i create file
'index.html' inside of a container running in a pod on node-a, i can see
the same file becomes available on another pod running on node-b. However,
if i delete the file on one node (say node-a), it becomes unavailable on
all the three nodes. The file should have been accessible from other nodes
even if i delete it from one pod on a given node. Correct me if i'm getting
it wrong.
I searched for index.html on the Gluster cluster and i could find it on all
three nodes with same content. Please check the attached log.
If i check /etc/fstab on any of the pods, i see the following:
root at nginx-deploy-4253768608-w645w:/# cat /etc/fstab
# UNCONFIGURED FSTAB FOR BASE SYSTEM
Attached log files.
Any idea why the file is not accessible from another pod on node-a or
node-b if i delete it from a different pod present on another node-c?
Regards,
Gaurav
On Tue, Sep 5, 2017 at 3:41 AM, Jose A. Rivera <jarrpa at redhat.com> wrote:
> Yup, I follow that repo and should be contributing to it in the
> near-future. :) I've never played with the glusterfs-client image, but
> here's hoping!
>
> On Mon, Sep 4, 2017 at 10:18 AM, Gaurav Chhabra
> <varuag.chhabra at gmail.com> wrote:
> > Actually the script is launching a container and i'm pretty sure that the
> > command that's programmed as part of the container launch process is
> > actually trying to run that mount command. Now the issue is that this
> again
> > falls under Docker area. :( Generally, Docker container providers (in
> this
> > case, nixel) provide a Dockerfile which specifies the command that's
> > executed as part of the container launch process however, the problem in
> > case of this container is that it's not maintained. It's 2 years old. If
> you
> > will check active projects such as gluster/gluster-containers, you will
> see
> > a proper Dockerfile explaining what it does.
> >
> > https://hub.docker.com/r/gluster/glusterfs-client/~/dockerfile/
> >
> > I am now planning to use the above container and see whether i'm lucky.
> If
> > you will see the above link, it's running
> >
> > dnf install -y glusterfs-fuse
> >
> > Hopefully, this should take care of this section which asks to make sure
> > that glusterfs-client is installed
> >
> > https://github.com/gluster/gluster-kubernetes/tree/
> master/docs/examples/dynamic_provisioning_external_gluster#
> validate-communication-and-gluster-prerequisites-on-the-kubernetes-nodes
> >
> >
> >
> >
> > On Mon, Sep 4, 2017 at 7:33 PM, Jose A. Rivera <jarrpa at redhat.com>
> wrote:
> >>
> >> Something is trying to mount a GlusterFS volume, and it's not that
> >> script. What's trying to mount?
> >>
> >> On Mon, Sep 4, 2017 at 8:52 AM, Gaurav Chhabra <
> varuag.chhabra at gmail.com>
> >> wrote:
> >> > I am just running this bash script which will start the
> >> > rancher-glusterfs-client container:
> >> >
> >> > [root at node-a ~]# cat rancher-glusterfs-client.sh
> >> > #!/bin/bash
> >> > sudo docker run --privileged \
> >> > --name=gluster-client \
> >> > -d \
> >> > -v /sys/fs/cgroup:/sys/fs/cgroup \
> >> > -v /var/log/glusterfs:/var/log/glusterfs \
> >> > --env GLUSTER_PEER=10.128.0.12,10.128.0.15,10.128.0.16 \
> >> > nixel/rancher-glusterfs-client
> >> >
> >> >
> >> >
> >> > On Mon, Sep 4, 2017 at 6:26 PM, Jose A. Rivera <jarrpa at redhat.com>
> >> > wrote:
> >> >>
> >> >> What is the exact command you're running?
> >> >>
> >> >> On Mon, Sep 4, 2017 at 4:26 AM, Gaurav Chhabra
> >> >> <varuag.chhabra at gmail.com>
> >> >> wrote:
> >> >> > Hi Jose,
> >> >> >
> >> >> >
> >> >> > From this link, it seems mount.glusterfs might actually be present
> in
> >> >> > the
> >> >> > container that launched and quickly terminated.
> >> >> >
> >> >> >
> >> >> >
> >> >> > https://unix.stackexchange.com/questions/312178/
> glusterfs-replicated-volume-mounting-issue
> >> >> >
> >> >> > If you check the question that the user posted, the same error
> (Mount
> >> >> > failed) was reported that i sent you in the last email.
> >> >> >
> >> >> > After seeing the above, i checked /var/log/glusterfs on my host
> >> >> > (RancherOS)
> >> >> > but it was empty. I ran the container again but with explicit
> volume
> >> >> > mount
> >> >> > as shown below:
> >> >> >
> >> >> > [root at node-a ~]# cat rancher-glusterfs-client.sh
> >> >> > #!/bin/bash
> >> >> > sudo docker run --privileged \
> >> >> > --name=gluster-client \
> >> >> > -d \
> >> >> > -v /sys/fs/cgroup:/sys/fs/cgroup \
> >> >> > -v /var/log/glusterfs:/var/log/glusterfs \
> >> >> > --env GLUSTER_PEER=10.128.0.12,10.128.0.15,10.128.0.16 \
> >> >> > nixel/rancher-glusterfs-client
> >> >> > This time, i could see a log file
> >> >> > (/var/log/glusterfs/mnt-ranchervol.log)
> >> >> > present. I have attached the content of the same. Also attached are
> >> >> > logs
> >> >> > from Heketi client/server (both on one node) and Gluster cluster.
> >> >> >
> >> >> >
> >> >> > Regards,
> >> >> > Gaurav
> >> >> >
> >> >> >
> >> >> > On Mon, Sep 4, 2017 at 2:29 PM, Gaurav Chhabra
> >> >> > <varuag.chhabra at gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> Hi Jose,
> >> >> >>
> >> >> >>
> >> >> >> I tried setting up things using the link you provided and i was
> able
> >> >> >> to
> >> >> >> get all steps working for 3 node Gluster cluster, all running on
> >> >> >> CentOS7,
> >> >> >> without any issue. However, as expected, when i tried configuring
> >> >> >> Kubernetes
> >> >> >> by installing nixel/rancher-glusterfs-client container, i got
> error:
> >> >> >>
> >> >> >> [root at node-a ~]# cat rancher-glusterfs-client.sh
> >> >> >> #!/bin/bash
> >> >> >> sudo docker run --privileged \
> >> >> >> --name=gluster-client \
> >> >> >> -d \
> >> >> >> -v /sys/fs/cgroup:/sys/fs/cgroup \
> >> >> >> --env GLUSTER_PEER=10.128.0.12,10.128.0.15,10.128.0.16 \
> >> >> >> nixel/rancher-glusterfs-client
> >> >> >>
> >> >> >> [root at node-a ~]# ./rancher-glusterfs-client.sh
> >> >> >> ac069caccdce147d6f423fc566166345191dbc1b11f3416c66207a1fd11fda6b
> >> >> >>
> >> >> >> [root at node-a ~]# docker logs gluster-client
> >> >> >> => Checking if I can reach GlusterFS node 10.128.0.12 ...
> >> >> >> => GlusterFS node 10.128.0.12 is alive
> >> >> >> => Mounting GlusterFS volume ranchervol from GlusterFS node
> >> >> >> 10.128.0.12
> >> >> >> ...
> >> >> >> Mount failed. Please check the log file for more details.
> >> >> >>
> >> >> >> If i try running the next step as described in your link, i get
> the
> >> >> >> following:
> >> >> >>
> >> >> >> [root at node-a ~]# modprobe fuse
> >> >> >> modprobe: module fuse not found in modules.dep
> >> >> >>
> >> >> >> Since the container failed to start, i could only check on the
> host
> >> >> >> (RancherOS) and i could only find two mount-related commands:
> mount
> >> >> >> &
> >> >> >> mountpoint
> >> >> >>
> >> >> >> Any pointers?
> >> >> >>
> >> >> >>
> >> >> >> Regards,
> >> >> >> Gaurav
> >> >> >>
> >> >> >> On Sun, Sep 3, 2017 at 11:18 PM, Jose A. Rivera <
> jarrpa at redhat.com>
> >> >> >> wrote:
> >> >> >>>
> >> >> >>> Installing the glusterfs-client container should be fine. :) The
> >> >> >>> main
> >> >> >>> thing that's needed is that all your Kubernetes nodes need to
> have
> >> >> >>> the
> >> >> >>> "mount.glusterfs" command available so Kube can mount the
> GlusterFS
> >> >> >>> volumes and present them to the pods.
> >> >> >>>
> >> >> >>> On Sun, Sep 3, 2017 at 12:14 PM, Gaurav Chhabra
> >> >> >>> <varuag.chhabra at gmail.com> wrote:
> >> >> >>> > Thanks Jose. The link you've suggested looks good but again it
> >> >> >>> > expects
> >> >> >>> > me to
> >> >> >>> > install gluster-client on Kubernetes node and i fall into the
> >> >> >>> > same
> >> >> >>> > issue of
> >> >> >>> > installing a container for glusterfs. Only difference is that
> >> >> >>> > this
> >> >> >>> > time
> >> >> >>> > it's
> >> >> >>> > glusterfs-client and not glusterfs-server. :)
> >> >> >>> >
> >> >> >>> > I will try this out and let you know tomorrow.
> >> >> >>> >
> >> >> >>> >
> >> >> >>> > Regards,
> >> >> >>> > Gaurav
> >> >> >>> >
> >> >> >>> >
> >> >> >>> > On Sun, Sep 3, 2017 at 2:11 AM, Jose A. Rivera
> >> >> >>> > <jarrpa at redhat.com>
> >> >> >>> > wrote:
> >> >> >>> >>
> >> >> >>> >> Hey, no problem! I'm eager to learn more about different
> flavors
> >> >> >>> >> of
> >> >> >>> >> Linux, I just apologize for my relative inexperience with
> them.
> >> >> >>> >> :)
> >> >> >>> >>
> >> >> >>> >> To that end, I will also admit I'm not very experienced with
> >> >> >>> >> direct
> >> >> >>> >> Docker myself. I understand the basic workflow and know some
> of
> >> >> >>> >> the
> >> >> >>> >> run options, but not having deep experience keeps me from
> having
> >> >> >>> >> a
> >> >> >>> >> better understanding of the patterns and consequences.
> >> >> >>> >>
> >> >> >>> >> Thus, I'd like to guide you in a direction I'd more apt to
> help
> >> >> >>> >> you
> >> >> >>> >> in
> >> >> >>> >> right now. I know that you can't have multiple GlusterFS
> servers
> >> >> >>> >> running on the same nodes, and I know that we have been
> >> >> >>> >> successfully
> >> >> >>> >> running several configurations using our
> gluster/gluster-centos
> >> >> >>> >> image.
> >> >> >>> >> If you follow the Kubernetes configuration on
> >> >> >>> >> gluster-kubernetes,
> >> >> >>> >> the
> >> >> >>> >> pod/container is run privileged and with host networking, and
> we
> >> >> >>> >> require that the node has all listed ports open, not just
> 2222.
> >> >> >>> >> The
> >> >> >>> >> sshd running in the container is listening on 2222, not 22,
> but
> >> >> >>> >> it
> >> >> >>> >> is
> >> >> >>> >> also not really required if you're not doing geo-replication.
> >> >> >>> >>
> >> >> >>> >> Alternatively, you can indeed run GlusterFS outside of
> >> >> >>> >> Kubernetes
> >> >> >>> >> but
> >> >> >>> >> still have Kubernetes apps access GlusterFS storage. The nodes
> >> >> >>> >> can
> >> >> >>> >> be
> >> >> >>> >> anything you want, they just need to be running GlusterFS and
> >> >> >>> >> you
> >> >> >>> >> need
> >> >> >>> >> a heketi service managing them. Here is an example of how to
> set
> >> >> >>> >> this
> >> >> >>> >> up using CentOS:
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >> https://github.com/gluster/gluster-kubernetes/tree/
> master/docs/examples/dynamic_provisioning_external_gluster
> >> >> >>> >>
> >> >> >>> >> Hope this is at least leading you in a useful direction. :)
> >> >> >>> >>
> >> >> >>> >> --Jose
> >> >> >>> >>
> >> >> >>> >> On Sat, Sep 2, 2017 at 3:16 PM, Gaurav Chhabra
> >> >> >>> >> <varuag.chhabra at gmail.com>
> >> >> >>> >> wrote:
> >> >> >>> >> > Hi Jose,
> >> >> >>> >> >
> >> >> >>> >> > webcenter/rancher-glusterfs-server is actually a container
> >> >> >>> >> > provided
> >> >> >>> >> > by
> >> >> >>> >> > Sebastien, its maintainer. It's a Docker container which has
> >> >> >>> >> > GlusterFS
> >> >> >>> >> > server running within it. On the host i.e., RancherOS, there
> >> >> >>> >> > is
> >> >> >>> >> > no
> >> >> >>> >> > separate
> >> >> >>> >> > GlusterFS server running because we cannot install anything
> >> >> >>> >> > that
> >> >> >>> >> > way.
> >> >> >>> >> > Running using container is the only way so i started
> >> >> >>> >> > ancher-glusterfs-server
> >> >> >>> >> > container with the following parameters:
> >> >> >>> >> >
> >> >> >>> >> > [root at node-1 rancher]# cat gluster-server.sh
> >> >> >>> >> > #!/bin/bash
> >> >> >>> >> > sudo docker run --name=gluster-server -d \
> >> >> >>> >> > --env 'SERVICE_NAME=gluster' \
> >> >> >>> >> > --restart always \
> >> >> >>> >> > --publish 2222:22 \
> >> >> >>> >> > webcenter/rancher-glusterfs-server
> >> >> >>> >> >
> >> >> >>> >> > Here's the link to the dockerfile:
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >> > https://hub.docker.com/r/webcenter/rancher-glusterfs-
> server/~/dockerfile/
> >> >> >>> >> >
> >> >> >>> >> > It's similar to other GlusteFS containers provided by other
> >> >> >>> >> > maintainers
> >> >> >>> >> > for
> >> >> >>> >> > different OS. For example, for CentOS, we have
> >> >> >>> >> > https://hub.docker.com/r/gluster/gluster-centos/~/
> dockerfile/
> >> >> >>> >> >
> >> >> >>> >> > From what i understand, Heketi does support container based
> >> >> >>> >> > GlusterFS
> >> >> >>> >> > server
> >> >> >>> >> > as mentioned in the prerequisite where it says:
> >> >> >>> >> >
> >> >> >>> >> > "Each node must have the following ports opened for
> GlusterFS
> >> >> >>> >> > communications:
> >> >> >>> >> > 2222 - GlusterFS pod's sshd"
> >> >> >>> >> >
> >> >> >>> >> > That's the reason i've exposed port 2222 for 22 as shown
> >> >> >>> >> > above.
> >> >> >>> >> > Please
> >> >> >>> >> > correct me if i misunderstood it.
> >> >> >>> >> >
> >> >> >>> >> > As soon as i run the above script (gluster-server.sh), it
> >> >> >>> >> > automatically
> >> >> >>> >> > creates the following directories on host. This should have
> >> >> >>> >> > ideally
> >> >> >>> >> > not
> >> >> >>> >> > been
> >> >> >>> >> > empty as you mentioned.
> >> >> >>> >> >
> >> >> >>> >> > /etc/glusterfs /var/lib/glusterd /var/log/glusterfs
> >> >> >>> >> >
> >> >> >>> >> > Just wanted to know in which circumstances do we get this
> >> >> >>> >> > specific
> >> >> >>> >> > error
> >> >> >>> >> > (Failed to get D-Bus connection: Operation not permitted)
> >> >> >>> >> > related
> >> >> >>> >> > to
> >> >> >>> >> > Readiness probe failing. Searching online took me to
> >> >> >>> >> > discussions
> >> >> >>> >> > around
> >> >> >>> >> > running container in privileged mode and some directory to
> be
> >> >> >>> >> > mounted.
> >> >> >>> >> > Based
> >> >> >>> >> > on that, i also modified my container startup script to the
> >> >> >>> >> > following:
> >> >> >>> >> >
> >> >> >>> >> > #!/bin/bash
> >> >> >>> >> > sudo docker run --privileged \
> >> >> >>> >> > --name=gluster-server \
> >> >> >>> >> > -d \
> >> >> >>> >> > -v /sys/fs/cgroup:/sys/fs/cgroup \
> >> >> >>> >> > -v /etc/glusterfs:/etc/glusterfs \
> >> >> >>> >> > -v /var/lib/glusterd:/var/lib/glusterd \
> >> >> >>> >> > -v /var/log/glusterfs:/var/log/glusterfs \
> >> >> >>> >> > --env 'SERVICE_NAME=gluster' \
> >> >> >>> >> > --restart always \
> >> >> >>> >> > --publish 2222:22 \
> >> >> >>> >> > webcenter/rancher-glusterfs-server
> >> >> >>> >> > Still, the issue persists.
> >> >> >>> >> >
> >> >> >>> >> > I also logged into the container and checked whether
> systemctl
> >> >> >>> >> > command
> >> >> >>> >> > is
> >> >> >>> >> > present. It was there but manualy running the command also
> >> >> >>> >> > doesn't
> >> >> >>> >> > work:
> >> >> >>> >> >
> >> >> >>> >> > [root at node-c ~]# docker exec -it gluster-server /bin/bash
> >> >> >>> >> > root at 42150f203f80:/app# systemctl status glusterd.service
> >> >> >>> >> > WARNING: terminal is not fully functional
> >> >> >>> >> > Failed to connect to bus: No such file or directory
> >> >> >>> >> >
> >> >> >>> >> > Under section 'ADVANCED OPTIONS - Security/Host' in this
> link,
> >> >> >>> >> > it
> >> >> >>> >> > talks
> >> >> >>> >> > about SYS_ADMIN setting. Any idea how i can try this?
> >> >> >>> >> >
> >> >> >>> >> > Also, there was this mentioned in the Heketi setup page:
> >> >> >>> >> > "If you are not able to deploy a hyper-converged GlusterFS
> >> >> >>> >> > cluster,
> >> >> >>> >> > you
> >> >> >>> >> > must
> >> >> >>> >> > have one running somewhere that the Kubernetes nodes can
> >> >> >>> >> > access"
> >> >> >>> >> >
> >> >> >>> >> >>>> Does it mean running the three node Gluster cluster
> outside
> >> >> >>> >> >>>> Kubernetes,
> >> >> >>> >> >>>> may be on some VM running on RHEL/CentOS etc? If yes,
> then
> >> >> >>> >> >>>> how
> >> >> >>> >> >>>> will i
> >> >> >>> >> >>>> be
> >> >> >>> >> >>>> able to tell Gluster which volume from the Kubernetes
> >> >> >>> >> >>>> cluster
> >> >> >>> >> >>>> pod
> >> >> >>> >> >>>> to
> >> >> >>> >> >>>> sync?
> >> >> >>> >> >>>> Any references?
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >> > I really appreciate your responses despite the fact that
> >> >> >>> >> > you've
> >> >> >>> >> > not
> >> >> >>> >> > used
> >> >> >>> >> > RancherOS but still trying to help.
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >> > Thanks,
> >> >> >>> >> > Gaurav
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >> > On Sat, Sep 2, 2017 at 7:35 PM, Jose A. Rivera
> >> >> >>> >> > <jarrpa at redhat.com>
> >> >> >>> >> > wrote:
> >> >> >>> >> >>
> >> >> >>> >> >> I'm afraid I have no experience with RancherOS, so I may be
> >> >> >>> >> >> missing
> >> >> >>> >> >> some things about how it works. My primary experience is
> with
> >> >> >>> >> >> Fedora,
> >> >> >>> >> >> CentOS, and Ubuntu.
> >> >> >>> >> >>
> >> >> >>> >> >> What is webcenter/rancher-glusterfs-server? If it's
> running
> >> >> >>> >> >> another
> >> >> >>> >> >> glusterd then you probably don't want to be running it and
> >> >> >>> >> >> should
> >> >> >>> >> >> remove it from your systems.
> >> >> >>> >> >>
> >> >> >>> >> >> The glusterfs pods mount hostpath volumes from the host
> >> >> >>> >> >> they're
> >> >> >>> >> >> running on to persist their configuration. Thus anything
> they
> >> >> >>> >> >> write
> >> >> >>> >> >> to
> >> >> >>> >> >> those directories should land on the host. If that's not
> >> >> >>> >> >> happening
> >> >> >>> >> >> then that's an additional problem.
> >> >> >>> >> >>
> >> >> >>> >> >> --Jose
> >> >> >>> >> >>
> >> >> >>> >> >> On Fri, Sep 1, 2017 at 11:17 PM, Gaurav Chhabra
> >> >> >>> >> >> <varuag.chhabra at gmail.com> wrote:
> >> >> >>> >> >> > Hi Jose,
> >> >> >>> >> >> >
> >> >> >>> >> >> >
> >> >> >>> >> >> > I tried your suggestion but there is one confusion
> >> >> >>> >> >> > regarding
> >> >> >>> >> >> > point
> >> >> >>> >> >> > #3.
> >> >> >>> >> >> > Since
> >> >> >>> >> >> > RancherOS has everything running as container, i am
> running
> >> >> >>> >> >> > webcenter/rancher-glusterfs-server container on all
> three
> >> >> >>> >> >> > nodes.
> >> >> >>> >> >> > Now
> >> >> >>> >> >> > as
> >> >> >>> >> >> > far
> >> >> >>> >> >> > as removing the directories are concerned, i hope you
> meant
> >> >> >>> >> >> > removing
> >> >> >>> >> >> > them on
> >> >> >>> >> >> > the host and _not_ from within the container. After
> >> >> >>> >> >> > completing
> >> >> >>> >> >> > step 1
> >> >> >>> >> >> > and 2,
> >> >> >>> >> >> > i checked the contents of all the directories that you
> >> >> >>> >> >> > specified
> >> >> >>> >> >> > in
> >> >> >>> >> >> > point
> >> >> >>> >> >> > #3. All were empty as you can see in the attached
> >> >> >>> >> >> > other_logs.txt.
> >> >> >>> >> >> > So
> >> >> >>> >> >> > i
> >> >> >>> >> >> > got
> >> >> >>> >> >> > confused. I ran the deploy again but the issue persists.
> >> >> >>> >> >> > Two
> >> >> >>> >> >> > pods
> >> >> >>> >> >> > show
> >> >> >>> >> >> > Liveness error and the third one, Readiness error.
> >> >> >>> >> >> >
> >> >> >>> >> >> > I then tried removing those directories (Step #3) from
> >> >> >>> >> >> > within
> >> >> >>> >> >> > the
> >> >> >>> >> >> > container
> >> >> >>> >> >> > but getting following error:
> >> >> >>> >> >> >
> >> >> >>> >> >> > root at c0f8ab4d92a2:/app# rm -rf /var/lib/heketi
> >> >> >>> >> >> > /etc/glusterfs
> >> >> >>> >> >> > /var/lib/glusterd /var/log/glusterfs
> >> >> >>> >> >> > rm: cannot remove '/var/lib/glusterd': Device or resource
> >> >> >>> >> >> > busy
> >> >> >>> >> >> >
> >> >> >>> >> >> >
> >> >> >>> >> >> >
> >> >> >>> >> >> > On Fri, Sep 1, 2017 at 8:21 PM, Jose A. Rivera
> >> >> >>> >> >> > <jarrpa at redhat.com>
> >> >> >>> >> >> > wrote:
> >> >> >>> >> >> >>
> >> >> >>> >> >> >> 1. Add a line to the ssh-exec portion of heketi.json of
> >> >> >>> >> >> >> the
> >> >> >>> >> >> >> sort:
> >> >> >>> >> >> >>
> >> >> >>> >> >> >> "sudo": true,
> >> >> >>> >> >> >>
> >> >> >>> >> >> >> 2. Run
> >> >> >>> >> >> >>
> >> >> >>> >> >> >> gk-deploy -g --abort
> >> >> >>> >> >> >>
> >> >> >>> >> >> >> 3. On the nodes that were/will be running GlusterFS
> pods,
> >> >> >>> >> >> >> run:
> >> >> >>> >> >> >>
> >> >> >>> >> >> >> rm -rf /var/lib/heketi /etc/glusterfs /var/lib/glusterd
> >> >> >>> >> >> >> /var/log/glusterfs
> >> >> >>> >> >> >>
> >> >> >>> >> >> >> Then try the deploy again.
> >> >> >>> >> >> >>
> >> >> >>> >> >> >> On Fri, Sep 1, 2017 at 6:05 AM, Gaurav Chhabra
> >> >> >>> >> >> >> <varuag.chhabra at gmail.com>
> >> >> >>> >> >> >> wrote:
> >> >> >>> >> >> >> > Hi Jose,
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > Thanks for the reply. It seems the three gluster pods
> >> >> >>> >> >> >> > might
> >> >> >>> >> >> >> > have
> >> >> >>> >> >> >> > been
> >> >> >>> >> >> >> > a
> >> >> >>> >> >> >> > copy-paste from another set of cluster where i was
> >> >> >>> >> >> >> > trying
> >> >> >>> >> >> >> > to
> >> >> >>> >> >> >> > setup
> >> >> >>> >> >> >> > the
> >> >> >>> >> >> >> > same
> >> >> >>> >> >> >> > thing using CentOS. Sorry for that. By the way, i did
> >> >> >>> >> >> >> > check
> >> >> >>> >> >> >> > for
> >> >> >>> >> >> >> > the
> >> >> >>> >> >> >> > kernel
> >> >> >>> >> >> >> > modules and it seems it's already there. Also, i am
> >> >> >>> >> >> >> > attaching
> >> >> >>> >> >> >> > fresh
> >> >> >>> >> >> >> > set
> >> >> >>> >> >> >> > of
> >> >> >>> >> >> >> > files because i created a new cluster and thought of
> >> >> >>> >> >> >> > giving
> >> >> >>> >> >> >> > it
> >> >> >>> >> >> >> > a
> >> >> >>> >> >> >> > try
> >> >> >>> >> >> >> > again.
> >> >> >>> >> >> >> > Issue still persists. :(
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > In heketi.json, there is a slight change w.r.t the
> user
> >> >> >>> >> >> >> > which
> >> >> >>> >> >> >> > connects
> >> >> >>> >> >> >> > to
> >> >> >>> >> >> >> > glusterfs node using SSH. I am not sure how Heketi was
> >> >> >>> >> >> >> > using
> >> >> >>> >> >> >> > root
> >> >> >>> >> >> >> > user
> >> >> >>> >> >> >> > to
> >> >> >>> >> >> >> > login because i wasn't able to use root and do manual
> >> >> >>> >> >> >> > SSH.
> >> >> >>> >> >> >> > With
> >> >> >>> >> >> >> > rancher
> >> >> >>> >> >> >> > user, i can login successfully so i think this should
> be
> >> >> >>> >> >> >> > fine.
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > /etc/heketi/heketi.json:
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >>> >> >> >> > "executor": "ssh",
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > "_sshexec_comment": "SSH username and private key
> >> >> >>> >> >> >> > file
> >> >> >>> >> >> >> > information",
> >> >> >>> >> >> >> > "sshexec": {
> >> >> >>> >> >> >> > "keyfile": "/var/lib/heketi/.ssh/id_rsa",
> >> >> >>> >> >> >> > "user": "rancher",
> >> >> >>> >> >> >> > "port": "22",
> >> >> >>> >> >> >> > "fstab": "/etc/fstab"
> >> >> >>> >> >> >> > },
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > Before running gk-deploy:
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >>> >> >> >> > [root at workstation deploy]# kubectl get
> >> >> >>> >> >> >> > nodes,pods,daemonset,deployments,services
> >> >> >>> >> >> >> > NAME STATUS AGE
> >> >> >>> >> >> >> > VERSION
> >> >> >>> >> >> >> > no/node-a.c.kubernetes-174104.internal Ready 3h
> >> >> >>> >> >> >> > v1.7.2-rancher1
> >> >> >>> >> >> >> > no/node-b.c.kubernetes-174104.internal Ready 3h
> >> >> >>> >> >> >> > v1.7.2-rancher1
> >> >> >>> >> >> >> > no/node-c.c.kubernetes-174104.internal Ready 3h
> >> >> >>> >> >> >> > v1.7.2-rancher1
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > NAME CLUSTER-IP EXTERNAL-IP PORT(S)
> >> >> >>> >> >> >> > AGE
> >> >> >>> >> >> >> > svc/kubernetes 10.43.0.1 <none> 443/TCP
> 3h
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > After running gk-deploy:
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >>> >> >> >> > [root at workstation messagegc]# kubectl get
> >> >> >>> >> >> >> > nodes,pods,daemonset,deployments,services
> >> >> >>> >> >> >> > NAME STATUS AGE
> >> >> >>> >> >> >> > VERSION
> >> >> >>> >> >> >> > no/node-a.c.kubernetes-174104.internal Ready 3h
> >> >> >>> >> >> >> > v1.7.2-rancher1
> >> >> >>> >> >> >> > no/node-b.c.kubernetes-174104.internal Ready 3h
> >> >> >>> >> >> >> > v1.7.2-rancher1
> >> >> >>> >> >> >> > no/node-c.c.kubernetes-174104.internal Ready 3h
> >> >> >>> >> >> >> > v1.7.2-rancher1
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > NAME READY STATUS RESTARTS
> AGE
> >> >> >>> >> >> >> > po/glusterfs-0j9l5 0/1 Running 0 2m
> >> >> >>> >> >> >> > po/glusterfs-gqz4c 0/1 Running 0 2m
> >> >> >>> >> >> >> > po/glusterfs-gxvcb 0/1 Running 0 2m
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > NAME DESIRED CURRENT READY
> UP-TO-DATE
> >> >> >>> >> >> >> > AVAILABLE
> >> >> >>> >> >> >> > NODE-SELECTOR AGE
> >> >> >>> >> >> >> > ds/glusterfs 3 3 0 3
> >> >> >>> >> >> >> > 0
> >> >> >>> >> >> >> > storagenode=glusterfs 2m
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > NAME CLUSTER-IP EXTERNAL-IP PORT(S)
> >> >> >>> >> >> >> > AGE
> >> >> >>> >> >> >> > svc/kubernetes 10.43.0.1 <none> 443/TCP
> 3h
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > Kernel module check on all three nodes:
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >>> >> >> >> > [root at node-a ~]# find /lib*/modules/$(uname -r) -name
> >> >> >>> >> >> >> > *.ko
> >> >> >>> >> >> >> > |
> >> >> >>> >> >> >> > grep
> >> >> >>> >> >> >> > 'thin-pool\|snapshot\|mirror' | xargs ls -ltr
> >> >> >>> >> >> >> > -rw-r--r-- 1 root root 92310 Jun 26
> 04:13
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > /lib64/modules/4.9.34-rancher/
> kernel/drivers/md/dm-thin-pool.ko
> >> >> >>> >> >> >> > -rw-r--r-- 1 root root 56982 Jun 26
> 04:13
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > /lib64/modules/4.9.34-rancher/
> kernel/drivers/md/dm-snapshot.ko
> >> >> >>> >> >> >> > -rw-r--r-- 1 root root 27070 Jun 26
> 04:13
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > /lib64/modules/4.9.34-rancher/
> kernel/drivers/md/dm-mirror.ko
> >> >> >>> >> >> >> > -rw-r--r-- 1 root root 92310 Jun 26
> 04:13
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > /lib/modules/4.9.34-rancher/
> kernel/drivers/md/dm-thin-pool.ko
> >> >> >>> >> >> >> > -rw-r--r-- 1 root root 56982 Jun 26
> 04:13
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > /lib/modules/4.9.34-rancher/
> kernel/drivers/md/dm-snapshot.ko
> >> >> >>> >> >> >> > -rw-r--r-- 1 root root 27070 Jun 26
> 04:13
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > /lib/modules/4.9.34-rancher/
> kernel/drivers/md/dm-mirror.ko
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > Error snapshot attached.
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > In my first mail, i checked that Readiness Probe
> failure
> >> >> >>> >> >> >> > check
> >> >> >>> >> >> >> > has
> >> >> >>> >> >> >> > this
> >> >> >>> >> >> >> > code
> >> >> >>> >> >> >> > in kube-templates/glusterfs-daemonset.yaml file:
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >>> >> >> >> > readinessProbe:
> >> >> >>> >> >> >> > timeoutSeconds: 3
> >> >> >>> >> >> >> > initialDelaySeconds: 40
> >> >> >>> >> >> >> > exec:
> >> >> >>> >> >> >> > command:
> >> >> >>> >> >> >> > - "/bin/bash"
> >> >> >>> >> >> >> > - "-c"
> >> >> >>> >> >> >> > - systemctl status glusterd.service
> >> >> >>> >> >> >> > periodSeconds: 25
> >> >> >>> >> >> >> > successThreshold: 1
> >> >> >>> >> >> >> > failureThreshold: 15
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > I tried logging into glustefs container on one of the
> >> >> >>> >> >> >> > node
> >> >> >>> >> >> >> > and
> >> >> >>> >> >> >> > ran
> >> >> >>> >> >> >> > the
> >> >> >>> >> >> >> > above
> >> >> >>> >> >> >> > command:
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > [root at node-a ~]# docker exec -it c0f8ab4d92a23b6df2
> >> >> >>> >> >> >> > /bin/bash
> >> >> >>> >> >> >> > root at c0f8ab4d92a2:/app# systemctl status
> >> >> >>> >> >> >> > glusterd.service
> >> >> >>> >> >> >> > WARNING: terminal is not fully functional
> >> >> >>> >> >> >> > Failed to connect to bus: No such file or directory
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > Any check that i can do manually on nodes to debug
> >> >> >>> >> >> >> > further?
> >> >> >>> >> >> >> > Any
> >> >> >>> >> >> >> > suggestions?
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> > On Thu, Aug 31, 2017 at 6:53 PM, Jose A. Rivera
> >> >> >>> >> >> >> > <jarrpa at redhat.com>
> >> >> >>> >> >> >> > wrote:
> >> >> >>> >> >> >> >>
> >> >> >>> >> >> >> >> Hey Gaurav,
> >> >> >>> >> >> >> >>
> >> >> >>> >> >> >> >> The kernel modules must be loaded on all nodes that
> >> >> >>> >> >> >> >> will
> >> >> >>> >> >> >> >> run
> >> >> >>> >> >> >> >> heketi
> >> >> >>> >> >> >> >> pods. Additionally, you must have at least three
> nodes
> >> >> >>> >> >> >> >> specified
> >> >> >>> >> >> >> >> in
> >> >> >>> >> >> >> >> your topology file. I'm not sure how you're getting
> >> >> >>> >> >> >> >> three
> >> >> >>> >> >> >> >> gluster
> >> >> >>> >> >> >> >> pods
> >> >> >>> >> >> >> >> when you only have two nodes defined... :)
> >> >> >>> >> >> >> >>
> >> >> >>> >> >> >> >> --Jose
> >> >> >>> >> >> >> >>
> >> >> >>> >> >> >> >> On Wed, Aug 30, 2017 at 5:27 AM, Gaurav Chhabra
> >> >> >>> >> >> >> >> <varuag.chhabra at gmail.com> wrote:
> >> >> >>> >> >> >> >> > Hi,
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > I have the following setup in place:
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > 1 node : RancherOS having Rancher application
> for
> >> >> >>> >> >> >> >> > Kubernetes
> >> >> >>> >> >> >> >> > setup
> >> >> >>> >> >> >> >> > 2 nodes : RancherOS having Rancher agent
> >> >> >>> >> >> >> >> > 1 node : CentOS 7 workstation having kubectl
> >> >> >>> >> >> >> >> > installed
> >> >> >>> >> >> >> >> > and
> >> >> >>> >> >> >> >> > folder
> >> >> >>> >> >> >> >> > cloned/downloaded from
> >> >> >>> >> >> >> >> > https://github.com/gluster/gluster-kubernetes
> >> >> >>> >> >> >> >> > using
> >> >> >>> >> >> >> >> > which i run Heketi setup (gk-deploy -g)
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > I also have rancher-glusterfs-server container
> >> >> >>> >> >> >> >> > running
> >> >> >>> >> >> >> >> > with
> >> >> >>> >> >> >> >> > the
> >> >> >>> >> >> >> >> > following
> >> >> >>> >> >> >> >> > configuration:
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >>> >> >> >> >> > [root at node-1 rancher]# cat gluster-server.sh
> >> >> >>> >> >> >> >> > #!/bin/bash
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > sudo docker run --name=gluster-server -d \
> >> >> >>> >> >> >> >> > --env 'SERVICE_NAME=gluster' \
> >> >> >>> >> >> >> >> > --restart always \
> >> >> >>> >> >> >> >> > --env 'GLUSTER_DATA=/srv/docker/gitlab' \
> >> >> >>> >> >> >> >> > --publish 2222:22 \
> >> >> >>> >> >> >> >> > webcenter/rancher-glusterfs-server
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > In /etc/heketi/heketi.json, following is the only
> >> >> >>> >> >> >> >> > modified
> >> >> >>> >> >> >> >> > portion:
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >>> >> >> >> >> > "executor": "ssh",
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > "_sshexec_comment": "SSH username and private
> key
> >> >> >>> >> >> >> >> > file
> >> >> >>> >> >> >> >> > information",
> >> >> >>> >> >> >> >> > "sshexec": {
> >> >> >>> >> >> >> >> > "keyfile": "/var/lib/heketi/.ssh/id_rsa",
> >> >> >>> >> >> >> >> > "user": "root",
> >> >> >>> >> >> >> >> > "port": "22",
> >> >> >>> >> >> >> >> > "fstab": "/etc/fstab"
> >> >> >>> >> >> >> >> > },
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > Status before running gk-deploy:
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > [root at workstation deploy]# kubectl get
> >> >> >>> >> >> >> >> > nodes,pods,services,deployments
> >> >> >>> >> >> >> >> > NAME STATUS
> >> >> >>> >> >> >> >> > AGE
> >> >> >>> >> >> >> >> > VERSION
> >> >> >>> >> >> >> >> > no/node-1.c.kubernetes-174104.internal Ready
> 2d
> >> >> >>> >> >> >> >> > v1.7.2-rancher1
> >> >> >>> >> >> >> >> > no/node-2.c.kubernetes-174104.internal Ready
> 2d
> >> >> >>> >> >> >> >> > v1.7.2-rancher1
> >> >> >>> >> >> >> >> > no/node-3.c.kubernetes-174104.internal Ready
> 2d
> >> >> >>> >> >> >> >> > v1.7.2-rancher1
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > NAME CLUSTER-IP EXTERNAL-IP PORT(S)
> >> >> >>> >> >> >> >> > AGE
> >> >> >>> >> >> >> >> > svc/kubernetes 10.43.0.1 <none> 443/TCP
> >> >> >>> >> >> >> >> > 2d
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > Now when i run 'gk-deploy -g', in the Rancher
> >> >> >>> >> >> >> >> > console, i
> >> >> >>> >> >> >> >> > see
> >> >> >>> >> >> >> >> > the
> >> >> >>> >> >> >> >> > following
> >> >> >>> >> >> >> >> > error:
> >> >> >>> >> >> >> >> > Readiness probe failed: Failed to get D-Bus
> >> >> >>> >> >> >> >> > connection:
> >> >> >>> >> >> >> >> > Operation
> >> >> >>> >> >> >> >> > not
> >> >> >>> >> >> >> >> > permitted
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > From the attached gk-deploy_log i see that it
> failed
> >> >> >>> >> >> >> >> > at:
> >> >> >>> >> >> >> >> > Waiting for GlusterFS pods to start ... pods not
> >> >> >>> >> >> >> >> > found.
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > In the kube-templates/glusterfs-daemonset.yaml
> file,
> >> >> >>> >> >> >> >> > i
> >> >> >>> >> >> >> >> > see
> >> >> >>> >> >> >> >> > this
> >> >> >>> >> >> >> >> > for
> >> >> >>> >> >> >> >> > Readiness probe section:
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >>> >> >> >> >> > readinessProbe:
> >> >> >>> >> >> >> >> > timeoutSeconds: 3
> >> >> >>> >> >> >> >> > initialDelaySeconds: 40
> >> >> >>> >> >> >> >> > exec:
> >> >> >>> >> >> >> >> > command:
> >> >> >>> >> >> >> >> > - "/bin/bash"
> >> >> >>> >> >> >> >> > - "-c"
> >> >> >>> >> >> >> >> > - systemctl status glusterd.service
> >> >> >>> >> >> >> >> > periodSeconds: 25
> >> >> >>> >> >> >> >> > successThreshold: 1
> >> >> >>> >> >> >> >> > failureThreshold: 15
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > ------------------------------
> ------------------------------------
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > Status after running gk-deploy:
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > [root at workstation deploy]# kubectl get
> >> >> >>> >> >> >> >> > nodes,pods,deployments,services
> >> >> >>> >> >> >> >> > NAME STATUS
> >> >> >>> >> >> >> >> > AGE
> >> >> >>> >> >> >> >> > VERSION
> >> >> >>> >> >> >> >> > no/node-1.c.kubernetes-174104.internal Ready
> 2d
> >> >> >>> >> >> >> >> > v1.7.2-rancher1
> >> >> >>> >> >> >> >> > no/node-2.c.kubernetes-174104.internal Ready
> 2d
> >> >> >>> >> >> >> >> > v1.7.2-rancher1
> >> >> >>> >> >> >> >> > no/node-3.c.kubernetes-174104.internal Ready
> 2d
> >> >> >>> >> >> >> >> > v1.7.2-rancher1
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > NAME READY STATUS RESTARTS
> >> >> >>> >> >> >> >> > AGE
> >> >> >>> >> >> >> >> > po/glusterfs-0s440 0/1 Running 0
> >> >> >>> >> >> >> >> > 1m
> >> >> >>> >> >> >> >> > po/glusterfs-j7dgr 0/1 Running 0
> >> >> >>> >> >> >> >> > 1m
> >> >> >>> >> >> >> >> > po/glusterfs-p6jl3 0/1 Running 0
> >> >> >>> >> >> >> >> > 1m
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > NAME CLUSTER-IP EXTERNAL-IP PORT(S)
> >> >> >>> >> >> >> >> > AGE
> >> >> >>> >> >> >> >> > svc/kubernetes 10.43.0.1 <none> 443/TCP
> >> >> >>> >> >> >> >> > 2d
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > Also, from prerequisite perspective, i was also
> >> >> >>> >> >> >> >> > seeing
> >> >> >>> >> >> >> >> > this
> >> >> >>> >> >> >> >> > mentioned:
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > The following kernel modules must be loaded:
> >> >> >>> >> >> >> >> > * dm_snapshot
> >> >> >>> >> >> >> >> > * dm_mirror
> >> >> >>> >> >> >> >> > * dm_thin_pool
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > Where exactly is this to be checked? On all Gluster
> >> >> >>> >> >> >> >> > server
> >> >> >>> >> >> >> >> > nodes?
> >> >> >>> >> >> >> >> > How
> >> >> >>> >> >> >> >> > can i
> >> >> >>> >> >> >> >> > check whether it's there?
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > I have attached topology.json and gk-deploy log for
> >> >> >>> >> >> >> >> > reference.
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > Does this issue has anything to do with the host OS
> >> >> >>> >> >> >> >> > (RancherOS)
> >> >> >>> >> >> >> >> > that
> >> >> >>> >> >> >> >> > i
> >> >> >>> >> >> >> >> > am
> >> >> >>> >> >> >> >> > using for Gluster nodes? Any idea how i can fix
> this?
> >> >> >>> >> >> >> >> > Any
> >> >> >>> >> >> >> >> > help
> >> >> >>> >> >> >> >> > will
> >> >> >>> >> >> >> >> > really
> >> >> >>> >> >> >> >> > be appreciated.
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > Thanks.
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > _______________________________________________
> >> >> >>> >> >> >> >> > heketi-devel mailing list
> >> >> >>> >> >> >> >> > heketi-devel at gluster.org
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >> > http://lists.gluster.org/
> mailman/listinfo/heketi-devel
> >> >> >>> >> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >> >
> >> >> >>> >> >> >
> >> >> >>> >> >> >
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >
> >> >> >>> >
> >> >> >>
> >> >> >>
> >> >> >
> >> >
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/heketi-devel/attachments/20170905/308d236d/attachment-0001.html>
-------------- next part --------------
[root at cent-gluster-workstation heketi]# cat topology.json
{
"clusters": [
{
"nodes": [
{
"node": {
"hostnames": {
"manage": [
"cent-gluster1"
],
"storage": [
"10.128.0.12"
]
},
"zone": 1
},
"devices": [
"/dev/sdb"
]
},
{
"node": {
"hostnames": {
"manage": [
"cent-gluster2"
],
"storage": [
"10.128.0.15"
]
},
"zone": 2
},
"devices": [
"/dev/sdb"
]
},
{
"node": {
"hostnames": {
"manage": [
"cent-gluster3"
],
"storage": [
"10.128.0.16"
]
},
"zone": 1
},
"devices": [
"/dev/sdb"
]
}
]
}
]
}
-------------------------------------------------------------------
[root at cent-gluster-workstation ~]# cat glusterfs-storageclass1.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
name: gluster-dyn
provisioner: kubernetes.io/glusterfs
parameters:
resturl: "http://cent-gluster-workstation.c.kubernetes-174104.internal:8080"
restauthenabled: "false"
-------------------------------------------------------------------
[root at cent-gluster-workstation ~]# cat glusterfs-pvc-storageclass.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gluster-dyn-pvc
annotations:
volume.beta.kubernetes.io/storage-class: gluster-dyn
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 3Gi
-------------------------------------------------------------------
[root at cent-gluster-workstation ~]# cat nginx-pod-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-deploy
spec:
replicas: 5
minReadySeconds: 10
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
template:
metadata:
labels:
app: gluster-pod1
spec:
containers:
- name: gluster-pod1
image: gcr.io/google_containers/nginx-slim:0.8
ports:
- name: web
containerPort: 80
securityContext:
privileged: true
volumeMounts:
- name: gluster-vol1
mountPath: /usr/share/nginx/html
volumes:
- name: gluster-vol1
persistentVolumeClaim:
claimName: gluster-dyn-pvc
-------------------------------------------------------------------
[root at cent-gluster-workstation ~]# cat nginx-pod-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx-pod-svc
labels:
app: gluster-pod1
spec:
type: NodePort
ports:
- port: 80
nodePort: 30001
protocol: TCP
selector:
app: gluster-pod1
-------------------------------------------------------------------
-------------- next part --------------
[root at cent-gluster-workstation heketi]# heketi-cli volume create --size=1 --replica=3
Name: vol_a63ce0cd99028689c0aad0fe580e445d
Size: 1
Volume Id: a63ce0cd99028689c0aad0fe580e445d
Cluster Id: 93a95249ac2b4bd5ea07ab84b11ef32d
Mount: 10.128.0.12:vol_a63ce0cd99028689c0aad0fe580e445d
Mount Options: backup-volfile-servers=10.128.0.15,10.128.0.16
Durability Type: replicate
Distributed+Replica: 3
Later i added more space because i thought 1 GB was too less
[root at cent-gluster-workstation ~]# heketi-cli volume expand --volume=a63ce0cd99028689c0aad0fe580e445d --expand-size=7
Name: vol_a63ce0cd99028689c0aad0fe580e445d
Size: 8
Volume Id: a63ce0cd99028689c0aad0fe580e445d
Cluster Id: 93a95249ac2b4bd5ea07ab84b11ef32d
Mount: 10.128.0.12:vol_a63ce0cd99028689c0aad0fe580e445d
Mount Options: backup-volfile-servers=10.128.0.15,10.128.0.16
Durability Type: replicate
Distributed+Replica: 3
[root at cent-gluster1 ~]# gluster volume info
Volume Name: vol_24a6778f7cd18eebe20fa9254a0c8b3a
Type: Replicate
Volume ID: 5352855f-f4d9-429b-abfc-d9b42baa3fd4
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.128.0.15:/var/lib/heketi/mounts/vg_1e22aabec961fb5a99f87e7dedd389b5/brick_ab674f22c98fb878bdc024027b93499d/brick
Brick2: 10.128.0.12:/var/lib/heketi/mounts/vg_6e7eeb800d7308b0c5c8db947a3a5e42/brick_ac36ae6542f7d5e180c9decff20b5eb5/brick
Brick3: 10.128.0.16:/var/lib/heketi/mounts/vg_de01ec4686701da668624b62901754c2/brick_5ad0ecf5ff71c42e08d992164d521bc0/brick
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
Volume Name: vol_a63ce0cd99028689c0aad0fe580e445d
Type: Distributed-Replicate
Volume ID: 6146b7f6-0aea-4dbd-8873-57b8a49c3692
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: 10.128.0.15:/var/lib/heketi/mounts/vg_1e22aabec961fb5a99f87e7dedd389b5/brick_38e13e05ef6b29a86a3b6d8e764b04c0/brick
Brick2: 10.128.0.16:/var/lib/heketi/mounts/vg_de01ec4686701da668624b62901754c2/brick_cff1a4aa7235fe98bddcc55b3dd41693/brick
Brick3: 10.128.0.12:/var/lib/heketi/mounts/vg_6e7eeb800d7308b0c5c8db947a3a5e42/brick_08cc9e4518af91e3dff5613a1183c83a/brick
Brick4: 10.128.0.15:/var/lib/heketi/mounts/vg_1e22aabec961fb5a99f87e7dedd389b5/brick_4f0623431dd00ec8281ae8bd726a4e55/brick
Brick5: 10.128.0.12:/var/lib/heketi/mounts/vg_6e7eeb800d7308b0c5c8db947a3a5e42/brick_df6d52f488f8cc948e3e5cde1c9ba1a5/brick
Brick6: 10.128.0.16:/var/lib/heketi/mounts/vg_de01ec4686701da668624b62901754c2/brick_4cabbfa235acb9d2b18fcafda58c495c/brick
Options Reconfigured:
nfs.disable: on
transport.address-family: inet
On Gluster node 3:
[root at cent-gluster3 ~]# find / -type f -mmin -180 -name index.html -exec ls -l {} \;
-rw-r--r--. 2 root 2000 0 Sep 5 10:31 /var/lib/heketi/mounts/vg_de01ec4686701da668624b62901754c2/brick_5ad0ecf5ff71c42e08d992164d521bc0/brick/index.html
[root at cent-gluster3 ~]# find / -type f -mmin -180 -name index.html -exec ls -l {} \;
-rw-r--r--. 2 root 2000 21 Sep 5 11:00 /var/lib/heketi/mounts/vg_de01ec4686701da668624b62901754c2/brick_5ad0ecf5ff71c42e08d992164d521bc0/brick/index.html
[root at cent-gluster3 ~]# cat /var/lib/heketi/mounts/vg_de01ec4686701da668624b62901754c2/brick_5ad0ecf5ff71c42e08d992164d521bc0/brick/index.html
Hello from GlusterFS
On Gluster node 2:
[root at cent-gluster2 ~]# find / -type f -mmin -180 -name index.html -exec ls -l {} \;
-rw-r--r--. 2 root 2000 21 Sep 5 11:00 /var/lib/heketi/mounts/vg_1e22aabec961fb5a99f87e7dedd389b5/brick_ab674f22c98fb878bdc024027b93499d/brick/index.html
On Gluster node 1:
[root at cent-gluster1 vols]# find / -type f -mmin -180 -name index.html -exec ls -l {} \;
-rw-r--r--. 2 root 2000 21 Sep 5 11:00 /var/lib/heketi/mounts/vg_6e7eeb800d7308b0c5c8db947a3a5e42/brick_ac36ae6542f7d5e180c9decff20b5eb5/brick/index.html
More information about the heketi-devel
mailing list