[heketi-devel] Unable to use Heketi setup to install Gluster for Kubernetes

Jose A. Rivera jarrpa at redhat.com
Tue Sep 5 16:04:29 UTC 2017


No problem! Glad we go you going. :)

On Tue, Sep 5, 2017 at 11:00 AM, Gaurav Chhabra
<varuag.chhabra at gmail.com> wrote:
> Oh...ok. I misinterpreted while thinking of HA. So it's persistent and HA in
> the sense that even if one node in the Gluster cluster goes down, others are
> available to serve request because of replication. I hope i now got it
> correct.
>
> I'm glad i posted my query here and i'm really thankful for your help. I was
> struggling to get this thing work for the last 6-7 days.
>
>
> Thanks again Jose! :)
> Gaurav
>
>
>
> On Tue, Sep 5, 2017 at 7:00 PM, Jose A. Rivera <jarrpa at redhat.com> wrote:
>>
>> That is expected behavior. :) GlusterFS volumes are shared storage,
>> meaning the same data is presented to all accessing users. By default
>> (in our configuration) it replicates data across multiple bricks, but
>> all bricks are represented as a single filesystem and all replicas of
>> a file are presented as a single file. Thus, when you delete a file
>> you delete it and all its replicas.
>>
>> On Tue, Sep 5, 2017 at 6:15 AM, Gaurav Chhabra <varuag.chhabra at gmail.com>
>> wrote:
>> > Hi Jose,
>> >
>> >
>> > There is some progress after using the  following container:
>> > https://hub.docker.com/r/gluster/glusterfs-client/
>> >
>> > I created deployment with 5 replicas for Nginx in Kubernetes using your
>> > doc.
>> > Now as described in the last section of your doc, if i create file
>> > 'index.html' inside of a container running in a pod on node-a, i can see
>> > the
>> > same file becomes available on another pod running on node-b. However,
>> > if i
>> > delete the file on one node (say node-a), it becomes unavailable on all
>> > the
>> > three nodes. The file should have been accessible from other nodes even
>> > if i
>> > delete it from one pod on a given node. Correct me if i'm getting it
>> > wrong.
>> >
>> > I searched for index.html on the Gluster cluster and i could find it on
>> > all
>> > three nodes with same content. Please check the attached log.
>> >
>> > If i check /etc/fstab on any of the pods, i see the following:
>> >
>> > root at nginx-deploy-4253768608-w645w:/# cat /etc/fstab
>> > # UNCONFIGURED FSTAB FOR BASE SYSTEM
>> >
>> > Attached log files.
>> >
>> > Any idea why the file is not accessible from another pod on node-a or
>> > node-b
>> > if i delete it from a different pod present on another node-c?
>> >
>> >
>> > Regards,
>> > Gaurav
>> >
>> >
>> > On Tue, Sep 5, 2017 at 3:41 AM, Jose A. Rivera <jarrpa at redhat.com>
>> > wrote:
>> >>
>> >> Yup, I follow that repo and should be contributing to it in the
>> >> near-future. :) I've never played with the glusterfs-client image, but
>> >> here's hoping!
>> >>
>> >> On Mon, Sep 4, 2017 at 10:18 AM, Gaurav Chhabra
>> >> <varuag.chhabra at gmail.com> wrote:
>> >> > Actually the script is launching a container and i'm pretty sure that
>> >> > the
>> >> > command that's programmed as part of the container launch process is
>> >> > actually trying to run that mount command. Now the issue is that this
>> >> > again
>> >> > falls under Docker area. :( Generally, Docker container providers (in
>> >> > this
>> >> > case, nixel) provide a Dockerfile which specifies the command that's
>> >> > executed as part of the container launch process however, the problem
>> >> > in
>> >> > case of this container is that it's not maintained. It's 2 years old.
>> >> > If
>> >> > you
>> >> > will check active projects such as gluster/gluster-containers, you
>> >> > will
>> >> > see
>> >> > a proper Dockerfile explaining what it does.
>> >> >
>> >> > https://hub.docker.com/r/gluster/glusterfs-client/~/dockerfile/
>> >> >
>> >> > I am now planning to use the above container and see whether i'm
>> >> > lucky.
>> >> > If
>> >> > you will see the above link, it's running
>> >> >
>> >> > dnf install -y glusterfs-fuse
>> >> >
>> >> > Hopefully, this should take care of this section which asks to make
>> >> > sure
>> >> > that glusterfs-client is installed
>> >> >
>> >> >
>> >> >
>> >> > https://github.com/gluster/gluster-kubernetes/tree/master/docs/examples/dynamic_provisioning_external_gluster#validate-communication-and-gluster-prerequisites-on-the-kubernetes-nodes
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Mon, Sep 4, 2017 at 7:33 PM, Jose A. Rivera <jarrpa at redhat.com>
>> >> > wrote:
>> >> >>
>> >> >> Something is trying to mount a GlusterFS volume, and it's not that
>> >> >> script. What's trying to mount?
>> >> >>
>> >> >> On Mon, Sep 4, 2017 at 8:52 AM, Gaurav Chhabra
>> >> >> <varuag.chhabra at gmail.com>
>> >> >> wrote:
>> >> >> > I am just running this bash script which will start the
>> >> >> > rancher-glusterfs-client container:
>> >> >> >
>> >> >> > [root at node-a ~]# cat rancher-glusterfs-client.sh
>> >> >> > #!/bin/bash
>> >> >> > sudo docker run --privileged \
>> >> >> >         --name=gluster-client \
>> >> >> >         -d \
>> >> >> >         -v /sys/fs/cgroup:/sys/fs/cgroup \
>> >> >> >         -v /var/log/glusterfs:/var/log/glusterfs \
>> >> >> >         --env GLUSTER_PEER=10.128.0.12,10.128.0.15,10.128.0.16 \
>> >> >> >         nixel/rancher-glusterfs-client
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Mon, Sep 4, 2017 at 6:26 PM, Jose A. Rivera <jarrpa at redhat.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> What is the exact command you're running?
>> >> >> >>
>> >> >> >> On Mon, Sep 4, 2017 at 4:26 AM, Gaurav Chhabra
>> >> >> >> <varuag.chhabra at gmail.com>
>> >> >> >> wrote:
>> >> >> >> > Hi Jose,
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > From this link, it seems mount.glusterfs might actually be
>> >> >> >> > present
>> >> >> >> > in
>> >> >> >> > the
>> >> >> >> > container that launched and quickly terminated.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > https://unix.stackexchange.com/questions/312178/glusterfs-replicated-volume-mounting-issue
>> >> >> >> >
>> >> >> >> > If you check the question that the user posted, the same error
>> >> >> >> > (Mount
>> >> >> >> > failed) was reported that i sent you in the last email.
>> >> >> >> >
>> >> >> >> > After seeing the above, i checked /var/log/glusterfs on my host
>> >> >> >> > (RancherOS)
>> >> >> >> > but it was empty. I ran the container again but with explicit
>> >> >> >> > volume
>> >> >> >> > mount
>> >> >> >> > as shown below:
>> >> >> >> >
>> >> >> >> > [root at node-a ~]# cat rancher-glusterfs-client.sh
>> >> >> >> > #!/bin/bash
>> >> >> >> > sudo docker run --privileged \
>> >> >> >> >         --name=gluster-client \
>> >> >> >> >         -d \
>> >> >> >> >         -v /sys/fs/cgroup:/sys/fs/cgroup \
>> >> >> >> >         -v /var/log/glusterfs:/var/log/glusterfs \
>> >> >> >> >         --env GLUSTER_PEER=10.128.0.12,10.128.0.15,10.128.0.16
>> >> >> >> > \
>> >> >> >> >         nixel/rancher-glusterfs-client
>> >> >> >> > This time, i could see a log file
>> >> >> >> > (/var/log/glusterfs/mnt-ranchervol.log)
>> >> >> >> > present. I have attached the content of the same. Also attached
>> >> >> >> > are
>> >> >> >> > logs
>> >> >> >> > from Heketi client/server (both on one node) and Gluster
>> >> >> >> > cluster.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > Regards,
>> >> >> >> > Gaurav
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On Mon, Sep 4, 2017 at 2:29 PM, Gaurav Chhabra
>> >> >> >> > <varuag.chhabra at gmail.com>
>> >> >> >> > wrote:
>> >> >> >> >>
>> >> >> >> >> Hi Jose,
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> I tried setting up things using the link you provided and i
>> >> >> >> >> was
>> >> >> >> >> able
>> >> >> >> >> to
>> >> >> >> >> get all steps working for 3 node Gluster cluster, all running
>> >> >> >> >> on
>> >> >> >> >> CentOS7,
>> >> >> >> >> without any issue. However, as expected, when i tried
>> >> >> >> >> configuring
>> >> >> >> >> Kubernetes
>> >> >> >> >> by installing nixel/rancher-glusterfs-client container, i got
>> >> >> >> >> error:
>> >> >> >> >>
>> >> >> >> >> [root at node-a ~]# cat rancher-glusterfs-client.sh
>> >> >> >> >> #!/bin/bash
>> >> >> >> >> sudo docker run --privileged \
>> >> >> >> >>         --name=gluster-client \
>> >> >> >> >>         -d \
>> >> >> >> >>         -v /sys/fs/cgroup:/sys/fs/cgroup \
>> >> >> >> >>         --env GLUSTER_PEER=10.128.0.12,10.128.0.15,10.128.0.16
>> >> >> >> >> \
>> >> >> >> >>         nixel/rancher-glusterfs-client
>> >> >> >> >>
>> >> >> >> >> [root at node-a ~]# ./rancher-glusterfs-client.sh
>> >> >> >> >>
>> >> >> >> >> ac069caccdce147d6f423fc566166345191dbc1b11f3416c66207a1fd11fda6b
>> >> >> >> >>
>> >> >> >> >> [root at node-a ~]# docker logs gluster-client
>> >> >> >> >> => Checking if I can reach GlusterFS node 10.128.0.12 ...
>> >> >> >> >> => GlusterFS node 10.128.0.12 is alive
>> >> >> >> >> => Mounting GlusterFS volume ranchervol from GlusterFS node
>> >> >> >> >> 10.128.0.12
>> >> >> >> >> ...
>> >> >> >> >> Mount failed. Please check the log file for more details.
>> >> >> >> >>
>> >> >> >> >> If i try running the next step as described in your link, i
>> >> >> >> >> get
>> >> >> >> >> the
>> >> >> >> >> following:
>> >> >> >> >>
>> >> >> >> >> [root at node-a ~]# modprobe fuse
>> >> >> >> >> modprobe: module fuse not found in modules.dep
>> >> >> >> >>
>> >> >> >> >> Since the container failed to start, i could only check on the
>> >> >> >> >> host
>> >> >> >> >> (RancherOS) and i could only find two mount-related commands:
>> >> >> >> >> mount
>> >> >> >> >> &
>> >> >> >> >> mountpoint
>> >> >> >> >>
>> >> >> >> >> Any pointers?
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> Regards,
>> >> >> >> >> Gaurav
>> >> >> >> >>
>> >> >> >> >> On Sun, Sep 3, 2017 at 11:18 PM, Jose A. Rivera
>> >> >> >> >> <jarrpa at redhat.com>
>> >> >> >> >> wrote:
>> >> >> >> >>>
>> >> >> >> >>> Installing the glusterfs-client container should be fine. :)
>> >> >> >> >>> The
>> >> >> >> >>> main
>> >> >> >> >>> thing that's needed is that all your Kubernetes nodes need to
>> >> >> >> >>> have
>> >> >> >> >>> the
>> >> >> >> >>> "mount.glusterfs" command available so Kube can mount the
>> >> >> >> >>> GlusterFS
>> >> >> >> >>> volumes and present them to the pods.
>> >> >> >> >>>
>> >> >> >> >>> On Sun, Sep 3, 2017 at 12:14 PM, Gaurav Chhabra
>> >> >> >> >>> <varuag.chhabra at gmail.com> wrote:
>> >> >> >> >>> > Thanks Jose. The link you've suggested looks good but again
>> >> >> >> >>> > it
>> >> >> >> >>> > expects
>> >> >> >> >>> > me to
>> >> >> >> >>> > install gluster-client on Kubernetes node and i fall into
>> >> >> >> >>> > the
>> >> >> >> >>> > same
>> >> >> >> >>> > issue of
>> >> >> >> >>> > installing a container for glusterfs. Only difference is
>> >> >> >> >>> > that
>> >> >> >> >>> > this
>> >> >> >> >>> > time
>> >> >> >> >>> > it's
>> >> >> >> >>> > glusterfs-client and not glusterfs-server. :)
>> >> >> >> >>> >
>> >> >> >> >>> > I will try this out and let you know tomorrow.
>> >> >> >> >>> >
>> >> >> >> >>> >
>> >> >> >> >>> > Regards,
>> >> >> >> >>> > Gaurav
>> >> >> >> >>> >
>> >> >> >> >>> >
>> >> >> >> >>> > On Sun, Sep 3, 2017 at 2:11 AM, Jose A. Rivera
>> >> >> >> >>> > <jarrpa at redhat.com>
>> >> >> >> >>> > wrote:
>> >> >> >> >>> >>
>> >> >> >> >>> >> Hey, no problem! I'm eager to learn more about different
>> >> >> >> >>> >> flavors
>> >> >> >> >>> >> of
>> >> >> >> >>> >> Linux, I just apologize for my relative inexperience with
>> >> >> >> >>> >> them.
>> >> >> >> >>> >> :)
>> >> >> >> >>> >>
>> >> >> >> >>> >> To that end, I will also admit I'm not very experienced
>> >> >> >> >>> >> with
>> >> >> >> >>> >> direct
>> >> >> >> >>> >> Docker myself. I understand the basic workflow and know
>> >> >> >> >>> >> some
>> >> >> >> >>> >> of
>> >> >> >> >>> >> the
>> >> >> >> >>> >> run options, but not having deep experience keeps me from
>> >> >> >> >>> >> having
>> >> >> >> >>> >> a
>> >> >> >> >>> >> better understanding of the patterns and consequences.
>> >> >> >> >>> >>
>> >> >> >> >>> >> Thus, I'd like to guide you in a direction I'd more apt to
>> >> >> >> >>> >> help
>> >> >> >> >>> >> you
>> >> >> >> >>> >> in
>> >> >> >> >>> >> right now. I know that you can't have multiple GlusterFS
>> >> >> >> >>> >> servers
>> >> >> >> >>> >> running on the same nodes, and I know that we have been
>> >> >> >> >>> >> successfully
>> >> >> >> >>> >> running several configurations using our
>> >> >> >> >>> >> gluster/gluster-centos
>> >> >> >> >>> >> image.
>> >> >> >> >>> >> If you follow the Kubernetes configuration on
>> >> >> >> >>> >> gluster-kubernetes,
>> >> >> >> >>> >> the
>> >> >> >> >>> >> pod/container is run privileged and with host networking,
>> >> >> >> >>> >> and
>> >> >> >> >>> >> we
>> >> >> >> >>> >> require that the node has all listed ports open, not just
>> >> >> >> >>> >> 2222.
>> >> >> >> >>> >> The
>> >> >> >> >>> >> sshd running in the container is listening on 2222, not
>> >> >> >> >>> >> 22,
>> >> >> >> >>> >> but
>> >> >> >> >>> >> it
>> >> >> >> >>> >> is
>> >> >> >> >>> >> also not really required if you're not doing
>> >> >> >> >>> >> geo-replication.
>> >> >> >> >>> >>
>> >> >> >> >>> >> Alternatively, you can indeed run GlusterFS outside of
>> >> >> >> >>> >> Kubernetes
>> >> >> >> >>> >> but
>> >> >> >> >>> >> still have Kubernetes apps access GlusterFS storage. The
>> >> >> >> >>> >> nodes
>> >> >> >> >>> >> can
>> >> >> >> >>> >> be
>> >> >> >> >>> >> anything you want, they just need to be running GlusterFS
>> >> >> >> >>> >> and
>> >> >> >> >>> >> you
>> >> >> >> >>> >> need
>> >> >> >> >>> >> a heketi service managing them. Here is an example of how
>> >> >> >> >>> >> to
>> >> >> >> >>> >> set
>> >> >> >> >>> >> this
>> >> >> >> >>> >> up using CentOS:
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >> >> >> >>> >> https://github.com/gluster/gluster-kubernetes/tree/master/docs/examples/dynamic_provisioning_external_gluster
>> >> >> >> >>> >>
>> >> >> >> >>> >> Hope this is at least leading you in a useful direction.
>> >> >> >> >>> >> :)
>> >> >> >> >>> >>
>> >> >> >> >>> >> --Jose
>> >> >> >> >>> >>
>> >> >> >> >>> >> On Sat, Sep 2, 2017 at 3:16 PM, Gaurav Chhabra
>> >> >> >> >>> >> <varuag.chhabra at gmail.com>
>> >> >> >> >>> >> wrote:
>> >> >> >> >>> >> > Hi Jose,
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > webcenter/rancher-glusterfs-server is actually a
>> >> >> >> >>> >> > container
>> >> >> >> >>> >> > provided
>> >> >> >> >>> >> > by
>> >> >> >> >>> >> > Sebastien, its maintainer. It's a Docker container which
>> >> >> >> >>> >> > has
>> >> >> >> >>> >> > GlusterFS
>> >> >> >> >>> >> > server running within it. On the host i.e., RancherOS,
>> >> >> >> >>> >> > there
>> >> >> >> >>> >> > is
>> >> >> >> >>> >> > no
>> >> >> >> >>> >> > separate
>> >> >> >> >>> >> > GlusterFS server running because we cannot install
>> >> >> >> >>> >> > anything
>> >> >> >> >>> >> > that
>> >> >> >> >>> >> > way.
>> >> >> >> >>> >> > Running using container is the only way so i started
>> >> >> >> >>> >> > ancher-glusterfs-server
>> >> >> >> >>> >> > container with the following parameters:
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > [root at node-1 rancher]# cat gluster-server.sh
>> >> >> >> >>> >> > #!/bin/bash
>> >> >> >> >>> >> > sudo docker run --name=gluster-server -d \
>> >> >> >> >>> >> >         --env 'SERVICE_NAME=gluster' \
>> >> >> >> >>> >> >         --restart always \
>> >> >> >> >>> >> >         --publish 2222:22 \
>> >> >> >> >>> >> >         webcenter/rancher-glusterfs-server
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > Here's the link to the dockerfile:
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > https://hub.docker.com/r/webcenter/rancher-glusterfs-server/~/dockerfile/
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > It's similar to other GlusteFS containers provided by
>> >> >> >> >>> >> > other
>> >> >> >> >>> >> > maintainers
>> >> >> >> >>> >> > for
>> >> >> >> >>> >> > different OS. For example, for CentOS, we have
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > https://hub.docker.com/r/gluster/gluster-centos/~/dockerfile/
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > From what i understand, Heketi does support container
>> >> >> >> >>> >> > based
>> >> >> >> >>> >> > GlusterFS
>> >> >> >> >>> >> > server
>> >> >> >> >>> >> > as mentioned in the prerequisite where it says:
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > "Each node must have the following ports opened for
>> >> >> >> >>> >> > GlusterFS
>> >> >> >> >>> >> > communications:
>> >> >> >> >>> >> >  2222 - GlusterFS pod's sshd"
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > That's the reason i've exposed port 2222 for 22 as shown
>> >> >> >> >>> >> > above.
>> >> >> >> >>> >> > Please
>> >> >> >> >>> >> > correct me if i misunderstood it.
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > As soon as i run the above script (gluster-server.sh),
>> >> >> >> >>> >> > it
>> >> >> >> >>> >> > automatically
>> >> >> >> >>> >> > creates the following directories on host. This should
>> >> >> >> >>> >> > have
>> >> >> >> >>> >> > ideally
>> >> >> >> >>> >> > not
>> >> >> >> >>> >> > been
>> >> >> >> >>> >> > empty as you mentioned.
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > /etc/glusterfs    /var/lib/glusterd
>> >> >> >> >>> >> > /var/log/glusterfs
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > Just wanted to know in which circumstances do we get
>> >> >> >> >>> >> > this
>> >> >> >> >>> >> > specific
>> >> >> >> >>> >> > error
>> >> >> >> >>> >> > (Failed to get D-Bus connection: Operation not
>> >> >> >> >>> >> > permitted)
>> >> >> >> >>> >> > related
>> >> >> >> >>> >> > to
>> >> >> >> >>> >> > Readiness probe failing. Searching online took me to
>> >> >> >> >>> >> > discussions
>> >> >> >> >>> >> > around
>> >> >> >> >>> >> > running container in privileged mode and some directory
>> >> >> >> >>> >> > to
>> >> >> >> >>> >> > be
>> >> >> >> >>> >> > mounted.
>> >> >> >> >>> >> > Based
>> >> >> >> >>> >> > on that, i also modified my container startup script to
>> >> >> >> >>> >> > the
>> >> >> >> >>> >> > following:
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > #!/bin/bash
>> >> >> >> >>> >> > sudo docker run --privileged \
>> >> >> >> >>> >> >         --name=gluster-server \
>> >> >> >> >>> >> >         -d \
>> >> >> >> >>> >> >         -v /sys/fs/cgroup:/sys/fs/cgroup \
>> >> >> >> >>> >> >         -v /etc/glusterfs:/etc/glusterfs \
>> >> >> >> >>> >> >         -v /var/lib/glusterd:/var/lib/glusterd \
>> >> >> >> >>> >> >         -v /var/log/glusterfs:/var/log/glusterfs \
>> >> >> >> >>> >> >         --env 'SERVICE_NAME=gluster' \
>> >> >> >> >>> >> >         --restart always \
>> >> >> >> >>> >> >         --publish 2222:22 \
>> >> >> >> >>> >> >         webcenter/rancher-glusterfs-server
>> >> >> >> >>> >> > Still, the issue persists.
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > I also logged into the container and checked whether
>> >> >> >> >>> >> > systemctl
>> >> >> >> >>> >> > command
>> >> >> >> >>> >> > is
>> >> >> >> >>> >> > present. It was there but manualy running the command
>> >> >> >> >>> >> > also
>> >> >> >> >>> >> > doesn't
>> >> >> >> >>> >> > work:
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > [root at node-c ~]# docker exec -it gluster-server
>> >> >> >> >>> >> > /bin/bash
>> >> >> >> >>> >> > root at 42150f203f80:/app# systemctl status
>> >> >> >> >>> >> > glusterd.service
>> >> >> >> >>> >> > WARNING: terminal is not fully functional
>> >> >> >> >>> >> > Failed to connect to bus: No such file or directory
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > Under section 'ADVANCED OPTIONS - Security/Host' in this
>> >> >> >> >>> >> > link,
>> >> >> >> >>> >> > it
>> >> >> >> >>> >> > talks
>> >> >> >> >>> >> > about SYS_ADMIN setting. Any idea how i can try this?
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > Also, there was this mentioned in the Heketi setup page:
>> >> >> >> >>> >> > "If you are not able to deploy a hyper-converged
>> >> >> >> >>> >> > GlusterFS
>> >> >> >> >>> >> > cluster,
>> >> >> >> >>> >> > you
>> >> >> >> >>> >> > must
>> >> >> >> >>> >> > have one running somewhere that the Kubernetes nodes can
>> >> >> >> >>> >> > access"
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >>>> Does it mean running the three node Gluster cluster
>> >> >> >> >>> >> >>>> outside
>> >> >> >> >>> >> >>>> Kubernetes,
>> >> >> >> >>> >> >>>> may be on some VM running on RHEL/CentOS etc? If yes,
>> >> >> >> >>> >> >>>> then
>> >> >> >> >>> >> >>>> how
>> >> >> >> >>> >> >>>> will i
>> >> >> >> >>> >> >>>> be
>> >> >> >> >>> >> >>>> able to tell Gluster which volume from the Kubernetes
>> >> >> >> >>> >> >>>> cluster
>> >> >> >> >>> >> >>>> pod
>> >> >> >> >>> >> >>>> to
>> >> >> >> >>> >> >>>> sync?
>> >> >> >> >>> >> >>>> Any references?
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > I really appreciate your responses despite the fact that
>> >> >> >> >>> >> > you've
>> >> >> >> >>> >> > not
>> >> >> >> >>> >> > used
>> >> >> >> >>> >> > RancherOS but still trying to help.
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > Thanks,
>> >> >> >> >>> >> > Gaurav
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >
>> >> >> >> >>> >> > On Sat, Sep 2, 2017 at 7:35 PM, Jose A. Rivera
>> >> >> >> >>> >> > <jarrpa at redhat.com>
>> >> >> >> >>> >> > wrote:
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >> I'm afraid I have no experience with RancherOS, so I
>> >> >> >> >>> >> >> may
>> >> >> >> >>> >> >> be
>> >> >> >> >>> >> >> missing
>> >> >> >> >>> >> >> some things about how it works. My primary experience
>> >> >> >> >>> >> >> is
>> >> >> >> >>> >> >> with
>> >> >> >> >>> >> >> Fedora,
>> >> >> >> >>> >> >> CentOS, and Ubuntu.
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >> What is webcenter/rancher-glusterfs-server? If it's
>> >> >> >> >>> >> >> running
>> >> >> >> >>> >> >> another
>> >> >> >> >>> >> >> glusterd then you probably don't want to be running it
>> >> >> >> >>> >> >> and
>> >> >> >> >>> >> >> should
>> >> >> >> >>> >> >> remove it from your systems.
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >> The glusterfs pods mount hostpath volumes from the host
>> >> >> >> >>> >> >> they're
>> >> >> >> >>> >> >> running on to persist their configuration. Thus
>> >> >> >> >>> >> >> anything
>> >> >> >> >>> >> >> they
>> >> >> >> >>> >> >> write
>> >> >> >> >>> >> >> to
>> >> >> >> >>> >> >> those directories should land on the host. If that's
>> >> >> >> >>> >> >> not
>> >> >> >> >>> >> >> happening
>> >> >> >> >>> >> >> then that's an additional problem.
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >> --Jose
>> >> >> >> >>> >> >>
>> >> >> >> >>> >> >> On Fri, Sep 1, 2017 at 11:17 PM, Gaurav Chhabra
>> >> >> >> >>> >> >> <varuag.chhabra at gmail.com> wrote:
>> >> >> >> >>> >> >> > Hi Jose,
>> >> >> >> >>> >> >> >
>> >> >> >> >>> >> >> >
>> >> >> >> >>> >> >> > I tried your suggestion but there is one confusion
>> >> >> >> >>> >> >> > regarding
>> >> >> >> >>> >> >> > point
>> >> >> >> >>> >> >> > #3.
>> >> >> >> >>> >> >> > Since
>> >> >> >> >>> >> >> > RancherOS has everything running as container, i am
>> >> >> >> >>> >> >> > running
>> >> >> >> >>> >> >> > webcenter/rancher-glusterfs-server container on all
>> >> >> >> >>> >> >> > three
>> >> >> >> >>> >> >> > nodes.
>> >> >> >> >>> >> >> > Now
>> >> >> >> >>> >> >> > as
>> >> >> >> >>> >> >> > far
>> >> >> >> >>> >> >> > as removing the directories are concerned, i hope you
>> >> >> >> >>> >> >> > meant
>> >> >> >> >>> >> >> > removing
>> >> >> >> >>> >> >> > them on
>> >> >> >> >>> >> >> > the host and _not_ from within the container. After
>> >> >> >> >>> >> >> > completing
>> >> >> >> >>> >> >> > step 1
>> >> >> >> >>> >> >> > and 2,
>> >> >> >> >>> >> >> > i checked the contents of all the directories that
>> >> >> >> >>> >> >> > you
>> >> >> >> >>> >> >> > specified
>> >> >> >> >>> >> >> > in
>> >> >> >> >>> >> >> > point
>> >> >> >> >>> >> >> > #3. All were empty as you can see in the attached
>> >> >> >> >>> >> >> > other_logs.txt.
>> >> >> >> >>> >> >> > So
>> >> >> >> >>> >> >> > i
>> >> >> >> >>> >> >> > got
>> >> >> >> >>> >> >> > confused. I ran the deploy again but the issue
>> >> >> >> >>> >> >> > persists.
>> >> >> >> >>> >> >> > Two
>> >> >> >> >>> >> >> > pods
>> >> >> >> >>> >> >> > show
>> >> >> >> >>> >> >> > Liveness error and the third one, Readiness error.
>> >> >> >> >>> >> >> >
>> >> >> >> >>> >> >> > I then tried removing those directories (Step #3)
>> >> >> >> >>> >> >> > from
>> >> >> >> >>> >> >> > within
>> >> >> >> >>> >> >> > the
>> >> >> >> >>> >> >> > container
>> >> >> >> >>> >> >> > but getting following error:
>> >> >> >> >>> >> >> >
>> >> >> >> >>> >> >> > root at c0f8ab4d92a2:/app# rm -rf /var/lib/heketi
>> >> >> >> >>> >> >> > /etc/glusterfs
>> >> >> >> >>> >> >> > /var/lib/glusterd /var/log/glusterfs
>> >> >> >> >>> >> >> > rm: cannot remove '/var/lib/glusterd': Device or
>> >> >> >> >>> >> >> > resource
>> >> >> >> >>> >> >> > busy
>> >> >> >> >>> >> >> >
>> >> >> >> >>> >> >> >
>> >> >> >> >>> >> >> >
>> >> >> >> >>> >> >> > On Fri, Sep 1, 2017 at 8:21 PM, Jose A. Rivera
>> >> >> >> >>> >> >> > <jarrpa at redhat.com>
>> >> >> >> >>> >> >> > wrote:
>> >> >> >> >>> >> >> >>
>> >> >> >> >>> >> >> >> 1. Add a line to the ssh-exec portion of heketi.json
>> >> >> >> >>> >> >> >> of
>> >> >> >> >>> >> >> >> the
>> >> >> >> >>> >> >> >> sort:
>> >> >> >> >>> >> >> >>
>> >> >> >> >>> >> >> >> "sudo": true,
>> >> >> >> >>> >> >> >>
>> >> >> >> >>> >> >> >> 2. Run
>> >> >> >> >>> >> >> >>
>> >> >> >> >>> >> >> >> gk-deploy -g --abort
>> >> >> >> >>> >> >> >>
>> >> >> >> >>> >> >> >> 3. On the nodes that were/will be running GlusterFS
>> >> >> >> >>> >> >> >> pods,
>> >> >> >> >>> >> >> >> run:
>> >> >> >> >>> >> >> >>
>> >> >> >> >>> >> >> >> rm -rf /var/lib/heketi /etc/glusterfs
>> >> >> >> >>> >> >> >> /var/lib/glusterd
>> >> >> >> >>> >> >> >> /var/log/glusterfs
>> >> >> >> >>> >> >> >>
>> >> >> >> >>> >> >> >> Then try the deploy again.
>> >> >> >> >>> >> >> >>
>> >> >> >> >>> >> >> >> On Fri, Sep 1, 2017 at 6:05 AM, Gaurav Chhabra
>> >> >> >> >>> >> >> >> <varuag.chhabra at gmail.com>
>> >> >> >> >>> >> >> >> wrote:
>> >> >> >> >>> >> >> >> > Hi Jose,
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > Thanks for the reply. It seems the three gluster
>> >> >> >> >>> >> >> >> > pods
>> >> >> >> >>> >> >> >> > might
>> >> >> >> >>> >> >> >> > have
>> >> >> >> >>> >> >> >> > been
>> >> >> >> >>> >> >> >> > a
>> >> >> >> >>> >> >> >> > copy-paste from another set of cluster where i was
>> >> >> >> >>> >> >> >> > trying
>> >> >> >> >>> >> >> >> > to
>> >> >> >> >>> >> >> >> > setup
>> >> >> >> >>> >> >> >> > the
>> >> >> >> >>> >> >> >> > same
>> >> >> >> >>> >> >> >> > thing using CentOS. Sorry for that. By the way, i
>> >> >> >> >>> >> >> >> > did
>> >> >> >> >>> >> >> >> > check
>> >> >> >> >>> >> >> >> > for
>> >> >> >> >>> >> >> >> > the
>> >> >> >> >>> >> >> >> > kernel
>> >> >> >> >>> >> >> >> > modules and it seems it's already there. Also, i
>> >> >> >> >>> >> >> >> > am
>> >> >> >> >>> >> >> >> > attaching
>> >> >> >> >>> >> >> >> > fresh
>> >> >> >> >>> >> >> >> > set
>> >> >> >> >>> >> >> >> > of
>> >> >> >> >>> >> >> >> > files because i created a new cluster and thought
>> >> >> >> >>> >> >> >> > of
>> >> >> >> >>> >> >> >> > giving
>> >> >> >> >>> >> >> >> > it
>> >> >> >> >>> >> >> >> > a
>> >> >> >> >>> >> >> >> > try
>> >> >> >> >>> >> >> >> > again.
>> >> >> >> >>> >> >> >> > Issue still persists. :(
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > In heketi.json, there is a slight change w.r.t the
>> >> >> >> >>> >> >> >> > user
>> >> >> >> >>> >> >> >> > which
>> >> >> >> >>> >> >> >> > connects
>> >> >> >> >>> >> >> >> > to
>> >> >> >> >>> >> >> >> > glusterfs node using SSH. I am not sure how Heketi
>> >> >> >> >>> >> >> >> > was
>> >> >> >> >>> >> >> >> > using
>> >> >> >> >>> >> >> >> > root
>> >> >> >> >>> >> >> >> > user
>> >> >> >> >>> >> >> >> > to
>> >> >> >> >>> >> >> >> > login because i wasn't able to use root and do
>> >> >> >> >>> >> >> >> > manual
>> >> >> >> >>> >> >> >> > SSH.
>> >> >> >> >>> >> >> >> > With
>> >> >> >> >>> >> >> >> > rancher
>> >> >> >> >>> >> >> >> > user, i can login successfully so i think this
>> >> >> >> >>> >> >> >> > should
>> >> >> >> >>> >> >> >> > be
>> >> >> >> >>> >> >> >> > fine.
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > /etc/heketi/heketi.json:
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >> >>> >> >> >> >     "executor": "ssh",
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >     "_sshexec_comment": "SSH username and private
>> >> >> >> >>> >> >> >> > key
>> >> >> >> >>> >> >> >> > file
>> >> >> >> >>> >> >> >> > information",
>> >> >> >> >>> >> >> >> >     "sshexec": {
>> >> >> >> >>> >> >> >> >       "keyfile": "/var/lib/heketi/.ssh/id_rsa",
>> >> >> >> >>> >> >> >> >       "user": "rancher",
>> >> >> >> >>> >> >> >> >       "port": "22",
>> >> >> >> >>> >> >> >> >       "fstab": "/etc/fstab"
>> >> >> >> >>> >> >> >> >     },
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > Before running gk-deploy:
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >> >>> >> >> >> > [root at workstation deploy]# kubectl get
>> >> >> >> >>> >> >> >> > nodes,pods,daemonset,deployments,services
>> >> >> >> >>> >> >> >> > NAME                                     STATUS
>> >> >> >> >>> >> >> >> > AGE
>> >> >> >> >>> >> >> >> > VERSION
>> >> >> >> >>> >> >> >> > no/node-a.c.kubernetes-174104.internal   Ready
>> >> >> >> >>> >> >> >> > 3h
>> >> >> >> >>> >> >> >> > v1.7.2-rancher1
>> >> >> >> >>> >> >> >> > no/node-b.c.kubernetes-174104.internal   Ready
>> >> >> >> >>> >> >> >> > 3h
>> >> >> >> >>> >> >> >> > v1.7.2-rancher1
>> >> >> >> >>> >> >> >> > no/node-c.c.kubernetes-174104.internal   Ready
>> >> >> >> >>> >> >> >> > 3h
>> >> >> >> >>> >> >> >> > v1.7.2-rancher1
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > NAME             CLUSTER-IP   EXTERNAL-IP
>> >> >> >> >>> >> >> >> > PORT(S)
>> >> >> >> >>> >> >> >> > AGE
>> >> >> >> >>> >> >> >> > svc/kubernetes   10.43.0.1    <none>
>> >> >> >> >>> >> >> >> > 443/TCP
>> >> >> >> >>> >> >> >> > 3h
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > After running gk-deploy:
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >> >>> >> >> >> > [root at workstation messagegc]# kubectl get
>> >> >> >> >>> >> >> >> > nodes,pods,daemonset,deployments,services
>> >> >> >> >>> >> >> >> > NAME                                     STATUS
>> >> >> >> >>> >> >> >> > AGE
>> >> >> >> >>> >> >> >> > VERSION
>> >> >> >> >>> >> >> >> > no/node-a.c.kubernetes-174104.internal   Ready
>> >> >> >> >>> >> >> >> > 3h
>> >> >> >> >>> >> >> >> > v1.7.2-rancher1
>> >> >> >> >>> >> >> >> > no/node-b.c.kubernetes-174104.internal   Ready
>> >> >> >> >>> >> >> >> > 3h
>> >> >> >> >>> >> >> >> > v1.7.2-rancher1
>> >> >> >> >>> >> >> >> > no/node-c.c.kubernetes-174104.internal   Ready
>> >> >> >> >>> >> >> >> > 3h
>> >> >> >> >>> >> >> >> > v1.7.2-rancher1
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > NAME                 READY     STATUS    RESTARTS
>> >> >> >> >>> >> >> >> > AGE
>> >> >> >> >>> >> >> >> > po/glusterfs-0j9l5   0/1       Running   0
>> >> >> >> >>> >> >> >> > 2m
>> >> >> >> >>> >> >> >> > po/glusterfs-gqz4c   0/1       Running   0
>> >> >> >> >>> >> >> >> > 2m
>> >> >> >> >>> >> >> >> > po/glusterfs-gxvcb   0/1       Running   0
>> >> >> >> >>> >> >> >> > 2m
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > NAME           DESIRED   CURRENT   READY
>> >> >> >> >>> >> >> >> > UP-TO-DATE
>> >> >> >> >>> >> >> >> > AVAILABLE
>> >> >> >> >>> >> >> >> > NODE-SELECTOR           AGE
>> >> >> >> >>> >> >> >> > ds/glusterfs   3         3         0         3
>> >> >> >> >>> >> >> >> > 0
>> >> >> >> >>> >> >> >> > storagenode=glusterfs   2m
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > NAME             CLUSTER-IP   EXTERNAL-IP
>> >> >> >> >>> >> >> >> > PORT(S)
>> >> >> >> >>> >> >> >> > AGE
>> >> >> >> >>> >> >> >> > svc/kubernetes   10.43.0.1    <none>
>> >> >> >> >>> >> >> >> > 443/TCP
>> >> >> >> >>> >> >> >> > 3h
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > Kernel module check on all three nodes:
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >> >>> >> >> >> > [root at node-a ~]# find /lib*/modules/$(uname -r)
>> >> >> >> >>> >> >> >> > -name
>> >> >> >> >>> >> >> >> > *.ko
>> >> >> >> >>> >> >> >> > |
>> >> >> >> >>> >> >> >> > grep
>> >> >> >> >>> >> >> >> > 'thin-pool\|snapshot\|mirror' | xargs ls -ltr
>> >> >> >> >>> >> >> >> > -rw-r--r--    1 root     root         92310 Jun 26
>> >> >> >> >>> >> >> >> > 04:13
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > /lib64/modules/4.9.34-rancher/kernel/drivers/md/dm-thin-pool.ko
>> >> >> >> >>> >> >> >> > -rw-r--r--    1 root     root         56982 Jun 26
>> >> >> >> >>> >> >> >> > 04:13
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > /lib64/modules/4.9.34-rancher/kernel/drivers/md/dm-snapshot.ko
>> >> >> >> >>> >> >> >> > -rw-r--r--    1 root     root         27070 Jun 26
>> >> >> >> >>> >> >> >> > 04:13
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > /lib64/modules/4.9.34-rancher/kernel/drivers/md/dm-mirror.ko
>> >> >> >> >>> >> >> >> > -rw-r--r--    1 root     root         92310 Jun 26
>> >> >> >> >>> >> >> >> > 04:13
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > /lib/modules/4.9.34-rancher/kernel/drivers/md/dm-thin-pool.ko
>> >> >> >> >>> >> >> >> > -rw-r--r--    1 root     root         56982 Jun 26
>> >> >> >> >>> >> >> >> > 04:13
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > /lib/modules/4.9.34-rancher/kernel/drivers/md/dm-snapshot.ko
>> >> >> >> >>> >> >> >> > -rw-r--r--    1 root     root         27070 Jun 26
>> >> >> >> >>> >> >> >> > 04:13
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > /lib/modules/4.9.34-rancher/kernel/drivers/md/dm-mirror.ko
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > Error snapshot attached.
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > In my first mail, i checked that Readiness Probe
>> >> >> >> >>> >> >> >> > failure
>> >> >> >> >>> >> >> >> > check
>> >> >> >> >>> >> >> >> > has
>> >> >> >> >>> >> >> >> > this
>> >> >> >> >>> >> >> >> > code
>> >> >> >> >>> >> >> >> > in kube-templates/glusterfs-daemonset.yaml file:
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >> >>> >> >> >> >         readinessProbe:
>> >> >> >> >>> >> >> >> >           timeoutSeconds: 3
>> >> >> >> >>> >> >> >> >           initialDelaySeconds: 40
>> >> >> >> >>> >> >> >> >           exec:
>> >> >> >> >>> >> >> >> >             command:
>> >> >> >> >>> >> >> >> >             - "/bin/bash"
>> >> >> >> >>> >> >> >> >             - "-c"
>> >> >> >> >>> >> >> >> >             - systemctl status glusterd.service
>> >> >> >> >>> >> >> >> >           periodSeconds: 25
>> >> >> >> >>> >> >> >> >           successThreshold: 1
>> >> >> >> >>> >> >> >> >           failureThreshold: 15
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > ------------------------------------------------------------------
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > I tried logging into glustefs container on one of
>> >> >> >> >>> >> >> >> > the
>> >> >> >> >>> >> >> >> > node
>> >> >> >> >>> >> >> >> > and
>> >> >> >> >>> >> >> >> > ran
>> >> >> >> >>> >> >> >> > the
>> >> >> >> >>> >> >> >> > above
>> >> >> >> >>> >> >> >> > command:
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > [root at node-a ~]# docker exec -it
>> >> >> >> >>> >> >> >> > c0f8ab4d92a23b6df2
>> >> >> >> >>> >> >> >> > /bin/bash
>> >> >> >> >>> >> >> >> > root at c0f8ab4d92a2:/app# systemctl status
>> >> >> >> >>> >> >> >> > glusterd.service
>> >> >> >> >>> >> >> >> > WARNING: terminal is not fully functional
>> >> >> >> >>> >> >> >> > Failed to connect to bus: No such file or
>> >> >> >> >>> >> >> >> > directory
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > Any check that i can do manually on nodes to debug
>> >> >> >> >>> >> >> >> > further?
>> >> >> >> >>> >> >> >> > Any
>> >> >> >> >>> >> >> >> > suggestions?
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> > On Thu, Aug 31, 2017 at 6:53 PM, Jose A. Rivera
>> >> >> >> >>> >> >> >> > <jarrpa at redhat.com>
>> >> >> >> >>> >> >> >> > wrote:
>> >> >> >> >>> >> >> >> >>
>> >> >> >> >>> >> >> >> >> Hey Gaurav,
>> >> >> >> >>> >> >> >> >>
>> >> >> >> >>> >> >> >> >> The kernel modules must be loaded on all nodes
>> >> >> >> >>> >> >> >> >> that
>> >> >> >> >>> >> >> >> >> will
>> >> >> >> >>> >> >> >> >> run
>> >> >> >> >>> >> >> >> >> heketi
>> >> >> >> >>> >> >> >> >> pods. Additionally, you must have at least three
>> >> >> >> >>> >> >> >> >> nodes
>> >> >> >> >>> >> >> >> >> specified
>> >> >> >> >>> >> >> >> >> in
>> >> >> >> >>> >> >> >> >> your topology file. I'm not sure how you're
>> >> >> >> >>> >> >> >> >> getting
>> >> >> >> >>> >> >> >> >> three
>> >> >> >> >>> >> >> >> >> gluster
>> >> >> >> >>> >> >> >> >> pods
>> >> >> >> >>> >> >> >> >> when you only have two nodes defined... :)
>> >> >> >> >>> >> >> >> >>
>> >> >> >> >>> >> >> >> >> --Jose
>> >> >> >> >>> >> >> >> >>
>> >> >> >> >>> >> >> >> >> On Wed, Aug 30, 2017 at 5:27 AM, Gaurav Chhabra
>> >> >> >> >>> >> >> >> >> <varuag.chhabra at gmail.com> wrote:
>> >> >> >> >>> >> >> >> >> > Hi,
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > I have the following setup in place:
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > 1 node    : RancherOS having Rancher
>> >> >> >> >>> >> >> >> >> > application
>> >> >> >> >>> >> >> >> >> > for
>> >> >> >> >>> >> >> >> >> > Kubernetes
>> >> >> >> >>> >> >> >> >> > setup
>> >> >> >> >>> >> >> >> >> > 2 nodes  : RancherOS having Rancher agent
>> >> >> >> >>> >> >> >> >> > 1 node   : CentOS 7 workstation having kubectl
>> >> >> >> >>> >> >> >> >> > installed
>> >> >> >> >>> >> >> >> >> > and
>> >> >> >> >>> >> >> >> >> > folder
>> >> >> >> >>> >> >> >> >> > cloned/downloaded from
>> >> >> >> >>> >> >> >> >> > https://github.com/gluster/gluster-kubernetes
>> >> >> >> >>> >> >> >> >> > using
>> >> >> >> >>> >> >> >> >> > which i run Heketi setup (gk-deploy -g)
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > I also have rancher-glusterfs-server container
>> >> >> >> >>> >> >> >> >> > running
>> >> >> >> >>> >> >> >> >> > with
>> >> >> >> >>> >> >> >> >> > the
>> >> >> >> >>> >> >> >> >> > following
>> >> >> >> >>> >> >> >> >> > configuration:
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > ------------------------------------------------------------------
>> >> >> >> >>> >> >> >> >> > [root at node-1 rancher]# cat gluster-server.sh
>> >> >> >> >>> >> >> >> >> > #!/bin/bash
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > sudo docker run --name=gluster-server -d \
>> >> >> >> >>> >> >> >> >> >         --env 'SERVICE_NAME=gluster' \
>> >> >> >> >>> >> >> >> >> >         --restart always \
>> >> >> >> >>> >> >> >> >> >         --env 'GLUSTER_DATA=/srv/docker/gitlab'
>> >> >> >> >>> >> >> >> >> > \
>> >> >> >> >>> >> >> >> >> >         --publish 2222:22 \
>> >> >> >> >>> >> >> >> >> >         webcenter/rancher-glusterfs-server
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > ------------------------------------------------------------------
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > In /etc/heketi/heketi.json, following is the
>> >> >> >> >>> >> >> >> >> > only
>> >> >> >> >>> >> >> >> >> > modified
>> >> >> >> >>> >> >> >> >> > portion:
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > ------------------------------------------------------------------
>> >> >> >> >>> >> >> >> >> >     "executor": "ssh",
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >     "_sshexec_comment": "SSH username and
>> >> >> >> >>> >> >> >> >> > private
>> >> >> >> >>> >> >> >> >> > key
>> >> >> >> >>> >> >> >> >> > file
>> >> >> >> >>> >> >> >> >> > information",
>> >> >> >> >>> >> >> >> >> >     "sshexec": {
>> >> >> >> >>> >> >> >> >> >       "keyfile": "/var/lib/heketi/.ssh/id_rsa",
>> >> >> >> >>> >> >> >> >> >       "user": "root",
>> >> >> >> >>> >> >> >> >> >       "port": "22",
>> >> >> >> >>> >> >> >> >> >       "fstab": "/etc/fstab"
>> >> >> >> >>> >> >> >> >> >     },
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > ------------------------------------------------------------------
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > Status before running gk-deploy:
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > [root at workstation deploy]# kubectl get
>> >> >> >> >>> >> >> >> >> > nodes,pods,services,deployments
>> >> >> >> >>> >> >> >> >> > NAME                                     STATUS
>> >> >> >> >>> >> >> >> >> > AGE
>> >> >> >> >>> >> >> >> >> > VERSION
>> >> >> >> >>> >> >> >> >> > no/node-1.c.kubernetes-174104.internal   Ready
>> >> >> >> >>> >> >> >> >> > 2d
>> >> >> >> >>> >> >> >> >> > v1.7.2-rancher1
>> >> >> >> >>> >> >> >> >> > no/node-2.c.kubernetes-174104.internal   Ready
>> >> >> >> >>> >> >> >> >> > 2d
>> >> >> >> >>> >> >> >> >> > v1.7.2-rancher1
>> >> >> >> >>> >> >> >> >> > no/node-3.c.kubernetes-174104.internal   Ready
>> >> >> >> >>> >> >> >> >> > 2d
>> >> >> >> >>> >> >> >> >> > v1.7.2-rancher1
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > NAME             CLUSTER-IP   EXTERNAL-IP
>> >> >> >> >>> >> >> >> >> > PORT(S)
>> >> >> >> >>> >> >> >> >> > AGE
>> >> >> >> >>> >> >> >> >> > svc/kubernetes   10.43.0.1    <none>
>> >> >> >> >>> >> >> >> >> > 443/TCP
>> >> >> >> >>> >> >> >> >> > 2d
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > Now when i run 'gk-deploy -g', in the Rancher
>> >> >> >> >>> >> >> >> >> > console, i
>> >> >> >> >>> >> >> >> >> > see
>> >> >> >> >>> >> >> >> >> > the
>> >> >> >> >>> >> >> >> >> > following
>> >> >> >> >>> >> >> >> >> > error:
>> >> >> >> >>> >> >> >> >> > Readiness probe failed: Failed to get D-Bus
>> >> >> >> >>> >> >> >> >> > connection:
>> >> >> >> >>> >> >> >> >> > Operation
>> >> >> >> >>> >> >> >> >> > not
>> >> >> >> >>> >> >> >> >> > permitted
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > From the attached gk-deploy_log i see that it
>> >> >> >> >>> >> >> >> >> > failed
>> >> >> >> >>> >> >> >> >> > at:
>> >> >> >> >>> >> >> >> >> > Waiting for GlusterFS pods to start ... pods
>> >> >> >> >>> >> >> >> >> > not
>> >> >> >> >>> >> >> >> >> > found.
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > In the kube-templates/glusterfs-daemonset.yaml
>> >> >> >> >>> >> >> >> >> > file,
>> >> >> >> >>> >> >> >> >> > i
>> >> >> >> >>> >> >> >> >> > see
>> >> >> >> >>> >> >> >> >> > this
>> >> >> >> >>> >> >> >> >> > for
>> >> >> >> >>> >> >> >> >> > Readiness probe section:
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > ------------------------------------------------------------------
>> >> >> >> >>> >> >> >> >> >         readinessProbe:
>> >> >> >> >>> >> >> >> >> >           timeoutSeconds: 3
>> >> >> >> >>> >> >> >> >> >           initialDelaySeconds: 40
>> >> >> >> >>> >> >> >> >> >           exec:
>> >> >> >> >>> >> >> >> >> >             command:
>> >> >> >> >>> >> >> >> >> >             - "/bin/bash"
>> >> >> >> >>> >> >> >> >> >             - "-c"
>> >> >> >> >>> >> >> >> >> >             - systemctl status glusterd.service
>> >> >> >> >>> >> >> >> >> >           periodSeconds: 25
>> >> >> >> >>> >> >> >> >> >           successThreshold: 1
>> >> >> >> >>> >> >> >> >> >           failureThreshold: 15
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > ------------------------------------------------------------------
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > Status after running gk-deploy:
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > [root at workstation deploy]# kubectl get
>> >> >> >> >>> >> >> >> >> > nodes,pods,deployments,services
>> >> >> >> >>> >> >> >> >> > NAME                                     STATUS
>> >> >> >> >>> >> >> >> >> > AGE
>> >> >> >> >>> >> >> >> >> > VERSION
>> >> >> >> >>> >> >> >> >> > no/node-1.c.kubernetes-174104.internal   Ready
>> >> >> >> >>> >> >> >> >> > 2d
>> >> >> >> >>> >> >> >> >> > v1.7.2-rancher1
>> >> >> >> >>> >> >> >> >> > no/node-2.c.kubernetes-174104.internal   Ready
>> >> >> >> >>> >> >> >> >> > 2d
>> >> >> >> >>> >> >> >> >> > v1.7.2-rancher1
>> >> >> >> >>> >> >> >> >> > no/node-3.c.kubernetes-174104.internal   Ready
>> >> >> >> >>> >> >> >> >> > 2d
>> >> >> >> >>> >> >> >> >> > v1.7.2-rancher1
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > NAME                 READY     STATUS
>> >> >> >> >>> >> >> >> >> > RESTARTS
>> >> >> >> >>> >> >> >> >> > AGE
>> >> >> >> >>> >> >> >> >> > po/glusterfs-0s440   0/1       Running   0
>> >> >> >> >>> >> >> >> >> > 1m
>> >> >> >> >>> >> >> >> >> > po/glusterfs-j7dgr   0/1       Running   0
>> >> >> >> >>> >> >> >> >> > 1m
>> >> >> >> >>> >> >> >> >> > po/glusterfs-p6jl3   0/1       Running   0
>> >> >> >> >>> >> >> >> >> > 1m
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > NAME             CLUSTER-IP   EXTERNAL-IP
>> >> >> >> >>> >> >> >> >> > PORT(S)
>> >> >> >> >>> >> >> >> >> > AGE
>> >> >> >> >>> >> >> >> >> > svc/kubernetes   10.43.0.1    <none>
>> >> >> >> >>> >> >> >> >> > 443/TCP
>> >> >> >> >>> >> >> >> >> > 2d
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > Also, from prerequisite perspective, i was also
>> >> >> >> >>> >> >> >> >> > seeing
>> >> >> >> >>> >> >> >> >> > this
>> >> >> >> >>> >> >> >> >> > mentioned:
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > The following kernel modules must be loaded:
>> >> >> >> >>> >> >> >> >> >  * dm_snapshot
>> >> >> >> >>> >> >> >> >> >  * dm_mirror
>> >> >> >> >>> >> >> >> >> >  * dm_thin_pool
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > Where exactly is this to be checked? On all
>> >> >> >> >>> >> >> >> >> > Gluster
>> >> >> >> >>> >> >> >> >> > server
>> >> >> >> >>> >> >> >> >> > nodes?
>> >> >> >> >>> >> >> >> >> > How
>> >> >> >> >>> >> >> >> >> > can i
>> >> >> >> >>> >> >> >> >> > check whether it's there?
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > I have attached topology.json and gk-deploy log
>> >> >> >> >>> >> >> >> >> > for
>> >> >> >> >>> >> >> >> >> > reference.
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > Does this issue has anything to do with the
>> >> >> >> >>> >> >> >> >> > host
>> >> >> >> >>> >> >> >> >> > OS
>> >> >> >> >>> >> >> >> >> > (RancherOS)
>> >> >> >> >>> >> >> >> >> > that
>> >> >> >> >>> >> >> >> >> > i
>> >> >> >> >>> >> >> >> >> > am
>> >> >> >> >>> >> >> >> >> > using for Gluster nodes? Any idea how i can fix
>> >> >> >> >>> >> >> >> >> > this?
>> >> >> >> >>> >> >> >> >> > Any
>> >> >> >> >>> >> >> >> >> > help
>> >> >> >> >>> >> >> >> >> > will
>> >> >> >> >>> >> >> >> >> > really
>> >> >> >> >>> >> >> >> >> > be appreciated.
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > Thanks.
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > _______________________________________________
>> >> >> >> >>> >> >> >> >> > heketi-devel mailing list
>> >> >> >> >>> >> >> >> >> > heketi-devel at gluster.org
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >> > http://lists.gluster.org/mailman/listinfo/heketi-devel
>> >> >> >> >>> >> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >> >
>> >> >> >> >>> >> >> >
>> >> >> >> >>> >> >> >
>> >> >> >> >>> >> >
>> >> >> >> >>> >> >
>> >> >> >> >>> >
>> >> >> >> >>> >
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >
>> >> >> >
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>
>


More information about the heketi-devel mailing list