[heketi-devel] Unable to use Heketi setup to install Gluster for Kubernetes

[Jose A. Rivera]

Hey Gaurav,

The kernel modules must be loaded on all nodes that will run heketi
pods. Additionally, you must have at least three nodes specified in
your topology file. I'm not sure how you're getting three gluster pods
when you only have two nodes defined... :)

--Jose

On Wed, Aug 30, 2017 at 5:27 AM, Gaurav Chhabra
<varuag.chhabra at gmail.com> wrote:
> Hi,
>
>
> I have the following setup in place:
>
> 1 node    : RancherOS having Rancher application for Kubernetes setup
> 2 nodes  : RancherOS having Rancher agent
> 1 node   : CentOS 7 workstation having kubectl installed and folder
> cloned/downloaded from https://github.com/gluster/gluster-kubernetes using
> which i run Heketi setup (gk-deploy -g)
>
> I also have rancher-glusterfs-server container running with the following
> configuration:
> ------------------------------------------------------------------
> [root at node-1 rancher]# cat gluster-server.sh
> #!/bin/bash
>
> sudo docker run --name=gluster-server -d \
>         --env 'SERVICE_NAME=gluster' \
>         --restart always \
>         --env 'GLUSTER_DATA=/srv/docker/gitlab' \
>         --publish 2222:22 \
>         webcenter/rancher-glusterfs-server
> ------------------------------------------------------------------
>
> In /etc/heketi/heketi.json, following is the only modified portion:
> ------------------------------------------------------------------
>     "executor": "ssh",
>
>     "_sshexec_comment": "SSH username and private key file information",
>     "sshexec": {
>       "keyfile": "/var/lib/heketi/.ssh/id_rsa",
>       "user": "root",
>       "port": "22",
>       "fstab": "/etc/fstab"
>     },
> ------------------------------------------------------------------
>
> Status before running gk-deploy:
>
> [root at workstation deploy]# kubectl get nodes,pods,services,deployments
> NAME                                     STATUS    AGE       VERSION
> no/node-1.c.kubernetes-174104.internal   Ready     2d        v1.7.2-rancher1
> no/node-2.c.kubernetes-174104.internal   Ready     2d        v1.7.2-rancher1
> no/node-3.c.kubernetes-174104.internal   Ready     2d        v1.7.2-rancher1
>
> NAME             CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
> svc/kubernetes   10.43.0.1    <none>        443/TCP   2d
>
>
> Now when i run 'gk-deploy -g', in the Rancher console, i see the following
> error:
> Readiness probe failed: Failed to get D-Bus connection: Operation not
> permitted
>
> From the attached gk-deploy_log i see that it failed at:
> Waiting for GlusterFS pods to start ... pods not found.
>
> In the kube-templates/glusterfs-daemonset.yaml file, i see this for
> Readiness probe section:
> ------------------------------------------------------------------
>         readinessProbe:
>           timeoutSeconds: 3
>           initialDelaySeconds: 40
>           exec:
>             command:
>             - "/bin/bash"
>             - "-c"
>             - systemctl status glusterd.service
>           periodSeconds: 25
>           successThreshold: 1
>           failureThreshold: 15
> ------------------------------------------------------------------
>
>
> Status after running gk-deploy:
>
> [root at workstation deploy]# kubectl get nodes,pods,deployments,services
> NAME                                     STATUS    AGE       VERSION
> no/node-1.c.kubernetes-174104.internal   Ready     2d        v1.7.2-rancher1
> no/node-2.c.kubernetes-174104.internal   Ready     2d        v1.7.2-rancher1
> no/node-3.c.kubernetes-174104.internal   Ready     2d        v1.7.2-rancher1
>
> NAME                 READY     STATUS    RESTARTS   AGE
> po/glusterfs-0s440   0/1       Running   0          1m
> po/glusterfs-j7dgr   0/1       Running   0          1m
> po/glusterfs-p6jl3   0/1       Running   0          1m
>
> NAME             CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
> svc/kubernetes   10.43.0.1    <none>        443/TCP   2d
>
>
> Also, from prerequisite perspective, i was also seeing this mentioned:
>
> The following kernel modules must be loaded:
>  * dm_snapshot
>  * dm_mirror
>  * dm_thin_pool
>
> Where exactly is this to be checked? On all Gluster server nodes? How can i
> check whether it's there?
>
> I have attached topology.json and gk-deploy log for reference.
>
> Does this issue has anything to do with the host OS (RancherOS) that i am
> using for Gluster nodes? Any idea how i can fix this? Any help will really
> be appreciated.
>
>
> Thanks.
>
>
>
> _______________________________________________
> heketi-devel mailing list
> heketi-devel at gluster.org
> http://lists.gluster.org/mailman/listinfo/heketi-devel
>