<div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">Hi Jose,</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">Thanks for the reply. It seems the three gluster pods might have been a copy-paste from another set of cluster where i was trying to setup the same thing using CentOS. Sorry for that. By the way, i did check for the kernel modules and it seems it's already there. Also, i am attaching <i>fresh set of files</i> because i created a new cluster and thought of giving it a try again. Issue still persists. :(</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">In <i>heketi.json</i>, there is a slight change w.r.t the user which connects to glusterfs node using SSH. I am not sure how Heketi was using root user to login because i wasn't able to use root and do manual SSH. With <i>rancher</i> user, i can login successfully so i think this should be fine.</div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><div style="color:rgb(80,0,80);font-size:12.8px">/etc/heketi/heketi.json:</div><div style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:12.8px"><div><font face="trebuchet ms, sans-serif">------------------------------<wbr>------------------------------<wbr>------</font></div><div><font face="trebuchet ms, sans-serif"> "executor": "ssh",</font></div><div><font face="trebuchet ms, sans-serif"><br></font></div><div><font face="trebuchet ms, sans-serif"> "_sshexec_comment": "SSH username and private key file information",</font></div><div><font face="trebuchet ms, sans-serif"> "sshexec": {</font></div><div><font face="trebuchet ms, sans-serif"> "keyfile": "/var/lib/heketi/.ssh/id_rsa",</font></div><div><font face="trebuchet ms, sans-serif"> "user": "<b>rancher</b>",</font></div><div><font face="trebuchet ms, sans-serif"> "port": "22",</font></div><div><font face="trebuchet ms, sans-serif"> "fstab": "/etc/fstab"</font></div><div><font face="trebuchet ms, sans-serif"> },</font></div><div style="font-family:verdana,sans-serif"><span style="font-family:"trebuchet ms",sans-serif">------------------------------<wbr>------------------------------<wbr>------</span></div></div></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">Before running gk-deploy:</div><div class="gmail_default"><div class="gmail_default"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px">------------------------------</span><wbr style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px">------------------------------</span><wbr style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px">------</span><font face="trebuchet ms, sans-serif"><br></font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">[root@workstation deploy]# kubectl get nodes,pods,daemonset,deployments,services</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">NAME STATUS AGE VERSION</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">no/node-a.c.kubernetes-174104.internal Ready 3h v1.7.2-rancher1</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">no/node-b.c.kubernetes-174104.internal Ready 3h v1.7.2-rancher1</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">no/node-c.c.kubernetes-174104.internal Ready 3h v1.7.2-rancher1</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"><br></font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">svc/kubernetes 10.43.0.1 <none> 443/TCP 3h</font></div></div><div class="gmail_default" style="font-family:verdana,sans-serif"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px">------------------------------</span><wbr style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px">------------------------------</span><wbr style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px">------</span><br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px"><br></span></div><div class="gmail_default"><span style="color:rgb(80,0,80);font-size:12.8px"><font face="verdana, sans-serif">After running gk-deploy:</font></span></div><div class="gmail_default"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px">------------------------------</span><wbr style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px">------------------------------</span><wbr style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px">------</span><span style="color:rgb(80,0,80);font-size:12.8px"><font face="verdana, sans-serif"><br></font></span></div><div class="gmail_default"><font color="#500050" face="trebuchet ms, sans-serif"><span style="font-size:12.8px"><div class="gmail_default">[root@workstation messagegc]# kubectl get nodes,pods,daemonset,deployments,services</div><div class="gmail_default">NAME STATUS AGE VERSION</div><div class="gmail_default">no/node-a.c.kubernetes-174104.internal Ready 3h v1.7.2-rancher1</div><div class="gmail_default">no/node-b.c.kubernetes-174104.internal Ready 3h v1.7.2-rancher1</div><div class="gmail_default">no/node-c.c.kubernetes-174104.internal Ready 3h v1.7.2-rancher1</div><div class="gmail_default"><br></div><div class="gmail_default">NAME READY STATUS RESTARTS AGE</div><div class="gmail_default">po/glusterfs-0j9l5 0/1 Running 0 2m</div><div class="gmail_default">po/glusterfs-gqz4c 0/1 Running 0 2m</div><div class="gmail_default">po/glusterfs-gxvcb 0/1 Running 0 2m</div><div class="gmail_default"><br></div><div class="gmail_default">NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE-SELECTOR AGE</div><div class="gmail_default">ds/glusterfs 3 3 0 3 0 storagenode=glusterfs 2m</div><div class="gmail_default"><br></div><div class="gmail_default">NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE</div><div class="gmail_default">svc/kubernetes 10.43.0.1 <none> 443/TCP 3h</div></span></font></div><div class="gmail_default" style="font-family:verdana,sans-serif"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px">------------------------------</span><wbr style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px">------------------------------</span><wbr style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px">------</span><br></div><div class="gmail_default" style="font-family:verdana,sans-serif"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px"><br></span></div><div class="gmail_default"><span style="color:rgb(80,0,80);font-size:12.8px"><font face="verdana, sans-serif">Kernel module check on all three nodes:</font></span></div><div class="gmail_default"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px">------------------------------</span><wbr style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px">------------------------------</span><wbr style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px">------</span><span style="color:rgb(80,0,80);font-size:12.8px"><font face="verdana, sans-serif"><br></font></span></div><div class="gmail_default"><span style="color:rgb(80,0,80);font-size:12.8px"><font face="trebuchet ms, sans-serif"><div class="gmail_default">[root@node-a ~]# find /lib*/modules/$(uname -r) -name *.ko | grep 'thin-pool\|snapshot\|mirror' | xargs ls -ltr</div><div class="gmail_default">-rw-r--r-- 1 root root 92310 Jun 26 04:13 /lib64/modules/4.9.34-rancher/kernel/drivers/md/dm-thin-pool.ko</div><div class="gmail_default">-rw-r--r-- 1 root root 56982 Jun 26 04:13 /lib64/modules/4.9.34-rancher/kernel/drivers/md/dm-snapshot.ko</div><div class="gmail_default">-rw-r--r-- 1 root root 27070 Jun 26 04:13 /lib64/modules/4.9.34-rancher/kernel/drivers/md/dm-mirror.ko</div><div class="gmail_default">-rw-r--r-- 1 root root 92310 Jun 26 04:13 /lib/modules/4.9.34-rancher/kernel/drivers/md/dm-thin-pool.ko</div><div class="gmail_default">-rw-r--r-- 1 root root 56982 Jun 26 04:13 /lib/modules/4.9.34-rancher/kernel/drivers/md/dm-snapshot.ko</div><div class="gmail_default">-rw-r--r-- 1 root root 27070 Jun 26 04:13 /lib/modules/4.9.34-rancher/kernel/drivers/md/dm-mirror.ko</div></font></span></div><div class="gmail_default" style="font-family:verdana,sans-serif"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px">------------------------------</span><wbr style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px">------------------------------</span><wbr style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px">------</span><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px"><br></span></div><div class="gmail_default" style="font-family:verdana,sans-serif"><span style="color:rgb(80,0,80);font-family:"trebuchet ms",sans-serif;font-size:12.8px"><br></span></div><div class="gmail_default"><span style="font-size:12.8px"><font face="verdana, sans-serif" color="#ff0000">Error snapshot</font><font face="verdana, sans-serif" color="#000000"> attached.</font></span></div><div class="gmail_default"><span style="font-size:12.8px"><font face="verdana, sans-serif" color="#000000"><br></font></span></div><div class="gmail_default"><span style="font-family:verdana,sans-serif;font-size:12.8px"><font color="#000000">In my first mail, i checked that Readiness Probe failure check has this code in </font></span><font face="trebuchet ms, sans-serif"><span style="color:rgb(0,0,0);font-size:12.8px">kube-templates/glusterfs-d</span><wbr style="color:rgb(0,0,0);font-size:12.8px"></font><span style="color:rgb(0,0,0);font-size:12.8px"><font face="trebuchet ms, sans-serif">aemonset.yaml</font><font face="verdana, sans-serif"> file: </font></span></div><div class="gmail_default"><div style="color:rgb(80,0,80);font-size:12.8px"><span style="font-family:"trebuchet ms",sans-serif">------------------------------<wbr>------------------------------<wbr>------</span><font face="trebuchet ms, sans-serif" color="#000000"><br></font></div><div style="color:rgb(80,0,80);font-size:12.8px"><font face="trebuchet ms, sans-serif" color="#000000"><div> readinessProbe:</div><div> timeoutSeconds: 3</div><div> initialDelaySeconds: 40</div><div> exec:</div><div> command:</div><div> - "/bin/bash"</div><div> - "-c"</div><div> - systemctl status glusterd.service</div><div> periodSeconds: 25</div><div> successThreshold: 1</div><div> failureThreshold: 15</div></font></div><div style="color:rgb(80,0,80);font-size:12.8px"><span style="font-family:"trebuchet ms",sans-serif">------------------------------<wbr>------------------------------<wbr>------</span></div></div><div class="gmail_default"><span style="font-size:12.8px"><font face="verdana, sans-serif" color="#000000"><br></font></span></div><div class="gmail_default"><span style="font-size:12.8px"><font face="verdana, sans-serif" color="#000000">I tried logging into glustefs container on one of the node and ran the above command:</font></span></div><div class="gmail_default"><span style="font-size:12.8px"><font face="verdana, sans-serif" color="#000000"><br></font></span></div><div class="gmail_default"><font color="#000000" face="trebuchet ms, sans-serif"><span style="font-size:12.8px">[root@node-a ~]# docker exec -it c0f8ab4d92a23b6df2 /bin/bash</span><br></font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"><span style="font-size:12.8px"><div class="gmail_default" style="color:rgb(0,0,0)">root@c0f8ab4d92a2:/app# systemctl status glusterd.service</div><div class="gmail_default" style="color:rgb(0,0,0)">WARNING: terminal is not fully functional</div><div class="gmail_default" style=""><font color="#ff0000">Failed to connect to bus: No such file or directory</font></div></span></font></div><div class="gmail_default"><font face="verdana, sans-serif" color="#000000"><span style="font-size:12.8px"><br></span></font></div><div class="gmail_default"><span style="font-size:12.8px"><font face="verdana, sans-serif" color="#000000"><br></font></span></div><div class="gmail_default"><span style="font-size:12.8px"><font face="verdana, sans-serif" color="#000000">Any check that i can do manually on nodes to debug further? Any suggestions?</font></span></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 31, 2017 at 6:53 PM, Jose A. Rivera <span dir="ltr"><<a href="mailto:jarrpa@redhat.com" target="_blank">jarrpa@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey Gaurav,<br>
<br>
The kernel modules must be loaded on all nodes that will run heketi<br>
pods. Additionally, you must have at least three nodes specified in<br>
your topology file. I'm not sure how you're getting three gluster pods<br>
when you only have two nodes defined... :)<br>
<br>
--Jose<br>
<div><div class="h5"><br>
On Wed, Aug 30, 2017 at 5:27 AM, Gaurav Chhabra<br>
<<a href="mailto:varuag.chhabra@gmail.com">varuag.chhabra@gmail.com</a>> wrote:<br>
> Hi,<br>
><br>
><br>
> I have the following setup in place:<br>
><br>
> 1 node : RancherOS having Rancher application for Kubernetes setup<br>
> 2 nodes : RancherOS having Rancher agent<br>
> 1 node : CentOS 7 workstation having kubectl installed and folder<br>
> cloned/downloaded from <a href="https://github.com/gluster/gluster-kubernetes" rel="noreferrer" target="_blank">https://github.com/gluster/<wbr>gluster-kubernetes</a> using<br>
> which i run Heketi setup (gk-deploy -g)<br>
><br>
> I also have rancher-glusterfs-server container running with the following<br>
> configuration:<br>
> ------------------------------<wbr>------------------------------<wbr>------<br>
> [root@node-1 rancher]# cat gluster-server.sh<br>
> #!/bin/bash<br>
><br>
> sudo docker run --name=gluster-server -d \<br>
> --env 'SERVICE_NAME=gluster' \<br>
> --restart always \<br>
> --env 'GLUSTER_DATA=/srv/docker/<wbr>gitlab' \<br>
> --publish 2222:22 \<br>
> webcenter/rancher-glusterfs-<wbr>server<br>
> ------------------------------<wbr>------------------------------<wbr>------<br>
><br>
> In /etc/heketi/heketi.json, following is the only modified portion:<br>
> ------------------------------<wbr>------------------------------<wbr>------<br>
> "executor": "ssh",<br>
><br>
> "_sshexec_comment": "SSH username and private key file information",<br>
> "sshexec": {<br>
> "keyfile": "/var/lib/heketi/.ssh/id_rsa",<br>
> "user": "root",<br>
> "port": "22",<br>
> "fstab": "/etc/fstab"<br>
> },<br>
> ------------------------------<wbr>------------------------------<wbr>------<br>
><br>
> Status before running gk-deploy:<br>
><br>
> [root@workstation deploy]# kubectl get nodes,pods,services,<wbr>deployments<br>
> NAME STATUS AGE VERSION<br>
> no/node-1.c.kubernetes-174104.<wbr>internal Ready 2d v1.7.2-rancher1<br>
> no/node-2.c.kubernetes-174104.<wbr>internal Ready 2d v1.7.2-rancher1<br>
> no/node-3.c.kubernetes-174104.<wbr>internal Ready 2d v1.7.2-rancher1<br>
><br>
> NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE<br>
> svc/kubernetes 10.43.0.1 <none> 443/TCP 2d<br>
><br>
><br>
> Now when i run 'gk-deploy -g', in the Rancher console, i see the following<br>
> error:<br>
> Readiness probe failed: Failed to get D-Bus connection: Operation not<br>
> permitted<br>
><br>
> From the attached gk-deploy_log i see that it failed at:<br>
> Waiting for GlusterFS pods to start ... pods not found.<br>
><br>
> In the kube-templates/glusterfs-<wbr>daemonset.yaml file, i see this for<br>
> Readiness probe section:<br>
> ------------------------------<wbr>------------------------------<wbr>------<br>
> readinessProbe:<br>
> timeoutSeconds: 3<br>
> initialDelaySeconds: 40<br>
> exec:<br>
> command:<br>
> - "/bin/bash"<br>
> - "-c"<br>
> - systemctl status glusterd.service<br>
> periodSeconds: 25<br>
> successThreshold: 1<br>
> failureThreshold: 15<br>
> ------------------------------<wbr>------------------------------<wbr>------<br>
><br>
><br>
> Status after running gk-deploy:<br>
><br>
> [root@workstation deploy]# kubectl get nodes,pods,deployments,<wbr>services<br>
> NAME STATUS AGE VERSION<br>
> no/node-1.c.kubernetes-174104.<wbr>internal Ready 2d v1.7.2-rancher1<br>
> no/node-2.c.kubernetes-174104.<wbr>internal Ready 2d v1.7.2-rancher1<br>
> no/node-3.c.kubernetes-174104.<wbr>internal Ready 2d v1.7.2-rancher1<br>
><br>
> NAME READY STATUS RESTARTS AGE<br>
> po/glusterfs-0s440 0/1 Running 0 1m<br>
> po/glusterfs-j7dgr 0/1 Running 0 1m<br>
> po/glusterfs-p6jl3 0/1 Running 0 1m<br>
><br>
> NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE<br>
> svc/kubernetes 10.43.0.1 <none> 443/TCP 2d<br>
><br>
><br>
> Also, from prerequisite perspective, i was also seeing this mentioned:<br>
><br>
> The following kernel modules must be loaded:<br>
> * dm_snapshot<br>
> * dm_mirror<br>
> * dm_thin_pool<br>
><br>
> Where exactly is this to be checked? On all Gluster server nodes? How can i<br>
> check whether it's there?<br>
><br>
> I have attached topology.json and gk-deploy log for reference.<br>
><br>
> Does this issue has anything to do with the host OS (RancherOS) that i am<br>
> using for Gluster nodes? Any idea how i can fix this? Any help will really<br>
> be appreciated.<br>
><br>
><br>
> Thanks.<br>
><br>
><br>
><br>
</div></div>> ______________________________<wbr>_________________<br>
> heketi-devel mailing list<br>
> <a href="mailto:heketi-devel@gluster.org">heketi-devel@gluster.org</a><br>
> <a href="http://lists.gluster.org/mailman/listinfo/heketi-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/heketi-devel</a><br>
><br>
</blockquote></div><br></div>