<div dir="ltr"><div class="gmail_default"><div class="gmail_default"><font face="verdana, sans-serif">Hi Jose,</font></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font face="verdana, sans-serif">webcenter/rancher-glusterfs-server is actually a container provided by Sebastien, its maintainer. It's a Docker container which has GlusterFS server running within it. On the host i.e., RancherOS, there is no separate GlusterFS server running because we cannot install anything that way. Running using container is the only way so i started ancher-glusterfs-server container with the following parameters:</font></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">[root@node-1 rancher]# cat gluster-server.sh</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">#!/bin/bash</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">sudo docker run --name=gluster-server -d \</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"> --env 'SERVICE_NAME=gluster' \</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"> --restart always \</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"> --publish 2222:22 \</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"> webcenter/rancher-glusterfs-server</font></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><span style="white-space:pre"><font face="verdana, sans-serif"> </font></span></div><div class="gmail_default"><font face="verdana, sans-serif">Here's the link to the dockerfile:</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"><a href="https://hub.docker.com/r/webcenter/rancher-glusterfs-server/~/dockerfile/">https://hub.docker.com/r/webcenter/rancher-glusterfs-server/~/dockerfile/</a></font></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font face="verdana, sans-serif">It's similar to other GlusteFS containers provided by other maintainers for different OS. For example, for CentOS, we have</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"><a href="https://hub.docker.com/r/gluster/gluster-centos/~/dockerfile/">https://hub.docker.com/r/gluster/gluster-centos/~/dockerfile/</a></font></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font face="verdana, sans-serif">From what i understand, Heketi does support container based GlusterFS server as mentioned in the <a href="https://github.com/gluster/gluster-kubernetes/blob/master/docs/setup-guide.md">prerequisite</a> where it says:</font></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font face="verdana, sans-serif">"</font><font face="trebuchet ms, sans-serif">Each node must have the following ports opened for GlusterFS communications:</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"> 2222 - GlusterFS pod's sshd</font><font face="verdana, sans-serif">"</font></div><div class="gmail_default"><font face="verdana, sans-serif"> </font></div><div class="gmail_default"><font face="verdana, sans-serif">That's the reason i've exposed port 2222 for 22 as shown above. </font><span style="font-family:verdana,sans-serif">Please correct me if i misunderstood it.</span></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font face="verdana, sans-serif">As soon as i run the above script (</font><font face="trebuchet ms, sans-serif">gluster-server.sh</font><font face="verdana, sans-serif">), it automatically creates the following directories on host. This should have ideally not been empty as you mentioned.</font></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">/etc/glusterfs /var/lib/glusterd /var/log/glusterfs</font></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font face="verdana, sans-serif">Just wanted to know in which circumstances do we get this specific error (</font><font face="trebuchet ms, sans-serif">Failed to get D-Bus connection: Operation not permitted</font><font face="verdana, sans-serif">) related to Readiness probe failing. Searching online took me to <a href="https://serverfault.com/questions/824975/failed-to-get-d-bus-connection-operation-not-permitted">discussions</a> around running container in <i>privileged</i> mode and some directory to be mounted. Based on that, i also modified my container startup script to the following:</font></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">#!/bin/bash</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">sudo docker run --privileged \</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"> --name=gluster-server \</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"> -d \</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"> -v /sys/fs/cgroup:/sys/fs/cgroup \</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"> -v /etc/glusterfs:/etc/glusterfs \</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"> -v /var/lib/glusterd:/var/lib/glusterd \</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"> -v /var/log/glusterfs:/var/log/glusterfs \</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"> --env 'SERVICE_NAME=gluster' \</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"> --restart always \</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"> --publish 2222:22 \</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif"> webcenter/rancher-glusterfs-server</font></div><div class="gmail_default"><span style="white-space:pre"><font face="verdana, sans-serif"> </font></span></div><div class="gmail_default"><font face="verdana, sans-serif">Still, the issue persists.</font></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font face="verdana, sans-serif">I also logged into the container and checked whether systemctl command is present. It was there but manualy running the command also doesn't work:</font></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">[root@node-c ~]# docker exec -it gluster-server /bin/bash</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">root@42150f203f80:/app# systemctl status glusterd.service</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">WARNING: terminal is not fully functional</font></div><div class="gmail_default"><font face="trebuchet ms, sans-serif">Failed to connect to bus: No such file or directory</font></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><span style="font-family:verdana,sans-serif">Under section 'ADVANCED OPTIONS - Security/Host' in </span><a href="https://github.com/disaster37/rancher-glusterfs-server" style="font-family:verdana,sans-serif">this</a><span style="font-family:verdana,sans-serif"> link, it talks about SYS_ADMIN setting. Any idea how i can try this? </span><br></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font face="verdana, sans-serif">Also, there was this mentioned in the Heketi <a href="https://github.com/gluster/gluster-kubernetes/blob/master/docs/setup-guide.md#infrastructure-requirements">setup</a> page:</font></div><div class="gmail_default"><font face="verdana, sans-serif">"</font><font face="trebuchet ms, sans-serif">If you are not able to deploy a hyper-converged GlusterFS cluster, you must have one running somewhere that the Kubernetes nodes can access</font><font face="verdana, sans-serif">"</font></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font face="verdana, sans-serif">>>> Does it mean running the three node Gluster cluster <i>outside</i> Kubernetes, may be on some VM running on RHEL/CentOS etc? If yes, then <u>how</u> will i be able to tell Gluster which volume from the Kubernetes cluster pod to sync? Any references?</font></div><div class="gmail_default"><br></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font face="verdana, sans-serif">I really appreciate your responses despite the fact that you've not used RancherOS but still trying to help.</font></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div><div class="gmail_default"><font face="verdana, sans-serif">Thanks,</font></div><div class="gmail_default"><font face="verdana, sans-serif">Gaurav</font></div><div class="gmail_default"><font face="verdana, sans-serif"><br></font></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Sep 2, 2017 at 7:35 PM, Jose A. Rivera <span dir="ltr"><<a href="mailto:jarrpa@redhat.com" target="_blank">jarrpa@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I'm afraid I have no experience with RancherOS, so I may be missing<br>
some things about how it works. My primary experience is with Fedora,<br>
CentOS, and Ubuntu.<br>
<br>
What is webcenter/rancher-glusterfs-<wbr>server? If it's running another<br>
glusterd then you probably don't want to be running it and should<br>
remove it from your systems.<br>
<br>
The glusterfs pods mount hostpath volumes from the host they're<br>
running on to persist their configuration. Thus anything they write to<br>
those directories should land on the host. If that's not happening<br>
then that's an additional problem.<br>
<br>
--Jose<br>
<br>
On Fri, Sep 1, 2017 at 11:17 PM, Gaurav Chhabra<br>
<div class="HOEnZb"><div class="h5"><<a href="mailto:varuag.chhabra@gmail.com">varuag.chhabra@gmail.com</a>> wrote:<br>
> Hi Jose,<br>
><br>
><br>
> I tried your suggestion but there is one confusion regarding point #3. Since<br>
> RancherOS has everything running as container, i am running<br>
> webcenter/rancher-glusterfs-<wbr>server container on all three nodes. Now as far<br>
> as removing the directories are concerned, i hope you meant removing them on<br>
> the host and _not_ from within the container. After completing step 1 and 2,<br>
> i checked the contents of all the directories that you specified in point<br>
> #3. All were empty as you can see in the attached other_logs.txt. So i got<br>
> confused. I ran the deploy again but the issue persists. Two pods show<br>
> Liveness error and the third one, Readiness error.<br>
><br>
> I then tried removing those directories (Step #3) from within the container<br>
> but getting following error:<br>
><br>
> root@c0f8ab4d92a2:/app# rm -rf /var/lib/heketi /etc/glusterfs<br>
> /var/lib/glusterd /var/log/glusterfs<br>
> rm: cannot remove '/var/lib/glusterd': Device or resource busy<br>
><br>
><br>
><br>
> On Fri, Sep 1, 2017 at 8:21 PM, Jose A. Rivera <<a href="mailto:jarrpa@redhat.com">jarrpa@redhat.com</a>> wrote:<br>
>><br>
>> 1. Add a line to the ssh-exec portion of heketi.json of the sort:<br>
>><br>
>> "sudo": true,<br>
>><br>
>> 2. Run<br>
>><br>
>> gk-deploy -g --abort<br>
>><br>
>> 3. On the nodes that were/will be running GlusterFS pods, run:<br>
>><br>
>> rm -rf /var/lib/heketi /etc/glusterfs /var/lib/glusterd /var/log/glusterfs<br>
>><br>
>> Then try the deploy again.<br>
>><br>
>> On Fri, Sep 1, 2017 at 6:05 AM, Gaurav Chhabra <<a href="mailto:varuag.chhabra@gmail.com">varuag.chhabra@gmail.com</a>><br>
>> wrote:<br>
>> > Hi Jose,<br>
>> ><br>
>> ><br>
>> > Thanks for the reply. It seems the three gluster pods might have been a<br>
>> > copy-paste from another set of cluster where i was trying to setup the<br>
>> > same<br>
>> > thing using CentOS. Sorry for that. By the way, i did check for the<br>
>> > kernel<br>
>> > modules and it seems it's already there. Also, i am attaching fresh set<br>
>> > of<br>
>> > files because i created a new cluster and thought of giving it a try<br>
>> > again.<br>
>> > Issue still persists. :(<br>
>> ><br>
>> > In heketi.json, there is a slight change w.r.t the user which connects<br>
>> > to<br>
>> > glusterfs node using SSH. I am not sure how Heketi was using root user<br>
>> > to<br>
>> > login because i wasn't able to use root and do manual SSH. With rancher<br>
>> > user, i can login successfully so i think this should be fine.<br>
>> ><br>
>> > /etc/heketi/heketi.json:<br>
>> > ------------------------------<wbr>------------------------------<wbr>------<br>
>> > "executor": "ssh",<br>
>> ><br>
>> > "_sshexec_comment": "SSH username and private key file information",<br>
>> > "sshexec": {<br>
>> > "keyfile": "/var/lib/heketi/.ssh/id_rsa",<br>
>> > "user": "rancher",<br>
>> > "port": "22",<br>
>> > "fstab": "/etc/fstab"<br>
>> > },<br>
>> > ------------------------------<wbr>------------------------------<wbr>------<br>
>> ><br>
>> > Before running gk-deploy:<br>
>> > ------------------------------<wbr>------------------------------<wbr>------<br>
>> > [root@workstation deploy]# kubectl get<br>
>> > nodes,pods,daemonset,<wbr>deployments,services<br>
>> > NAME STATUS AGE VERSION<br>
>> > no/node-a.c.kubernetes-174104.<wbr>internal Ready 3h<br>
>> > v1.7.2-rancher1<br>
>> > no/node-b.c.kubernetes-174104.<wbr>internal Ready 3h<br>
>> > v1.7.2-rancher1<br>
>> > no/node-c.c.kubernetes-174104.<wbr>internal Ready 3h<br>
>> > v1.7.2-rancher1<br>
>> ><br>
>> > NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE<br>
>> > svc/kubernetes 10.43.0.1 <none> 443/TCP 3h<br>
>> > ------------------------------<wbr>------------------------------<wbr>------<br>
>> ><br>
>> > After running gk-deploy:<br>
>> > ------------------------------<wbr>------------------------------<wbr>------<br>
>> > [root@workstation messagegc]# kubectl get<br>
>> > nodes,pods,daemonset,<wbr>deployments,services<br>
>> > NAME STATUS AGE VERSION<br>
>> > no/node-a.c.kubernetes-174104.<wbr>internal Ready 3h<br>
>> > v1.7.2-rancher1<br>
>> > no/node-b.c.kubernetes-174104.<wbr>internal Ready 3h<br>
>> > v1.7.2-rancher1<br>
>> > no/node-c.c.kubernetes-174104.<wbr>internal Ready 3h<br>
>> > v1.7.2-rancher1<br>
>> ><br>
>> > NAME READY STATUS RESTARTS AGE<br>
>> > po/glusterfs-0j9l5 0/1 Running 0 2m<br>
>> > po/glusterfs-gqz4c 0/1 Running 0 2m<br>
>> > po/glusterfs-gxvcb 0/1 Running 0 2m<br>
>> ><br>
>> > NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE<br>
>> > NODE-SELECTOR AGE<br>
>> > ds/glusterfs 3 3 0 3 0<br>
>> > storagenode=glusterfs 2m<br>
>> ><br>
>> > NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE<br>
>> > svc/kubernetes 10.43.0.1 <none> 443/TCP 3h<br>
>> > ------------------------------<wbr>------------------------------<wbr>------<br>
>> ><br>
>> > Kernel module check on all three nodes:<br>
>> > ------------------------------<wbr>------------------------------<wbr>------<br>
>> > [root@node-a ~]# find /lib*/modules/$(uname -r) -name *.ko | grep<br>
>> > 'thin-pool\|snapshot\|mirror' | xargs ls -ltr<br>
>> > -rw-r--r-- 1 root root 92310 Jun 26 04:13<br>
>> > /lib64/modules/4.9.34-rancher/<wbr>kernel/drivers/md/dm-thin-<wbr>pool.ko<br>
>> > -rw-r--r-- 1 root root 56982 Jun 26 04:13<br>
>> > /lib64/modules/4.9.34-rancher/<wbr>kernel/drivers/md/dm-snapshot.<wbr>ko<br>
>> > -rw-r--r-- 1 root root 27070 Jun 26 04:13<br>
>> > /lib64/modules/4.9.34-rancher/<wbr>kernel/drivers/md/dm-mirror.ko<br>
>> > -rw-r--r-- 1 root root 92310 Jun 26 04:13<br>
>> > /lib/modules/4.9.34-rancher/<wbr>kernel/drivers/md/dm-thin-<wbr>pool.ko<br>
>> > -rw-r--r-- 1 root root 56982 Jun 26 04:13<br>
>> > /lib/modules/4.9.34-rancher/<wbr>kernel/drivers/md/dm-snapshot.<wbr>ko<br>
>> > -rw-r--r-- 1 root root 27070 Jun 26 04:13<br>
>> > /lib/modules/4.9.34-rancher/<wbr>kernel/drivers/md/dm-mirror.ko<br>
>> > ------------------------------<wbr>------------------------------<wbr>------<br>
>> ><br>
>> > Error snapshot attached.<br>
>> ><br>
>> > In my first mail, i checked that Readiness Probe failure check has this<br>
>> > code<br>
>> > in kube-templates/glusterfs-<wbr>daemonset.yaml file:<br>
>> > ------------------------------<wbr>------------------------------<wbr>------<br>
>> > readinessProbe:<br>
>> > timeoutSeconds: 3<br>
>> > initialDelaySeconds: 40<br>
>> > exec:<br>
>> > command:<br>
>> > - "/bin/bash"<br>
>> > - "-c"<br>
>> > - systemctl status glusterd.service<br>
>> > periodSeconds: 25<br>
>> > successThreshold: 1<br>
>> > failureThreshold: 15<br>
>> > ------------------------------<wbr>------------------------------<wbr>------<br>
>> ><br>
>> > I tried logging into glustefs container on one of the node and ran the<br>
>> > above<br>
>> > command:<br>
>> ><br>
>> > [root@node-a ~]# docker exec -it c0f8ab4d92a23b6df2 /bin/bash<br>
>> > root@c0f8ab4d92a2:/app# systemctl status glusterd.service<br>
>> > WARNING: terminal is not fully functional<br>
>> > Failed to connect to bus: No such file or directory<br>
>> ><br>
>> ><br>
>> > Any check that i can do manually on nodes to debug further? Any<br>
>> > suggestions?<br>
>> ><br>
>> ><br>
>> > On Thu, Aug 31, 2017 at 6:53 PM, Jose A. Rivera <<a href="mailto:jarrpa@redhat.com">jarrpa@redhat.com</a>><br>
>> > wrote:<br>
>> >><br>
>> >> Hey Gaurav,<br>
>> >><br>
>> >> The kernel modules must be loaded on all nodes that will run heketi<br>
>> >> pods. Additionally, you must have at least three nodes specified in<br>
>> >> your topology file. I'm not sure how you're getting three gluster pods<br>
>> >> when you only have two nodes defined... :)<br>
>> >><br>
>> >> --Jose<br>
>> >><br>
>> >> On Wed, Aug 30, 2017 at 5:27 AM, Gaurav Chhabra<br>
>> >> <<a href="mailto:varuag.chhabra@gmail.com">varuag.chhabra@gmail.com</a>> wrote:<br>
>> >> > Hi,<br>
>> >> ><br>
>> >> ><br>
>> >> > I have the following setup in place:<br>
>> >> ><br>
>> >> > 1 node : RancherOS having Rancher application for Kubernetes setup<br>
>> >> > 2 nodes : RancherOS having Rancher agent<br>
>> >> > 1 node : CentOS 7 workstation having kubectl installed and folder<br>
>> >> > cloned/downloaded from <a href="https://github.com/gluster/gluster-kubernetes" rel="noreferrer" target="_blank">https://github.com/gluster/<wbr>gluster-kubernetes</a><br>
>> >> > using<br>
>> >> > which i run Heketi setup (gk-deploy -g)<br>
>> >> ><br>
>> >> > I also have rancher-glusterfs-server container running with the<br>
>> >> > following<br>
>> >> > configuration:<br>
>> >> > ------------------------------<wbr>------------------------------<wbr>------<br>
>> >> > [root@node-1 rancher]# cat gluster-server.sh<br>
>> >> > #!/bin/bash<br>
>> >> ><br>
>> >> > sudo docker run --name=gluster-server -d \<br>
>> >> > --env 'SERVICE_NAME=gluster' \<br>
>> >> > --restart always \<br>
>> >> > --env 'GLUSTER_DATA=/srv/docker/<wbr>gitlab' \<br>
>> >> > --publish 2222:22 \<br>
>> >> > webcenter/rancher-glusterfs-<wbr>server<br>
>> >> > ------------------------------<wbr>------------------------------<wbr>------<br>
>> >> ><br>
>> >> > In /etc/heketi/heketi.json, following is the only modified portion:<br>
>> >> > ------------------------------<wbr>------------------------------<wbr>------<br>
>> >> > "executor": "ssh",<br>
>> >> ><br>
>> >> > "_sshexec_comment": "SSH username and private key file<br>
>> >> > information",<br>
>> >> > "sshexec": {<br>
>> >> > "keyfile": "/var/lib/heketi/.ssh/id_rsa",<br>
>> >> > "user": "root",<br>
>> >> > "port": "22",<br>
>> >> > "fstab": "/etc/fstab"<br>
>> >> > },<br>
>> >> > ------------------------------<wbr>------------------------------<wbr>------<br>
>> >> ><br>
>> >> > Status before running gk-deploy:<br>
>> >> ><br>
>> >> > [root@workstation deploy]# kubectl get<br>
>> >> > nodes,pods,services,<wbr>deployments<br>
>> >> > NAME STATUS AGE VERSION<br>
>> >> > no/node-1.c.kubernetes-174104.<wbr>internal Ready 2d<br>
>> >> > v1.7.2-rancher1<br>
>> >> > no/node-2.c.kubernetes-174104.<wbr>internal Ready 2d<br>
>> >> > v1.7.2-rancher1<br>
>> >> > no/node-3.c.kubernetes-174104.<wbr>internal Ready 2d<br>
>> >> > v1.7.2-rancher1<br>
>> >> ><br>
>> >> > NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE<br>
>> >> > svc/kubernetes 10.43.0.1 <none> 443/TCP 2d<br>
>> >> ><br>
>> >> ><br>
>> >> > Now when i run 'gk-deploy -g', in the Rancher console, i see the<br>
>> >> > following<br>
>> >> > error:<br>
>> >> > Readiness probe failed: Failed to get D-Bus connection: Operation not<br>
>> >> > permitted<br>
>> >> ><br>
>> >> > From the attached gk-deploy_log i see that it failed at:<br>
>> >> > Waiting for GlusterFS pods to start ... pods not found.<br>
>> >> ><br>
>> >> > In the kube-templates/glusterfs-<wbr>daemonset.yaml file, i see this for<br>
>> >> > Readiness probe section:<br>
>> >> > ------------------------------<wbr>------------------------------<wbr>------<br>
>> >> > readinessProbe:<br>
>> >> > timeoutSeconds: 3<br>
>> >> > initialDelaySeconds: 40<br>
>> >> > exec:<br>
>> >> > command:<br>
>> >> > - "/bin/bash"<br>
>> >> > - "-c"<br>
>> >> > - systemctl status glusterd.service<br>
>> >> > periodSeconds: 25<br>
>> >> > successThreshold: 1<br>
>> >> > failureThreshold: 15<br>
>> >> > ------------------------------<wbr>------------------------------<wbr>------<br>
>> >> ><br>
>> >> ><br>
>> >> > Status after running gk-deploy:<br>
>> >> ><br>
>> >> > [root@workstation deploy]# kubectl get<br>
>> >> > nodes,pods,deployments,<wbr>services<br>
>> >> > NAME STATUS AGE VERSION<br>
>> >> > no/node-1.c.kubernetes-174104.<wbr>internal Ready 2d<br>
>> >> > v1.7.2-rancher1<br>
>> >> > no/node-2.c.kubernetes-174104.<wbr>internal Ready 2d<br>
>> >> > v1.7.2-rancher1<br>
>> >> > no/node-3.c.kubernetes-174104.<wbr>internal Ready 2d<br>
>> >> > v1.7.2-rancher1<br>
>> >> ><br>
>> >> > NAME READY STATUS RESTARTS AGE<br>
>> >> > po/glusterfs-0s440 0/1 Running 0 1m<br>
>> >> > po/glusterfs-j7dgr 0/1 Running 0 1m<br>
>> >> > po/glusterfs-p6jl3 0/1 Running 0 1m<br>
>> >> ><br>
>> >> > NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE<br>
>> >> > svc/kubernetes 10.43.0.1 <none> 443/TCP 2d<br>
>> >> ><br>
>> >> ><br>
>> >> > Also, from prerequisite perspective, i was also seeing this<br>
>> >> > mentioned:<br>
>> >> ><br>
>> >> > The following kernel modules must be loaded:<br>
>> >> > * dm_snapshot<br>
>> >> > * dm_mirror<br>
>> >> > * dm_thin_pool<br>
>> >> ><br>
>> >> > Where exactly is this to be checked? On all Gluster server nodes? How<br>
>> >> > can i<br>
>> >> > check whether it's there?<br>
>> >> ><br>
>> >> > I have attached topology.json and gk-deploy log for reference.<br>
>> >> ><br>
>> >> > Does this issue has anything to do with the host OS (RancherOS) that<br>
>> >> > i<br>
>> >> > am<br>
>> >> > using for Gluster nodes? Any idea how i can fix this? Any help will<br>
>> >> > really<br>
>> >> > be appreciated.<br>
>> >> ><br>
>> >> ><br>
>> >> > Thanks.<br>
>> >> ><br>
>> >> ><br>
>> >> ><br>
>> >> > ______________________________<wbr>_________________<br>
>> >> > heketi-devel mailing list<br>
>> >> > <a href="mailto:heketi-devel@gluster.org">heketi-devel@gluster.org</a><br>
>> >> > <a href="http://lists.gluster.org/mailman/listinfo/heketi-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/heketi-devel</a><br>
>> >> ><br>
>> ><br>
>> ><br>
><br>
><br>
</div></div></blockquote></div><br></div>