[heketi-devel] are there any examples of running heketi as a pod in kubernetes with non-containerized gluster?
Scott Creeley
screeley at redhat.com
Thu Mar 9 18:49:33 UTC 2017
thanks Raghavendra...
Yeah, that is what I'm missing, I have misconfigured something in my cluster/pod/network to be able to establish the correct ssh connection (sshexec), but I know my pod can reach the external gluster cluster otherwise because it has created and successfully written to and read from the heketi.db file which lives on a volume in my external gluster cluster as you can see from within my heketi pod the mount to the gluster system.
[root at ocpdev openshift]# oc exec -ti deploy-heketi-1-p7z87 /bin/sh
sh-4.2$ mount
...
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,context="system_u:object_r:svirt_sandbox_file_t:s0:c51,c134",size=65536k)
192.168.1.205:vol_1b2b344fa375ea17a403e35c3f91dc89 on /var/lib/heketi type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
tmpfs on /run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime,seclabel)
...
The logs explicitly tell me that is is failing on the ssh connection, do I need a custom image for heketi that also include sshd installed?
[root at ocpdev openshift]# oc logs deploy-heketi-1-p7z87
Heketi 3.1.0
[heketi] INFO 2017/03/09 18:29:19 Loaded ssh executor
[heketi] INFO 2017/03/09 18:29:19 Loaded simple allocator
[heketi] INFO 2017/03/09 18:29:19 GlusterFS Application Loaded
Listening on port 8080
[negroni] Started GET /clusters
[negroni] Completed 200 OK in 152.315µs
[negroni] Started GET /clusters/d2ff22f516c76f9c3e2e60af0cd97dd2
[negroni] Completed 200 OK in 206.196µs
[negroni] Started GET /nodes/01aeeab788f5975d4c5d38a2b4f2c217
[negroni] Completed 200 OK in 238.99µs
[negroni] Started POST /clusters
[negroni] Completed 201 Created in 14.180427ms
[negroni] Started POST /nodes
[negroni] Completed 409 Conflict in 2.018839ms
[negroni] Started POST /nodes
[heketi] INFO 2017/03/09 18:29:30 Adding node 192.168.1.206
[negroni] Completed 202 Accepted in 8.437315ms
[asynchttp] INFO 2017/03/09 18:29:30 Started job b0424f65a0b64fd48683122d02dddf02
[negroni] Started GET /queue/b0424f65a0b64fd48683122d02dddf02
[negroni] Completed 200 OK in 23.59µs
[heketi] INFO 2017/03/09 18:29:30 Added node 297357eb7ec5e23216e1d6e723fe7722
[asynchttp] INFO 2017/03/09 18:29:30 Completed job b0424f65a0b64fd48683122d02dddf02 in 15.200443ms
[negroni] Started GET /queue/b0424f65a0b64fd48683122d02dddf02
[negroni] Completed 303 See Other in 97.969µs
[negroni] Started GET /nodes/297357eb7ec5e23216e1d6e723fe7722
[negroni] Completed 200 OK in 5.190295ms
[negroni] Started POST /devices
[heketi] INFO 2017/03/09 18:29:30 Adding device /dev/sdd to node 297357eb7ec5e23216e1d6e723fe7722
[negroni] Completed 202 Accepted in 9.833559ms
[asynchttp] INFO 2017/03/09 18:29:30 Started job 0523bb542a80223ae0838515687e6034
[negroni] Started GET /queue/0523bb542a80223ae0838515687e6034
[negroni] Completed 200 OK in 20.401µs
[negroni] Started GET /queue/0523bb542a80223ae0838515687e6034
[negroni] Completed 200 OK in 29.571µs
[negroni] Started GET /queue/0523bb542a80223ae0838515687e6034
[negroni] Completed 200 OK in 67.755µs
[negroni] Started GET /queue/0523bb542a80223ae0838515687e6034
[negroni] Completed 200 OK in 71.391µs
[negroni] Completed 200 OK in 33.227µs
[negroni] Started GET /queue/0523bb542a80223ae0838515687e6034
[negroni] Completed 200 OK in 71.524µs
[negroni] Started GET /queue/0523bb542a80223ae0838515687e6034
[negroni] Completed 200 OK in 73.868µs
[negroni] Started GET /queue/0523bb542a80223ae0838515687e6034
[negroni] Completed 200 OK in 72.156µs
[negroni] Started GET /queue/0523bb542a80223ae0838515687e6034
[negroni] Completed 200 OK in 69.434µs
[sshexec] WARNING 2017/03/09 18:31:37 Failed to create SSH connection to 192.168.1.206:22: dial tcp 192.168.1.206:22: getsockopt: connection timed out
[asynchttp] INFO 2017/03/09 18:31:37 Completed job 0523bb542a80223ae0838515687e6034 in 2m7.196555102s
[negroni] Started GET /queue/0523bb542a80223ae0838515687e6034
[negroni] Completed 500 Internal Server Error in 2.701149ms
[negroni] Started GET /clusters/daa03c7403609c314ade2cc44a07657f
[negroni] Completed 200 OK in 1.849512ms
my topology that did get loaded:
[root at ocpdev openshift]# oc exec -ti deploy-heketi-1-prxdl /bin/sh
sh-4.2$ heketi-cli --server http://localhost:8080 topology info
Cluster Id: d2ff22f516c76f9c3e2e60af0cd97dd2
Volumes:
Nodes:
Node Id: 01aeeab788f5975d4c5d38a2b4f2c217
State: online
Cluster Id: d2ff22f516c76f9c3e2e60af0cd97dd2
Zone: 1
Management Hostname: 192.168.1.205
Storage Hostname: 192.168.1.205
Devices:
Cluster Id: daa03c7403609c314ade2cc44a07657f
Volumes:
Nodes:
Node Id: 297357eb7ec5e23216e1d6e723fe7722
State: online
Cluster Id: daa03c7403609c314ade2cc44a07657f
Zone: 1
Management Hostname: 192.168.1.206
Storage Hostname: 192.168.1.206
Devices:
and from my host (running the cluster), I can telnet to port 22 on my router, the same router that allows me to access my deploy-heketi pod...any idea what I'm missing?
[root at ocpdev heketi]# telnet deploy-heketi-default.router.default.svc.cluster.local 22
Trying 192.168.1.230...
Connected to deploy-heketi-default.router.default.svc.cluster.local.
Escape character is '^]'.
SSH-2.0-OpenSSH_6.6.1
^]
telnet> quit
Connection closed.
[root at ocpdev heketi]# curl http://deploy-heketi-default.router.default.svc.cluster.local/hello
Hello from Heketi
----- Original Message -----
From: "Raghavendra Talur" <rtalur at redhat.com>
To: "Scott Creeley" <screeley at redhat.com>
Cc: heketi-devel at gluster.org
Sent: Thursday, March 9, 2017 11:23:35 AM
Subject: Re: [heketi-devel] are there any examples of running heketi as a pod in kubernetes with non-containerized gluster?
On Thu, Mar 9, 2017 at 9:48 PM, Scott Creeley <screeley at redhat.com> wrote:
> Hi All,
> Are there any examples of running heketi as a pod in kubernetes with non-containerized gluster?
This should work as long as heketi pod can reach the nodes where
Gluster is running. Also you will have to use sshexec with the ssh
keys setup.
>
> I was able to start heketi as a pod in kubernetes and start writing the heketi.db to my external gluster cluster (non-containerized) but it fails on loading the topology most likely due to failed ssh connection to the gluster cluster from my heketi pod. Missing a network piece I'm sure, but not sure what other logs to look at to help?
>
> [root at ocpdev heketi]# oc logs deploy-heketi-1-sx1fw
> Heketi 3.1.0
> [heketi] INFO 2017/03/08 23:23:50 Loaded ssh executor
> [heketi] WARNING 2017/03/08 23:23:53 Unable to open database. Retrying using read only mode
NOTE this ^^^^^
Your db file isn't opened for writing and that is also a problem.
Please check the heketi.json file and determine where db is located.
Make sure heketi pod has write access to it.
Thanks,
Raghavendra Talur
> [heketi] INFO 2017/03/08 23:23:53 Loaded simple allocator
> [heketi] INFO 2017/03/08 23:23:53 GlusterFS Application Loaded
> Listening on port 8080
> [negroni] Started GET /clusters
> [negroni] Completed 200 OK in 250.381µs
> [negroni] Started GET /clusters/d2ff22f516c76f9c3e2e60af0cd97dd2
> [negroni] Completed 200 OK in 466.435µs
> [negroni] Started GET /nodes/01aeeab788f5975d4c5d38a2b4f2c217
> [negroni] Completed 200 OK in 534.738µs
> [negroni] Started GET /clusters/d2ff22f516c76f9c3e2e60af0cd97dd2
> [negroni] Completed 200 OK in 228.541µs
> [negroni] Started POST /devices
> [negroni] Completed 0 in 2.166515ms
> [negroni] Started POST /nodes
> [negroni] Completed 0 in 110.326µs
>
>
> [root at ocpdev ~]# oc exec -ti deploy-heketi-1-sx1fw /bin/sh
> sh-4.2$ heketi-cli --server http://localhost:8080 topology info
>
> Cluster Id: d2ff22f516c76f9c3e2e60af0cd97dd2
>
> Volumes:
>
> Nodes:
>
> Node Id: 01aeeab788f5975d4c5d38a2b4f2c217
> State: online
> Cluster Id: d2ff22f516c76f9c3e2e60af0cd97dd2
> Zone: 1
> Management Hostname: gluster25.rhs
> Storage Hostname: 192.168.1.205
> Devices:
>
>
>
> thanks,
> Scott
> _______________________________________________
> heketi-devel mailing list
> heketi-devel at gluster.org
> http://lists.gluster.org/mailman/listinfo/heketi-devel
More information about the heketi-devel
mailing list