<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(36, 41, 46); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px; margin-top: 0px !important;" class="">
Hi</p>
<p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(36, 41, 46); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px; margin-top: 0px !important;" class="">
I seem to have developed a strange problem with my heketi/glusterfs storage cluster. It’s been running perfectly fine with a couple of weeks now in our production k8s cluster. But since yesterday we cannot create any new pvc’s</p>
<pre style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 11.899999618530273px; margin-top: 0px; margin-bottom: 16px; word-wrap: normal; padding: 16px; overflow: auto; line-height: 1.45; background-color: rgb(246, 248, 250); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px; color: rgb(36, 41, 46);" class=""><code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; padding: 0px; margin: 0px; background-color: transparent; border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px; word-break: normal; border: 0px; display: inline; overflow: visible; line-height: inherit; word-wrap: normal;" class="">kubectl describe pvc filesystem-example
Name: filesystem-example
Namespace: default
StorageClass: filesystem-storage
Status: Pending
Volume:
Labels: <none>
Annotations: <a href="http://volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs" class="">volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs</a>
Capacity:
Access Modes:
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
33s 33s 1 persistentvolume-controller Warning ProvisioningFailed Failed to provision volume with StorageClass "filesystem-storage": glusterfs: create volume err: error creating volume Post <a href="http://172.18.6.63:8080/volumes:" class="">http://172.18.6.63:8080/volumes:</a> dial tcp 172.18.6.63:8080: i/o timeout
</code></pre>
<p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; color: rgb(36, 41, 46); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px;" class="">
the <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 11.899999618530273px; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.0470588); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px;" class=""><a href="http://172.18.6.63:8080" class="">http://172.18.6.63:8080</a></code> being
the heketi endpoint<br style="box-sizing: border-box;" class="">
the strange thing is, if I port forward my heketi pod <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 11.899999618530273px; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.0470588); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px;" class="">kubectl
port-forward heketi-4196954491-7v60x :8080</code><br style="box-sizing: border-box;" class="">
I can use heketi-cli and get all my topology information etc. etc. and everything looks fine<br style="box-sizing: border-box;" class="">
and using a pod I use for testing, I can actually reach the endpoint as well and list the volumes</p>
<pre style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 11.899999618530273px; margin-top: 0px; margin-bottom: 16px; word-wrap: normal; padding: 16px; overflow: auto; line-height: 1.45; background-color: rgb(246, 248, 250); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px; color: rgb(36, 41, 46);" class=""><code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; padding: 0px; margin: 0px; background-color: transparent; border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px; word-break: normal; border: 0px; display: inline; overflow: visible; line-height: inherit; word-wrap: normal;" class="">kubectl exec net-test-617505214-4kg8r curl <a href="http://172.18.6.63:8080/volumes" class="">http://172.18.6.63:8080/volumes</a>
{"volumes": ["05ae72950de2f3bbcf4e07c1c94967aa","0cf6f91d5efa2f0b370ac920cadfd8d1","1320ec81d9b4a764c864c29b96fcb008","13a6ea425c5b648ec595a4d061aba094","1c296c5a9202ba28ea0b8b49b4af95ca","1d10be7e0b01c743b3e9f76249064701","1ffa2309a04a8ce506a1f40a5290d6a5","213cfa6aedab8d46782fab518a632639","2319d77b8a0ba0dd9b4e67b0b7c33e56","2319dcf6ba7eb73c7cfba307c835c636","27577afc14af878f80638e0bd95a21d4","2809215127febc8faa062bf8b361884a","334b572c54b620327b76d40a99c64523","38997090b0b840fcb230e891cc934698","3a746c89362f449597fcb90cbf1ccb76","3af01151dbe1073d0d7181e1a7959c2f","4cecc1147cbf5bbe0af0bfae01e0080f","4fca55ff90dc2e75cd8bd9c905a9e32f","55c68c2fc9e20d47d933ab98e94f8d60","55f917143f6e258f48bbc9715698081b","57d081ab0aaccc6c0b5efb337a0be9bd","5b85941c288bc676cf906a619870816c","5e2ee42507040e39730df54aadf2645a","5f2135f3417a0aa423fdb7a7513c2beb","624726d3fe79d06d06045a8dcdc8ab40","6bc106dbc01bd82bd48110040b3c9f10","7fc6771e5b235656b3ac69f1cca798d8","83cfe458eace5668664a664164cf2845","8489f04bbba0d43fd54693b6b7a0a3f4","875483d8362afc484df37b81e466b025","8b4851b0f88492d0a4409658c2d0500e","8dd7ff72a7cce6770cdcf17bc2376591","95c8aa1ec430a0dd1622020aa52adaae","a41c8cb381e1c614b36f8b7e849d95ac","a5ea04607df6aa96a5298c6eee5f3cd3","aa329752ed5e7a519aacfc1f82ea5d4d","ac0385316a46a6ba61b8c66b089401c1","ad7102054304831d6107e87908677261","b2b1ec297a99d5dfa15bf20f9ae4527e","b68678aa5a720b8d388fc4193c224889","b8c553da460e2f1f1ccfa7d128f638dc","c33631dcc26d6cf6dd95de06c0639d06","c4133fd40a4548e9cb61ea074fc86dc3","c84ad4ca2c671fa5642d3015e67afdf9","cae88832e26e82e987c4ee52b5bc5bc3","ce05ad2a80ba36548562368e4020fd2e","d417b634dd9685fb72b59055d2f87c90","d6bc8c960113d9266b6b0cc4831a620b","d825cfab9d20a931b5a3fc6338862122","dc218319892fbc7966228dc9f2fbdb79","dc322753948cc8478b549fe4d559802d","e8ac20c47304dd3be6aea37086960a4a","ea8782ce72c764cd91804d8c8a9ca14c","eab4523323f6639c227d3e03cd4795dd","eb3541b001ae96585c8183a5123cfca2","ed4d641ab5f06c1addc6b99e1b8249bd","eec206d15df1249e00eb312311434251","f067593418214b1e6137b46a3613e2a7","fa4174151cb7a85480428a1ac71cc5a4","fb1c2bdb6d627f37e06f1da5049608e2"]}
</code></pre>
<div style="box-sizing: border-box; margin-top: 0px; color: rgb(36, 41, 46); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px; margin-bottom: 0px !important;" class="">
does anyone have any idea what might be wrong?<br style="box-sizing: border-box;" class="">
looking on the cluster side as well, running <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 11.899999618530273px; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.0470588); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px;" class="">kubectl
exec -it glusterfs-81qhx gluster</code> and then <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 11.899999618530273px; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.0470588); border-top-left-radius: 3px; border-top-right-radius: 3px; border-bottom-right-radius: 3px; border-bottom-left-radius: 3px;" class="">volume
info all</code> I get all my volumes with bricks back as well and it looks good</div>
<div style="box-sizing: border-box; margin-top: 0px; color: rgb(36, 41, 46); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px; margin-bottom: 0px !important;" class="">
<br class="">
</div>
<div style="box-sizing: border-box; margin-top: 0px; color: rgb(36, 41, 46); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 14px; margin-bottom: 0px !important;" class="">
Creating a volume using heketi-cli works works perfectly.</div>
</body>
</html>