[heketi-devel] heketi endpoint I/O timeout

[werner.gillmer at kpn.com]

Hi

I seem to have developed a strange problem with my heketi/glusterfs storage cluster. It’s been running perfectly fine with a couple of weeks now in our production k8s cluster. But since yesterday we cannot create any new pvc’s

kubectl describe pvc filesystem-example
Name:       filesystem-example
Namespace:  default
StorageClass:   filesystem-storage
Status:     Pending
Volume:
Labels:     <none>
Annotations:    volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs<http://volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs>
Capacity:
Access Modes:
Events:
  FirstSeen LastSeen    Count   From                SubObjectPath   Type        Reason          Message
  --------- --------    -----   ----                -------------   --------    ------          -------
  33s       33s     1   persistentvolume-controller         Warning     ProvisioningFailed  Failed to provision    volume with StorageClass "filesystem-storage": glusterfs: create volume err: error creating volume   Post http://172.18.6.63:8080/volumes: dial tcp 172.18.6.63:8080: i/o timeout


the http://172.18.6.63:8080 being the heketi endpoint
the strange thing is, if I port forward my heketi pod kubectl port-forward heketi-4196954491-7v60x :8080
I can use heketi-cli and get all my topology information etc. etc. and everything looks fine
and using a pod I use for testing, I can actually reach the endpoint as well and list the volumes

kubectl exec net-test-617505214-4kg8r curl http://172.18.6.63:8080/volumes
{"volumes": ["05ae72950de2f3bbcf4e07c1c94967aa","0cf6f91d5efa2f0b370ac920cadfd8d1","1320ec81d9b4a764c864c29b96fcb008","13a6ea425c5b648ec595a4d061aba094","1c296c5a9202ba28ea0b8b49b4af95ca","1d10be7e0b01c743b3e9f76249064701","1ffa2309a04a8ce506a1f40a5290d6a5","213cfa6aedab8d46782fab518a632639","2319d77b8a0ba0dd9b4e67b0b7c33e56","2319dcf6ba7eb73c7cfba307c835c636","27577afc14af878f80638e0bd95a21d4","2809215127febc8faa062bf8b361884a","334b572c54b620327b76d40a99c64523","38997090b0b840fcb230e891cc934698","3a746c89362f449597fcb90cbf1ccb76","3af01151dbe1073d0d7181e1a7959c2f","4cecc1147cbf5bbe0af0bfae01e0080f","4fca55ff90dc2e75cd8bd9c905a9e32f","55c68c2fc9e20d47d933ab98e94f8d60","55f917143f6e258f48bbc9715698081b","57d081ab0aaccc6c0b5efb337a0be9bd","5b85941c288bc676cf906a619870816c","5e2ee42507040e39730df54aadf2645a","5f2135f3417a0aa423fdb7a7513c2beb","624726d3fe79d06d06045a8dcdc8ab40","6bc106dbc01bd82bd48110040b3c9f10","7fc6771e5b235656b3ac69f1cca798d8","83cfe458eace5668664a664164cf2845","8489f04bbba0d43fd54693b6b7a0a3f4","875483d8362afc484df37b81e466b025","8b4851b0f88492d0a4409658c2d0500e","8dd7ff72a7cce6770cdcf17bc2376591","95c8aa1ec430a0dd1622020aa52adaae","a41c8cb381e1c614b36f8b7e849d95ac","a5ea04607df6aa96a5298c6eee5f3cd3","aa329752ed5e7a519aacfc1f82ea5d4d","ac0385316a46a6ba61b8c66b089401c1","ad7102054304831d6107e87908677261","b2b1ec297a99d5dfa15bf20f9ae4527e","b68678aa5a720b8d388fc4193c224889","b8c553da460e2f1f1ccfa7d128f638dc","c33631dcc26d6cf6dd95de06c0639d06","c4133fd40a4548e9cb61ea074fc86dc3","c84ad4ca2c671fa5642d3015e67afdf9","cae88832e26e82e987c4ee52b5bc5bc3","ce05ad2a80ba36548562368e4020fd2e","d417b634dd9685fb72b59055d2f87c90","d6bc8c960113d9266b6b0cc4831a620b","d825cfab9d20a931b5a3fc6338862122","dc218319892fbc7966228dc9f2fbdb79","dc322753948cc8478b549fe4d559802d","e8ac20c47304dd3be6aea37086960a4a","ea8782ce72c764cd91804d8c8a9ca14c","eab4523323f6639c227d3e03cd4795dd","eb3541b001ae96585c8183a5123cfca2","ed4d641ab5f06c1addc6b99e1b8249bd","eec206d15df1249e00eb312311434251","f067593418214b1e6137b46a3613e2a7","fa4174151cb7a85480428a1ac71cc5a4","fb1c2bdb6d627f37e06f1da5049608e2"]}


does anyone have any idea what might be wrong?
looking on the cluster side as well, running kubectl exec -it glusterfs-81qhx gluster and then volume info all I get all my volumes with bricks back as well and it looks good

Creating a volume using heketi-cli works works perfectly.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/heketi-devel/attachments/20171123/dc1889ce/attachment.html>