[heketi-devel] Strange behaviour with Kubernetes StatefulSet and dynamic provisioning

[Christopher Schmidt]

Hi all,

I've just tried Prometheus on Kubernetes, using
- CoreOS Prometheus Operator (0.10.2)
- K8s 1.6.8
- GlusterFS 3.10.4 (disabled all performance translators)
- Heketi 5

The Operator created a StatefulSet with a
VolumeClaimTemplate prometheus-prometheus-db-prometheus-prometheus-0
As you can see below, the claim has issues creating volumes.
But actually, the volumes are created. LoggedIn in my GlusterFS Container I
can see f.e. vol_9903142828b2954544430d0f9dd385bf but not in Heketi

Any idea how this can happen?

best Christopher




> kubectl -n metrics describe pvc
prometheus-prometheus-db-prometheus-prometheus-0
Name: prometheus-prometheus-db-prometheus-prometheus-0
Namespace: metrics
StorageClass: distributed
Status: Pending
Volume:
Labels: app=prometheus
prometheus=prometheus
Annotations: volume.beta.kubernetes.io/storage-class=distributed
volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs
Capacity:
Access Modes:
Events:
  FirstSeen LastSeen Count From SubObjectPath Type Reason Message
  --------- -------- ----- ---- ------------- -------- ------ -------
  39s 39s 1 persistentvolume-controller Warning ProvisioningFailed Failed
to provision volume with StorageClass "distributed": glusterfs: create
volume err: error creating volume Unable to execute command on
glusterfs-vl1tl: volume start: vol_9903142828b2954544430d0f9dd385bf:
failed: Another transaction is in progress Please try again after sometime..
  28s 28s 1 persistentvolume-controller Warning ProvisioningFailed Failed
to provision volume with StorageClass "distributed": glusterfs: create
volume err: failed to create endpoint/service <nil>.
  15s 15s 1 persistentvolume-controller Warning ProvisioningFailed Failed
to provision volume with StorageClass "distributed": glusterfs: create
volume err: error creating volume Unable to execute command on
glusterfs-vl1tl: volume start: vol_f4e20daae202e6190085fd4bbbe837bb:
failed: Another transaction is in progress Please try again after sometime..
  0s 0s 1 persistentvolume-controller Warning ProvisioningFailed Failed to
provision volume with StorageClass "distributed": glusterfs: create volume
err: error creating volume Unable to execute command on glusterfs-4sl4b:
volume start: vol_8c913f988caa2328f57cc37e164d7d80: failed: Another
transaction is in progress Please try again after sometime..
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/heketi-devel/attachments/20170817/311e65e0/attachment.html>