<div dir="ltr"><br><div>Hi all,</div><div><br></div><div>I&#39;ve just tried Prometheus on Kubernetes, using</div><div>- CoreOS Prometheus Operator (0.10.2)</div><div>- K8s 1.6.8</div><div>- GlusterFS 3.10.4 (disabled all performance translators)</div><div>- Heketi 5</div><div><br></div><div>The Operator created a StatefulSet with a VolumeClaimTemplate prometheus-prometheus-db-prometheus-prometheus-0</div><div>As you can see below, the claim has issues creating volumes. </div><div>But actually, the volumes are created. LoggedIn in my GlusterFS Container I can see f.e. vol_9903142828b2954544430d0f9dd385bf but not in Heketi</div><div><br></div><div>Any idea how this can happen?</div><div><br></div><div>best Christopher</div><div><br></div><div> </div><div><br></div><div><br></div><div><div><font face="monospace">&gt; kubectl -n metrics describe pvc prometheus-prometheus-db-prometheus-prometheus-0</font></div><div><font face="monospace">Name:<span style="white-space:pre">                </span>prometheus-prometheus-db-prometheus-prometheus-0</font></div><div><font face="monospace">Namespace:<span style="white-space:pre">        </span>metrics</font></div><div><font face="monospace">StorageClass:<span style="white-space:pre">        </span>distributed</font></div><div><font face="monospace">Status:<span style="white-space:pre">                </span>Pending</font></div><div><font face="monospace">Volume:</font></div><div><font face="monospace">Labels:<span style="white-space:pre">                </span>app=prometheus</font></div><div><font face="monospace"><span style="white-space:pre">                </span>prometheus=prometheus</font></div><div><font face="monospace">Annotations:<span style="white-space:pre">        </span><a href="http://volume.beta.kubernetes.io/storage-class=distributed">volume.beta.kubernetes.io/storage-class=distributed</a></font></div><div><font face="monospace"><span style="white-space:pre">                </span><a href="http://volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs">volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs</a></font></div><div><font face="monospace">Capacity:</font></div><div><font face="monospace">Access Modes:</font></div><div><font face="monospace">Events:</font></div><div><font face="monospace">  FirstSeen<span style="white-space:pre">        </span>LastSeen<span style="white-space:pre">        </span>Count<span style="white-space:pre">        </span>From<span style="white-space:pre">                                </span>SubObjectPath<span style="white-space:pre">        </span>Type<span style="white-space:pre">                </span>Reason<span style="white-space:pre">                        </span>Message</font></div><div><font face="monospace">  ---------<span style="white-space:pre">        </span>--------<span style="white-space:pre">        </span>-----<span style="white-space:pre">        </span>----<span style="white-space:pre">                                </span>-------------<span style="white-space:pre">        </span>--------<span style="white-space:pre">        </span>------<span style="white-space:pre">                        </span>-------</font></div><div><font face="monospace">  39s<span style="white-space:pre">                </span>39s<span style="white-space:pre">                </span>1<span style="white-space:pre">        </span>persistentvolume-controller<span style="white-space:pre">                        </span>Warning<span style="white-space:pre">                </span>ProvisioningFailed<span style="white-space:pre">        </span>Failed to provision volume with StorageClass &quot;distributed&quot;: glusterfs: create volume err: error creating volume Unable to execute command on glusterfs-vl1tl: volume start: vol_9903142828b2954544430d0f9dd385bf: failed: Another transaction is in progress Please try again after sometime..</font></div><div><font face="monospace">  28s<span style="white-space:pre">                </span>28s<span style="white-space:pre">                </span>1<span style="white-space:pre">        </span>persistentvolume-controller<span style="white-space:pre">                        </span>Warning<span style="white-space:pre">                </span>ProvisioningFailed<span style="white-space:pre">        </span>Failed to provision volume with StorageClass &quot;distributed&quot;: glusterfs: create volume err: failed to create endpoint/service &lt;nil&gt;.</font></div><div><font face="monospace">  15s<span style="white-space:pre">                </span>15s<span style="white-space:pre">                </span>1<span style="white-space:pre">        </span>persistentvolume-controller<span style="white-space:pre">                        </span>Warning<span style="white-space:pre">                </span>ProvisioningFailed<span style="white-space:pre">        </span>Failed to provision volume with StorageClass &quot;distributed&quot;: glusterfs: create volume err: error creating volume Unable to execute command on glusterfs-vl1tl: volume start: vol_f4e20daae202e6190085fd4bbbe837bb: failed: Another transaction is in progress Please try again after sometime..</font></div><div><font face="monospace">  0s<span style="white-space:pre">                </span>0s<span style="white-space:pre">                </span>1<span style="white-space:pre">        </span>persistentvolume-controller<span style="white-space:pre">                        </span>Warning<span style="white-space:pre">                </span>ProvisioningFailed<span style="white-space:pre">        </span>Failed to provision volume with StorageClass &quot;distributed&quot;: glusterfs: create volume err: error creating volume Unable to execute command on glusterfs-4sl4b: volume start: vol_8c913f988caa2328f57cc37e164d7d80: failed: Another transaction is in progress Please try again after sometime..</font></div></div></div>