<div dir="ltr"><div>Hi Christopher,<br><br></div><div>I believe this is what happened here:<br><br></div><div>heketi placed a request to gluster to create a volume which was succesful, once its done, heketi will issue volume start command and this command failed.  For heketi, this is &#39;one transaction&#39; which is volume creation and it &quot;failed&quot;, so it would have tried to &quot;delete the volume&quot; ( this command would have failed in gluster too) created just now and will also wipe the DB entry from heketi. <br><br></div><div>Thats the reason for heketi not aware of this volume atm and volume still exist in the backend. <br></div><div><div><div><div class="gmail_extra"><br></div><div class="gmail_extra">From the gluster error, it looks like &quot;glusterd&quot; is  busy atm to serve the request: &quot;<font face="monospace"> Another transaction is in progress Please try again after sometime..<span class="">&quot;</span><br><br></font></div><div class="gmail_extra"><font face="monospace">I dont know what caused this ( may be many requests concurrent requests in glustrd or slowness in executing gluster commands..etc) and the setup much. Yeah, gluster and heketi logs could tell. <br><br></font></div><div class="gmail_extra"><font face="monospace"><br></font></div><div class="gmail_extra"><font face="monospace"><br><br></font></div><div class="gmail_extra"><div class="gmail_quote">On Fri, Aug 18, 2017 at 11:55 AM, Luis Pabon <span dir="ltr">&lt;<a href="mailto:lpabon@chrysalix.org" target="_blank">lpabon@chrysalix.org</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">That is strange. I wonder if the volume creation was successful. Can show attach the logs for Heketi?<div><br></div><div>- Luis</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="gmail-h5">On Thu, Aug 17, 2017 at 8:31 AM, Christopher Schmidt <span dir="ltr">&lt;<a href="mailto:fakod666@gmail.com" target="_blank">fakod666@gmail.com</a>&gt;</span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-h5"><div dir="ltr"><br><div>Hi all,</div><div><br></div><div>I&#39;ve just tried Prometheus on Kubernetes, using</div><div>- CoreOS Prometheus Operator (0.10.2)</div><div>- K8s 1.6.8</div><div>- GlusterFS 3.10.4 (disabled all performance translators)</div><div>- Heketi 5</div><div><br></div><div>The Operator created a StatefulSet with a VolumeClaimTemplate prometheus<wbr>-prometheus-db-prometheus-<wbr>prometheus-0</div><div>As you can see below, the claim has issues creating volumes. </div><div>But actually, the volumes are created. LoggedIn in my GlusterFS Container I can see f.e. vol_9903142828b2954544430d0f9d<wbr>d385bf but not in Heketi</div><div><br></div><div>Any idea how this can happen?</div><div><br></div><div>best Christopher</div><div><br></div><div> </div><div><br></div><div><br></div><div><div><font face="monospace">&gt; kubectl -n metrics describe pvc prometheus-prometheus-db-prome<wbr>theus-prometheus-0</font></div><div><font face="monospace">Name:<span style="white-space:pre-wrap">                </span>prometheus-prometheus-db-prome<wbr>theus-prometheus-0</font></div><div><font face="monospace">Namespace:<span style="white-space:pre-wrap">        </span>metrics</font></div><div><font face="monospace">StorageClass:<span style="white-space:pre-wrap">        </span>distributed</font></div><div><font face="monospace">Status:<span style="white-space:pre-wrap">                </span>Pending</font></div><div><font face="monospace">Volume:</font></div><div><font face="monospace">Labels:<span style="white-space:pre-wrap">                </span>app=prometheus</font></div><div><font face="monospace"><span style="white-space:pre-wrap">                </span>prometheus=prometheus</font></div><div><font face="monospace">Annotations:<span style="white-space:pre-wrap">        </span><a href="http://volume.beta.kubernetes.io/storage-class=distributed" target="_blank">volume.beta.kubernetes.io/stor<wbr>age-class=distributed</a></font></div><div><font face="monospace"><span style="white-space:pre-wrap">                </span><a href="http://volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs" target="_blank">volume.beta.kubernetes.io/stor<wbr>age-provisioner=kubernetes.io/<wbr>glusterfs</a></font></div><div><font face="monospace">Capacity:</font></div><div><font face="monospace">Access Modes:</font></div><div><font face="monospace">Events:</font></div><div><font face="monospace">  FirstSeen<span style="white-space:pre-wrap">        </span>LastSeen<span style="white-space:pre-wrap">        </span>Count<span style="white-space:pre-wrap">        </span>From<span style="white-space:pre-wrap">                                </span>SubObjectPath<span style="white-space:pre-wrap">        </span>Type<span style="white-space:pre-wrap">                </span>Reason<span style="white-space:pre-wrap">                        </span>Message</font></div><div><font face="monospace">  ---------<span style="white-space:pre-wrap">        </span>--------<span style="white-space:pre-wrap">        </span>-----<span style="white-space:pre-wrap">        </span>----<span style="white-space:pre-wrap">                                </span>-------------<span style="white-space:pre-wrap">        </span>--------<span style="white-space:pre-wrap">        </span>------<span style="white-space:pre-wrap">                        </span>-------</font></div><div><font face="monospace">  39s<span style="white-space:pre-wrap">                </span>39s<span style="white-space:pre-wrap">                </span>1<span style="white-space:pre-wrap">        </span>persistentvolume-controller<span style="white-space:pre-wrap">                        </span>Warning<span style="white-space:pre-wrap">                </span>ProvisioningFailed<span style="white-space:pre-wrap">        </span>Failed to provision volume with StorageClass &quot;distributed&quot;: glusterfs: create volume err: error creating volume Unable to execute command on glusterfs-vl1tl: volume start: vol_9903142828b2954544430d0f9d<wbr>d385bf: failed: Another transaction is in progress Please try again after sometime..</font></div><div><font face="monospace">  28s<span style="white-space:pre-wrap">                </span>28s<span style="white-space:pre-wrap">                </span>1<span style="white-space:pre-wrap">        </span>persistentvolume-controller<span style="white-space:pre-wrap">                        </span>Warning<span style="white-space:pre-wrap">                </span>ProvisioningFailed<span style="white-space:pre-wrap">        </span>Failed to provision volume with StorageClass &quot;distributed&quot;: glusterfs: create volume err: failed to create endpoint/service &lt;nil&gt;.</font></div><div><font face="monospace">  15s<span style="white-space:pre-wrap">                </span>15s<span style="white-space:pre-wrap">                </span>1<span style="white-space:pre-wrap">        </span>persistentvolume-controller<span style="white-space:pre-wrap">                        </span>Warning<span style="white-space:pre-wrap">                </span>ProvisioningFailed<span style="white-space:pre-wrap">        </span>Failed to provision volume with StorageClass &quot;distributed&quot;: glusterfs: create volume err: error creating volume Unable to execute command on glusterfs-vl1tl: volume start: vol_f4e20daae202e6190085fd4bbb<wbr>e837bb: failed: Another transaction is in progress Please try again after sometime..</font></div><div><font face="monospace">  0s<span style="white-space:pre-wrap">                </span>0s<span style="white-space:pre-wrap">                </span>1<span style="white-space:pre-wrap">        </span>persistentvolume-controller<span style="white-space:pre-wrap">                        </span>Warning<span style="white-space:pre-wrap">                </span>ProvisioningFailed<span style="white-space:pre-wrap">        </span>Failed to provision volume with StorageClass &quot;distributed&quot;: glusterfs: create volume err: error creating volume Unable to execute command on glusterfs-4sl4b: volume start: vol_8c913f988caa2328f57cc37e16<wbr>4d7d80: failed: Another transaction is in progress Please try again after sometime..</font></div></div></div>
<br></div></div>______________________________<wbr>_________________<br>
heketi-devel mailing list<br>
<a href="mailto:heketi-devel@gluster.org" target="_blank">heketi-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/heketi-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/heketi-devel</a><br>
<br></blockquote></div><br></div>
<br>______________________________<wbr>_________________<br>
heketi-devel mailing list<br>
<a href="mailto:heketi-devel@gluster.org">heketi-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/heketi-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/heketi-devel</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div>Cheers,<br>Humble<br><br></div>Red Hat Storage Engineering<br></div><div>Mastering KVM Virtualization: <span><a href="http://amzn.to/2vFTXaW" target="_blank">http://amzn.to/2vFTXaW</a></span></div>Website: <a href="http://humblec.com" target="_blank">http://humblec.com</a><br></div></div></div></div></div></div>
</div></div></div></div></div>