<div dir="ltr">Ok, here are a couple of logs (see ZIP)<div><br></div><div>Made two things: </div><div>- Create two PVC manually and deleted them</div><div>- Used the CoreOS Prometheus Operator, that creates a statefulset</div><div><br></div><div>What I must say is, that creating a Volume takes some time. Replicated about a minute or so because there is always another transactions ongoing</div><div><br></div><div>I've downgraded to HEKETI 4</div><div>GlusterFS Pods 3.10.5</div><div>K8s 1.6.8</div><div>running on Container Linux</div><div>I disabled all performance translators for every volume</div><div><br></div><div>Thanks for helping...</div><div>Christopher</div><br><div class="gmail_quote"><div dir="ltr">Humble Chirammal <<a href="mailto:hchiramm@redhat.com">hchiramm@redhat.com</a>> schrieb am Fr., 18. Aug. 2017 um 08:39 Uhr:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Christopher,<br><br></div><div>I believe this is what happened here:<br><br></div><div>heketi placed a request to gluster to create a volume which was succesful, once its done, heketi will issue volume start command and this command failed. For heketi, this is 'one transaction' which is volume creation and it "failed", so it would have tried to "delete the volume" ( this command would have failed in gluster too) created just now and will also wipe the DB entry from heketi. <br><br></div><div>Thats the reason for heketi not aware of this volume atm and volume still exist in the backend. <br></div><div><div><div><div class="gmail_extra"><br></div><div class="gmail_extra">>From the gluster error, it looks like "glusterd" is busy atm to serve the request: "<font face="monospace"> Another transaction is in progress Please try again after sometime..<span>"</span><br><br></font></div><div class="gmail_extra"><font face="monospace">I dont know what caused this ( may be many requests concurrent requests in glustrd or slowness in executing gluster commands..etc) and the setup much. Yeah, gluster and heketi logs could tell. <br><br></font></div><div class="gmail_extra"><font face="monospace"><br></font></div><div class="gmail_extra"><font face="monospace"><br><br></font></div><div class="gmail_extra"></div></div></div></div></div><div dir="ltr"><div><div><div><div class="gmail_extra"><div class="gmail_quote">On Fri, Aug 18, 2017 at 11:55 AM, Luis Pabon <span dir="ltr"><<a href="mailto:lpabon@chrysalix.org" target="_blank">lpabon@chrysalix.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">That is strange. I wonder if the volume creation was successful. Can show attach the logs for Heketi?<div><br></div><div>- Luis</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_8666526558937215511gmail-h5">On Thu, Aug 17, 2017 at 8:31 AM, Christopher Schmidt <span dir="ltr"><<a href="mailto:fakod666@gmail.com" target="_blank">fakod666@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_8666526558937215511gmail-h5"><div dir="ltr"><br><div>Hi all,</div><div><br></div><div>I've just tried Prometheus on Kubernetes, using</div><div>- CoreOS Prometheus Operator (0.10.2)</div><div>- K8s 1.6.8</div><div>- GlusterFS 3.10.4 (disabled all performance translators)</div><div>- Heketi 5</div><div><br></div><div>The Operator created a StatefulSet with a VolumeClaimTemplate prometheus-prometheus-db-prometheus-prometheus-0</div><div>As you can see below, the claim has issues creating volumes. </div><div>But actually, the volumes are created. LoggedIn in my GlusterFS Container I can see f.e. vol_9903142828b2954544430d0f9dd385bf but not in Heketi</div><div><br></div><div>Any idea how this can happen?</div><div><br></div><div>best Christopher</div><div><br></div><div> </div><div><br></div><div><br></div><div><div><font face="monospace">> kubectl -n metrics describe pvc prometheus-prometheus-db-prometheus-prometheus-0</font></div><div><font face="monospace">Name:<span style="white-space:pre-wrap"> </span>prometheus-prometheus-db-prometheus-prometheus-0</font></div><div><font face="monospace">Namespace:<span style="white-space:pre-wrap"> </span>metrics</font></div><div><font face="monospace">StorageClass:<span style="white-space:pre-wrap"> </span>distributed</font></div><div><font face="monospace">Status:<span style="white-space:pre-wrap"> </span>Pending</font></div><div><font face="monospace">Volume:</font></div><div><font face="monospace">Labels:<span style="white-space:pre-wrap"> </span>app=prometheus</font></div><div><font face="monospace"><span style="white-space:pre-wrap"> </span>prometheus=prometheus</font></div><div><font face="monospace">Annotations:<span style="white-space:pre-wrap"> </span><a href="http://volume.beta.kubernetes.io/storage-class=distributed" target="_blank">volume.beta.kubernetes.io/storage-class=distributed</a></font></div><div><font face="monospace"><span style="white-space:pre-wrap"> </span><a href="http://volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs" target="_blank">volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs</a></font></div><div><font face="monospace">Capacity:</font></div><div><font face="monospace">Access Modes:</font></div><div><font face="monospace">Events:</font></div><div><font face="monospace"> FirstSeen<span style="white-space:pre-wrap"> </span>LastSeen<span style="white-space:pre-wrap"> </span>Count<span style="white-space:pre-wrap"> </span>From<span style="white-space:pre-wrap"> </span>SubObjectPath<span style="white-space:pre-wrap"> </span>Type<span style="white-space:pre-wrap"> </span>Reason<span style="white-space:pre-wrap"> </span>Message</font></div><div><font face="monospace"> ---------<span style="white-space:pre-wrap"> </span>--------<span style="white-space:pre-wrap"> </span>-----<span style="white-space:pre-wrap"> </span>----<span style="white-space:pre-wrap"> </span>-------------<span style="white-space:pre-wrap"> </span>--------<span style="white-space:pre-wrap"> </span>------<span style="white-space:pre-wrap"> </span>-------</font></div><div><font face="monospace"> 39s<span style="white-space:pre-wrap"> </span>39s<span style="white-space:pre-wrap"> </span>1<span style="white-space:pre-wrap"> </span>persistentvolume-controller<span style="white-space:pre-wrap"> </span>Warning<span style="white-space:pre-wrap"> </span>ProvisioningFailed<span style="white-space:pre-wrap"> </span>Failed to provision volume with StorageClass "distributed": glusterfs: create volume err: error creating volume Unable to execute command on glusterfs-vl1tl: volume start: vol_9903142828b2954544430d0f9dd385bf: failed: Another transaction is in progress Please try again after sometime..</font></div><div><font face="monospace"> 28s<span style="white-space:pre-wrap"> </span>28s<span style="white-space:pre-wrap"> </span>1<span style="white-space:pre-wrap"> </span>persistentvolume-controller<span style="white-space:pre-wrap"> </span>Warning<span style="white-space:pre-wrap"> </span>ProvisioningFailed<span style="white-space:pre-wrap"> </span>Failed to provision volume with StorageClass "distributed": glusterfs: create volume err: failed to create endpoint/service <nil>.</font></div><div><font face="monospace"> 15s<span style="white-space:pre-wrap"> </span>15s<span style="white-space:pre-wrap"> </span>1<span style="white-space:pre-wrap"> </span>persistentvolume-controller<span style="white-space:pre-wrap"> </span>Warning<span style="white-space:pre-wrap"> </span>ProvisioningFailed<span style="white-space:pre-wrap"> </span>Failed to provision volume with StorageClass "distributed": glusterfs: create volume err: error creating volume Unable to execute command on glusterfs-vl1tl: volume start: vol_f4e20daae202e6190085fd4bbbe837bb: failed: Another transaction is in progress Please try again after sometime..</font></div><div><font face="monospace"> 0s<span style="white-space:pre-wrap"> </span>0s<span style="white-space:pre-wrap"> </span>1<span style="white-space:pre-wrap"> </span>persistentvolume-controller<span style="white-space:pre-wrap"> </span>Warning<span style="white-space:pre-wrap"> </span>ProvisioningFailed<span style="white-space:pre-wrap"> </span>Failed to provision volume with StorageClass "distributed": glusterfs: create volume err: error creating volume Unable to execute command on glusterfs-4sl4b: volume start: vol_8c913f988caa2328f57cc37e164d7d80: failed: Another transaction is in progress Please try again after sometime..</font></div></div></div>
<br></div></div>_______________________________________________<br>
heketi-devel mailing list<br>
<a href="mailto:heketi-devel@gluster.org" target="_blank">heketi-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/heketi-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/mailman/listinfo/heketi-devel</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>
heketi-devel mailing list<br>
<a href="mailto:heketi-devel@gluster.org" target="_blank">heketi-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/heketi-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/mailman/listinfo/heketi-devel</a><br>
<br></blockquote></div><br><br clear="all"><br></div></div></div></div></div><div dir="ltr"><div><div><div><div class="gmail_extra">-- <br><div class="m_8666526558937215511gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div>Cheers,<br>Humble<br><br></div>Red Hat Storage Engineering<br></div><div>Mastering KVM Virtualization: <span><a href="http://amzn.to/2vFTXaW" target="_blank">http://amzn.to/2vFTXaW</a></span></div>Website: <a href="http://humblec.com" target="_blank">http://humblec.com</a><br></div></div></div></div></div></div>
</div></div></div></div></div></blockquote></div></div>