<font size=2 face="sans-serif">Hi Madhu.</font>
<br>
<br><font size=2 face="sans-serif">I tried lot of times restarted heketi
pod but not resolved.</font>
<br>
<br><font size=2 face="sans-serif">sh-4.4# heketi-cli server operations
info</font>
<br><font size=2 face="sans-serif">Operation Counts:</font>
<br><font size=2 face="sans-serif"> Total: 0</font>
<br><font size=2 face="sans-serif"> In-Flight: 0</font>
<br><font size=2 face="sans-serif"> New: 0</font>
<br><font size=2 face="sans-serif"> Stale: 0</font>
<br>
<br><font size=2 face="sans-serif">Now you can see all operations are zero.
Now I try to create single volume below is observation in-flight reaching
slowly to 8.</font>
<br>
<br><font size=2 face="sans-serif">sh-4.4# heketi-cli server operations
infoCLI_SERVER=http://localhost:8080 ; export HEKETI_CLI_USE
Operation Counts:</font>
<br><font size=2 face="sans-serif"> Total: 0</font>
<br><font size=2 face="sans-serif"> In-Flight: 6</font>
<br><font size=2 face="sans-serif"> New: 0</font>
<br><font size=2 face="sans-serif"> Stale: 0</font>
<br><font size=2 face="sans-serif">sh-4.4# heketi-cli server operations
info</font>
<br><font size=2 face="sans-serif">Operation Counts:</font>
<br><font size=2 face="sans-serif"> Total: 0</font>
<br><font size=2 face="sans-serif"> In-Flight: 7</font>
<br><font size=2 face="sans-serif"> New: 0</font>
<br><font size=2 face="sans-serif"> Stale: 0</font>
<br><font size=2 face="sans-serif">sh-4.4# heketi-cli server operations
info</font>
<br><font size=2 face="sans-serif">Operation Counts:</font>
<br><font size=2 face="sans-serif"> Total: 0</font>
<br><font size=2 face="sans-serif"> In-Flight: 7</font>
<br><font size=2 face="sans-serif"> New: 0</font>
<br><font size=2 face="sans-serif"> Stale: 0</font>
<br><font size=2 face="sans-serif">sh-4.4# heketi-cli server operations
info</font>
<br><font size=2 face="sans-serif">Operation Counts:</font>
<br><font size=2 face="sans-serif"> Total: 0</font>
<br><font size=2 face="sans-serif"> In-Flight: 7</font>
<br><font size=2 face="sans-serif"> New: 0</font>
<br><font size=2 face="sans-serif"> Stale: 0</font>
<br><font size=2 face="sans-serif">sh-4.4# heketi-cli server operations
info</font>
<br><font size=2 face="sans-serif">Operation Counts:</font>
<br><font size=2 face="sans-serif"> Total: 0</font>
<br><font size=2 face="sans-serif"> In-Flight: 7</font>
<br><font size=2 face="sans-serif"> New: 0</font>
<br><font size=2 face="sans-serif"> Stale: 0</font>
<br>
<br><font size=2 face="sans-serif">[negroni] Completed 200 OK in 186.286µs</font>
<br><font size=2 face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size=2 face="sans-serif">[negroni] Started GET /operations</font>
<br><font size=2 face="sans-serif">[negroni] Completed 200 OK in 166.294µs</font>
<br><font size=2 face="sans-serif">[negroni] Started GET /operations</font>
<br><font size=2 face="sans-serif">[negroni] Completed 200 OK in 186.411µs</font>
<br><font size=2 face="sans-serif">[negroni] Started GET /operations</font>
<br><font size=2 face="sans-serif">[negroni] Completed 200 OK in 179.796µs</font>
<br><font size=2 face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size=2 face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size=2 face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size=2 face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size=2 face="sans-serif">[negroni] Started GET /operations</font>
<br><font size=2 face="sans-serif">[negroni] Completed 200 OK in 131.108µs</font>
<br><font size=2 face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size=2 face="sans-serif">[negroni] Started GET /operations</font>
<br><font size=2 face="sans-serif">[negroni] Completed 200 OK in 111.392µs</font>
<br><font size=2 face="sans-serif">[negroni] Started GET /operations</font>
<br><font size=2 face="sans-serif">[negroni] Completed 200 OK in 265.023µs</font>
<br><font size=2 face="sans-serif">[negroni] Started GET /operations</font>
<br><font size=2 face="sans-serif">[negroni] Completed 200 OK in 179.364µs</font>
<br><font size=2 face="sans-serif">[negroni] Started GET /operations</font>
<br><font size=2 face="sans-serif">[negroni] Completed 200 OK in 295.058µs</font>
<br><font size=2 face="sans-serif">[negroni] Started GET /operations</font>
<br><font size=2 face="sans-serif">[negroni] Completed 200 OK in 146.857µs</font>
<br><font size=2 face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size=2 face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size=2 face="sans-serif">[heketi] WARNING 2019/01/24 07:43:36
operations in-flight (8) exceeds limit (8)</font>
<br><font size=2 face="sans-serif">[negroni] Completed 429 Too Many Requests
in 403.166µs</font>
<br><font size=2 face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size=2 face="sans-serif">[heketi] WARNING 2019/01/24 07:43:51
operations in-flight (8) exceeds limit (8)</font>
<br><font size=2 face="sans-serif">[negroni] Completed 429 Too Many Requests
in 193.554µs</font>
<br>
<br>
<br><font size=2 face="sans-serif">But for pod volume is not creating.</font>
<br>
<br>
<table width=1078 style="border-collapse:collapse;">
<tr height=8>
<td width=67 bgcolor=white style="border-style:solid solid solid solid;border-color:#D1D1D1;border-width:1px 1px 1px 1px;padding:1px 1px;"><font size=2 color=#2f2f2f face="Helvetica">1:15:36
PM</font>
<td width=52 bgcolor=white style="border-style:solid solid solid solid;border-color:#D1D1D1;border-width:1px 1px 1px 1px;padding:1px 1px;">
<div align=center><font size=2 color=#2f2f2f face="Helvetica">Warning</font></div>
<td width=111 bgcolor=white style="border-style:solid solid solid solid;border-color:#D1D1D1;border-width:1px 1px 1px 1px;padding:1px 1px;"><font size=2 color=#2f2f2f face="Helvetica">Provisioning
failed </font>
<td width=832 bgcolor=white style="border-style:solid solid solid solid;border-color:#D1D1D1;border-width:1px 1px 1px 1px;padding:1px 1px;"><font size=2 color=#2f2f2f face="Helvetica">Failed
to provision volume with StorageClass "glusterfs-storage": glusterfs:
create volume err: error creating volume Server busy. Retry operation later..</font>
<br><font size=1 color=#a2a2a2 face="Helvetica">9 times in the last 2
minutes</font>
<tr height=8>
<td width=67 bgcolor=white style="border-style:solid solid solid solid;border-color:#D1D1D1;border-width:1px 1px 1px 1px;padding:1px 1px;"><font size=2 color=#2f2f2f face="Helvetica">1:13:21
PM</font>
<td width=52 bgcolor=white style="border-style:solid solid solid solid;border-color:#D1D1D1;border-width:1px 1px 1px 1px;padding:1px 1px;">
<div align=center><font size=2 color=#2f2f2f face="Helvetica">Warning</font></div>
<td width=111 bgcolor=white style="border-style:solid solid solid solid;border-color:#D1D1D1;border-width:1px 1px 1px 1px;padding:1px 1px;"><font size=2 color=#2f2f2f face="Helvetica">Provisioning
failed </font>
<td width=832 bgcolor=white style="border-style:solid solid solid solid;border-color:#D1D1D1;border-width:1px 1px 1px 1px;padding:1px 1px;"><font size=2 color=#2f2f2f face="Helvetica">Failed
to provision volume with StorageClass "glusterfs-storage": glusterfs:
create volume err: error creating volume .</font>
<br><font size=1 color=#a2a2a2 face="Helvetica">8 times in the last </font></table>
<br>
<br>
<br>
<br>
<br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">"Madhu Rajanna"
<mrajanna@redhat.com></font>
<br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">"Shaik Salam"
<shaik.salam@tcs.com></font>
<br><font size=1 color=#5f5f5f face="sans-serif">Cc:
</font><font size=1 face="sans-serif">"gluster-users@gluster.org
List" <gluster-users@gluster.org>, "Michael Adam"
<madam@redhat.com></font>
<br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">01/24/2019 12:51 PM</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: Failed to
provision volume with StorageClass "glusterfs-storage": glusterfs:
server busy</font>
<br>
<hr noshade>
<br>
<br>
<br><font size=2 color=#ff8141><b>"External email. Open with Caution"</b></font>
<br><font size=3>HI Shaik,</font>
<br>
<br><font size=3> can you provide me the outpout of </font><font size=3 color=red>$heketi-cli
server operations info</font><font size=3> from heketi pod</font>
<br>
<p><tt><font size=3>as a workround you can try restarting the heketi pod.
This will cause the current operations to go stale, but other pending
pvcs may go to Bound state</font></tt>
<p><tt><font size=3>Regards,</font></tt>
<p><tt><font size=3>Madhu R</font></tt>
<br>
<br><font size=3>On Thu, Jan 24, 2019 at 12:36 PM Shaik Salam <</font><a href=mailto:shaik.salam@tcs.com><font size=3 color=blue><u>shaik.salam@tcs.com</u></font></a><font size=3>>
wrote:</font>
<br><font size=2 face="sans-serif">H Madhu,</font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
Could you please have look my issue If you have time (atleast workaround).</font><font size=3>
</font><font size=2 face="sans-serif"><br>
I am unable to send mail to "J</font><font size=1 face="sans-serif">ohn
Mulligan" <</font><a href=mailto:John_Mulligan@redhat.com target=_blank><font size=1 color=blue face="sans-serif"><u>John_Mulligan@redhat.com</u></font></a><font size=1 face="sans-serif">></font><font size=2 face="sans-serif">"
who is currently handling issue</font><font size=3> </font><font size=3 color=blue><u><br>
</u></font><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1636912" target=_blank><font size=2 color=blue face="sans-serif"><u>https://bugzilla.redhat.com/show_bug.cgi?id=1636912</u></font></a><font size=3>
<br>
</font><font size=2 face="sans-serif"><br>
BR</font><font size=3> </font><font size=2 face="sans-serif"><br>
Salam</font><font size=3> <br>
<br>
</font><font size=1 color=#5f5f5f face="sans-serif"><br>
From: </font><font size=1 face="sans-serif">Shaik
Salam/HYD/TCS</font><font size=3> </font><font size=1 color=#5f5f5f face="sans-serif"><br>
To: </font><font size=1 face="sans-serif">"John
Mulligan" <</font><a href=mailto:John_Mulligan@redhat.com target=_blank><font size=1 color=blue face="sans-serif"><u>John_Mulligan@redhat.com</u></font></a><font size=1 face="sans-serif">>,
"Michael Adam" <</font><a href=mailto:madam@redhat.com target=_blank><font size=1 color=blue face="sans-serif"><u>madam@redhat.com</u></font></a><font size=1 face="sans-serif">>,
"Madhu Rajanna" <</font><a href=mailto:mrajanna@redhat.com target=_blank><font size=1 color=blue face="sans-serif"><u>mrajanna@redhat.com</u></font></a><font size=1 face="sans-serif">></font><font size=3>
</font><font size=1 color=#5f5f5f face="sans-serif"><br>
Cc: </font><font size=1 face="sans-serif">"</font><a href="mailto:gluster-users@gluster.org" target=_blank><font size=1 color=blue face="sans-serif"><u>gluster-users@gluster.org</u></font></a><font size=1 face="sans-serif">
List" <</font><a href="mailto:gluster-users@gluster.org" target=_blank><font size=1 color=blue face="sans-serif"><u>gluster-users@gluster.org</u></font></a><font size=1 face="sans-serif">></font><font size=3>
</font><font size=1 color=#5f5f5f face="sans-serif"><br>
Date: </font><font size=1 face="sans-serif">01/24/2019
12:21 PM</font><font size=3> </font><font size=1 color=#5f5f5f face="sans-serif"><br>
Subject: </font><font size=1 face="sans-serif">Failed
to provision volume with StorageClass "glusterfs-storage": glusterfs:
server busy</font><font size=3> <br>
</font>
<hr noshade><font size=3><br>
</font><font size=2 face="sans-serif"><br>
</font><font size=3> <br>
</font><font size=2 face="sans-serif"><br>
Hi All,</font><font size=3> <br>
</font><font size=2 color=#2f2f2f face="Segoe UI"><br>
We are facing also following issue on openshift origin while we are creating
pvc for pods.</font><font size=3> </font><font size=2 color=#2f2f2f face="Segoe UI"> (atlease
provide workaround to move further)</font><font size=3> </font><font size=2 color=#2f2f2f face="Segoe UI"><br>
<br>
Failed to provision volume with StorageClass "glusterfs-storage":
glusterfs: create volume err: error creating volume<br>
Failed to provision volume with StorageClass "glusterfs-storage":
glusterfs: create volume err: error creating volume Server busy. Retry
operation later..</font><font size=3> </font><font size=2 color=#2f2f2f face="Segoe UI"><br>
<br>
Please find heketidb dump and log</font><font size=3> </font><font size=2 face="sans-serif"><br>
<br>
[negroni] Completed 429 Too Many Requests in 250.763µs</font><font size=3>
</font><font size=2 face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size=3> </font><font size=2 face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds limit
(8)</font><font size=3> </font><font size=2 face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 169.08µs</font><font size=3>
</font><font size=2 face="sans-serif"><br>
[negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c</font><font size=3>
</font><font size=2 face="sans-serif"><br>
[negroni] Completed 404 Not Found in 148.125µs</font><font size=3> </font><font size=2 face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size=3> </font><font size=2 face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit
(8)</font><font size=3> </font><font size=2 face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 496.624µs</font><font size=3>
</font><font size=2 face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size=3> </font><font size=2 face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit
(8)</font><font size=3> </font><font size=2 face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 101.673µs</font><font size=3>
</font><font size=2 face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size=3> </font><font size=2 face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit
(8)</font><font size=3> </font><font size=2 face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 209.681µs</font><font size=3>
</font><font size=2 face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size=3> </font><font size=2 face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit
(8)</font><font size=3> </font><font size=2 face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 103.595µs</font><font size=3>
</font><font size=2 face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size=3> </font><font size=2 face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit
(8)</font><font size=3> </font><font size=2 face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 297.594µs</font><font size=3>
</font><font size=2 face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size=3> </font><font size=2 face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit
(8)</font><font size=3> </font><font size=2 face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 96.75µs</font><font size=3>
</font><font size=2 face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size=3> </font><font size=2 face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size=3> </font><font size=2 face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit
(8)</font><font size=3> </font><font size=2 face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 477.007µs</font><font size=3>
</font><font size=2 face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit
(8)</font><font size=3> </font><font size=2 face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 165.38µs</font><font size=3>
</font><font size=2 face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size=3> </font><font size=2 face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit
(8)</font><font size=3> </font><font size=2 face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 488.253µs</font><font size=3>
</font><font size=2 face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size=3> </font><font size=2 face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit
(8)</font><font size=3> </font><font size=2 face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 171.836µs</font><font size=3>
</font><font size=2 face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size=3> </font><font size=2 face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit
(8)</font><font size=3> </font><font size=2 face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 208.59µs</font><font size=3>
</font><font size=2 face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size=3> </font><font size=2 face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit
(8)</font><font size=3> </font><font size=2 face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 125.141µs</font><font size=3>
</font><font size=2 face="sans-serif"><br>
[negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606</font><font size=3>
</font><font size=2 face="sans-serif"><br>
[negroni] Completed 404 Not Found in 138.687µs</font><font size=3> </font><font size=2 face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size=3> </font><font size=2 face="sans-serif"><br>
<br>
<br>
BR</font><font size=3> </font><font size=2 face="sans-serif"><br>
Salam</font><font size=3> </font>
<br><font size=3>=====-----=====-----=====<br>
Notice: The information contained in this e-mail<br>
message and/or attachments to it may contain <br>
confidential or privileged information. If you are <br>
not the intended recipient, any dissemination, use, <br>
review, distribution, printing or copying of the <br>
information contained in this e-mail message <br>
and/or attachments to it are strictly prohibited. If <br>
you have received this communication in error, <br>
please notify us by reply e-mail or telephone and <br>
immediately and permanently delete the message <br>
and any attachments. Thank you</font>
<p>
<br>
<br><font size=3>-- </font>
<br><font size=3>Madhu Rajanna</font>
<br><font size=3>Software Engineer</font>
<br><a href=https://www.redhat.com/ target=_blank><font size=1 color=#0082bf face="sans-serif">Red
Hat Bangalore, India</font></a>
<br><a href=mailto:mrajanna@redhat.com target=_blank><font size=1 color=#0082bf face="sans-serif">mrajanna@redhat.com</font></a><font size=1 color=#a2a2a2>
M: +91-9741133155 </font>
<p><a href=https://red.ht/sig target=_blank></a>
<p>