<div dir="ltr">Adding John who is having more idea about how to debug this one.<div><br></div><div><a class="gmail_plusreply" id="plusReplyChip-0" href="mailto:shaik.salam@tcs.com" tabindex="-1">@Shaik Salam</a> can you some more info on the hardware on which you are running heketi (kernel details)<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 24, 2019 at 7:42 PM Shaik Salam <<a href="mailto:shaik.salam@tcs.com">shaik.salam@tcs.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font size="2" face="sans-serif">Hi Madhu,</font>
<br>
<br><font size="2" face="sans-serif">Sorry to disturb could you please provide
atleast work around (to clear requests which stuck) to move further.</font>
<br><font size="2" face="sans-serif">We are also not able to find root cause
from glusterd logs. Please find attachment.</font>
<br>
<br><font size="2" face="sans-serif">BR</font>
<br><font size="2" face="sans-serif">Salam</font>
<br>
<br><font size="2" face="sans-serif"> </font>
<br>
<br>
<br>
<br><font size="1" color="#5f5f5f" face="sans-serif">From:
</font><font size="1" face="sans-serif">Shaik Salam/HYD/TCS</font>
<br><font size="1" color="#5f5f5f" face="sans-serif">To:
</font><font size="1" face="sans-serif">"Madhu Rajanna"
<<a href="mailto:mrajanna@redhat.com" target="_blank">mrajanna@redhat.com</a>></font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Cc:
</font><font size="1" face="sans-serif">"<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>
List" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>>, "Michael Adam"
<<a href="mailto:madam@redhat.com" target="_blank">madam@redhat.com</a>></font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Date:
</font><font size="1" face="sans-serif">01/24/2019 04:12 PM</font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Subject:
</font><font size="1" face="sans-serif">Re: Failed to
provision volume with StorageClass "glusterfs-storage": glusterfs:
server busy</font>
<br>
<hr noshade>
<br>
<br><font size="2" face="sans-serif">Hi Madhu,</font>
<br>
<br><font size="2" face="sans-serif">Please let me know If any other information
required.</font>
<br>
<br><font size="2" face="sans-serif">BR</font>
<br><font size="2" face="sans-serif">Salam</font>
<br>
<br>
<br>
<br>
<br><font size="1" color="#5f5f5f" face="sans-serif">From:
</font><font size="1" face="sans-serif">Shaik Salam/HYD/TCS</font>
<br><font size="1" color="#5f5f5f" face="sans-serif">To:
</font><font size="1" face="sans-serif">"Madhu Rajanna"
<<a href="mailto:mrajanna@redhat.com" target="_blank">mrajanna@redhat.com</a>></font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Cc:
</font><font size="1" face="sans-serif">"<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>
List" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>>, "Michael Adam"
<<a href="mailto:madam@redhat.com" target="_blank">madam@redhat.com</a>></font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Date:
</font><font size="1" face="sans-serif">01/24/2019 03:23 PM</font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Subject:
</font><font size="1" face="sans-serif">Re: Failed to
provision volume with StorageClass "glusterfs-storage": glusterfs:
server busy</font>
<br>
<hr noshade>
<br>
<br><font size="2" face="sans-serif">Hi Madhu,</font>
<br>
<br><font size="2" face="sans-serif">This is complete one after restart of
heketi pod and process log.</font>
<br>
<br><font size="2" face="sans-serif">BR</font>
<br><font size="2" face="sans-serif">Salam</font>
<br>
<br><font size="2" face="sans-serif">[attachment "heketi-pod-complete.log"
deleted by Shaik Salam/HYD/TCS] [attachment "ps-aux.txt"
deleted by Shaik Salam/HYD/TCS] </font>
<br>
<br>
<br>
<br>
<br><font size="1" color="#5f5f5f" face="sans-serif">From:
</font><font size="1" face="sans-serif">"Madhu Rajanna"
<<a href="mailto:mrajanna@redhat.com" target="_blank">mrajanna@redhat.com</a>></font>
<br><font size="1" color="#5f5f5f" face="sans-serif">To:
</font><font size="1" face="sans-serif">"Shaik Salam"
<<a href="mailto:shaik.salam@tcs.com" target="_blank">shaik.salam@tcs.com</a>></font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Cc:
</font><font size="1" face="sans-serif">"<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>
List" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>>, "Michael Adam"
<<a href="mailto:madam@redhat.com" target="_blank">madam@redhat.com</a>></font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Date:
</font><font size="1" face="sans-serif">01/24/2019 01:55 PM</font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Subject:
</font><font size="1" face="sans-serif">Re: Failed to
provision volume with StorageClass "glusterfs-storage": glusterfs:
server busy</font>
<br>
<hr noshade>
<br>
<br>
<br><font size="2" color="#ff8141"><b>"External email. Open with Caution"</b></font>
<br><font size="3">the logs you provided is not complete, not able to figure
out which command is struck, can you reattach the complete output of `ps
aux` and also attach complete heketi logs. </font>
<br>
<br><font size="3">On Thu, Jan 24, 2019 at 1:41 PM Shaik Salam <</font><a href="mailto:shaik.salam@tcs.com" target="_blank"><font size="3" color="blue"><u>shaik.salam@tcs.com</u></font></a><font size="3">>
wrote:</font>
<br><font size="2" face="sans-serif">Hi Madhu,</font><font size="3"> <br>
</font><font size="2" face="sans-serif"><br>
Please find requested info.</font><font size="3"> <br>
</font><font size="2" face="sans-serif"><br>
BR</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Salam</font><font size="3"> <br>
</font><font size="2" face="sans-serif"><br>
</font><font size="3"> <br>
<br>
<br>
</font><font size="1" color="#5f5f5f" face="sans-serif"><br>
From: </font><font size="1" face="sans-serif">Madhu
Rajanna <</font><a href="mailto:mrajanna@redhat.com" target="_blank"><font size="1" color="blue" face="sans-serif"><u>mrajanna@redhat.com</u></font></a><font size="1" face="sans-serif">></font><font size="3">
</font><font size="1" color="#5f5f5f" face="sans-serif"><br>
To: </font><font size="1" face="sans-serif">Shaik
Salam <</font><a href="mailto:shaik.salam@tcs.com" target="_blank"><font size="1" color="blue" face="sans-serif"><u>shaik.salam@tcs.com</u></font></a><font size="1" face="sans-serif">></font><font size="3">
</font><font size="1" color="#5f5f5f" face="sans-serif"><br>
Cc: </font><font size="1" face="sans-serif">"</font><a href="mailto:gluster-users@gluster.org" target="_blank"><font size="1" color="blue" face="sans-serif"><u>gluster-users@gluster.org</u></font></a><font size="1" face="sans-serif">
List" <</font><a href="mailto:gluster-users@gluster.org" target="_blank"><font size="1" color="blue" face="sans-serif"><u>gluster-users@gluster.org</u></font></a><font size="1" face="sans-serif">>,
Michael Adam <</font><a href="mailto:madam@redhat.com" target="_blank"><font size="1" color="blue" face="sans-serif"><u>madam@redhat.com</u></font></a><font size="1" face="sans-serif">></font><font size="3">
</font><font size="1" color="#5f5f5f" face="sans-serif"><br>
Date: </font><font size="1" face="sans-serif">01/24/2019
01:33 PM</font><font size="3"> </font><font size="1" color="#5f5f5f" face="sans-serif"><br>
Subject: </font><font size="1" face="sans-serif">Re:
Failed to provision volume with StorageClass "glusterfs-storage":
glusterfs: server busy</font><font size="3"> <br>
</font>
<hr noshade><font size="3"><br>
<br>
</font><font size="2" color="#ff8141"><b><br>
"External email. Open with Caution"</b></font><font size="3"> <br>
the heketi logs you have attached is not complete i believe, can you povide
the complete heketi logs <br>
and also an we get the output of "ps aux" from the gluster
pods ? I want to see if any lvm commands or gluster commands are "stuck".
<br>
<br>
<br>
On Thu, Jan 24, 2019 at 1:16 PM Shaik Salam <</font><a href="mailto:shaik.salam@tcs.com" target="_blank"><font size="3" color="blue"><u>shaik.salam@tcs.com</u></font></a><font size="3">>
wrote: </font><font size="2" face="sans-serif"><br>
Hi Madhu.</font><font size="3"> </font><font size="2" face="sans-serif"><br>
<br>
I tried lot of times restarted heketi pod but not resolved.</font><font size="3">
</font><font size="2" face="sans-serif"><br>
<br>
sh-4.4# heketi-cli server operations info</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Operation Counts:</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Total: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
In-Flight: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
New: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Stale: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
<br>
Now you can see all operations are zero. Now I try to create single volume
below is observation in-flight reaching slowly to 8.</font><font size="3">
</font><font size="2" face="sans-serif"><br>
<br>
sh-4.4# heketi-cli server operations infoCLI_SERVER=</font><a href="http://localhost:8080/" target="_blank"><font size="2" color="blue" face="sans-serif"><u>http://localhost:8080</u></font></a><font size="2" face="sans-serif">
; export HEKETI_CLI_USE
Operation Counts:</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Total: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
In-Flight: 6</font><font size="3"> </font><font size="2" face="sans-serif"><br>
New: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Stale: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
sh-4.4# heketi-cli server operations info</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Operation Counts:</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Total: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
In-Flight: 7</font><font size="3"> </font><font size="2" face="sans-serif"><br>
New: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Stale: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
sh-4.4# heketi-cli server operations info</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Operation Counts:</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Total: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
In-Flight: 7</font><font size="3"> </font><font size="2" face="sans-serif"><br>
New: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Stale: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
sh-4.4# heketi-cli server operations info</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Operation Counts:</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Total: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
In-Flight: 7</font><font size="3"> </font><font size="2" face="sans-serif"><br>
New: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Stale: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
sh-4.4# heketi-cli server operations info</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Operation Counts:</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Total: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
In-Flight: 7</font><font size="3"> </font><font size="2" face="sans-serif"><br>
New: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Stale: 0</font><font size="3"> </font><font size="2" face="sans-serif"><br>
<br>
[negroni] Completed 200 OK in 186.286µs</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started GET /operations</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 200 OK in 166.294µs</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started GET /operations</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 200 OK in 186.411µs</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started GET /operations</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 200 OK in 179.796µs</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started GET /operations</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 200 OK in 131.108µs</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started GET /operations</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 200 OK in 111.392µs</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started GET /operations</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 200 OK in 265.023µs</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started GET /operations</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 200 OK in 179.364µs</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started GET /operations</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 200 OK in 295.058µs</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started GET /operations</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 200 OK in 146.857µs</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[heketi] WARNING 2019/01/24 07:43:36 operations in-flight (8) exceeds limit
(8)</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 403.166µs</font><font size="3">
</font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[heketi] WARNING 2019/01/24 07:43:51 operations in-flight (8) exceeds limit
(8)</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 193.554µs</font><font size="3">
<br>
</font><font size="2" face="sans-serif"><br>
<br>
But for pod volume is not creating.</font><font size="3"> </font>
<table width="1077" style="border-collapse:collapse">
<tbody><tr height="8">
<td width="65" bgcolor="white" style="border-style:solid;border-color:rgb(209,209,209);border-width:1px;padding:1px"><font size="2" color="#2f2f2f">1:15:36
PM</font><font size="3"> </font>
</td><td width="50" bgcolor="white" style="border-style:solid;border-color:rgb(209,209,209);border-width:1px;padding:1px">
<div align="center"><font size="2" color="#2f2f2f">Warning</font></div>
</td><td width="109" bgcolor="white" style="border-style:solid;border-color:rgb(209,209,209);border-width:1px;padding:1px"><font size="2" color="#2f2f2f">Provisioning
failed </font><font size="3"> </font>
</td><td width="840" bgcolor="white" style="border-style:solid;border-color:rgb(209,209,209);border-width:1px;padding:1px"><font size="2" color="#2f2f2f">Failed
to provision volume with StorageClass "glusterfs-storage": glusterfs:
create volume err: error creating volume Server busy. Retry operation later..</font><font size="3">
</font><font size="1" color="#a2a2a2"><br>
9 times in the last 2 minutes</font><font size="3"> </font>
</td></tr><tr height="8">
<td width="65" bgcolor="white" style="border-style:solid;border-color:rgb(209,209,209);border-width:1px;padding:1px"><font size="2" color="#2f2f2f">1:13:21
PM</font><font size="3"> </font>
</td><td width="50" bgcolor="white" style="border-style:solid;border-color:rgb(209,209,209);border-width:1px;padding:1px">
<div align="center"><font size="2" color="#2f2f2f">Warning</font></div>
</td><td width="109" bgcolor="white" style="border-style:solid;border-color:rgb(209,209,209);border-width:1px;padding:1px"><font size="2" color="#2f2f2f">Provisioning
failed </font><font size="3"> </font>
</td><td width="840" bgcolor="white" style="border-style:solid;border-color:rgb(209,209,209);border-width:1px;padding:1px"><font size="2" color="#2f2f2f">Failed
to provision volume with StorageClass "glusterfs-storage": glusterfs:
create volume err: error creating volume .</font><font size="3"> </font><font size="1" color="#a2a2a2"><br>
8 times in the last </font></td></tr></tbody></table>
<br><font size="3"><br>
<br>
<br>
<br>
</font><font size="1" color="#5f5f5f" face="sans-serif"><br>
<br>
From: </font><font size="1" face="sans-serif">"Madhu
Rajanna" <</font><a href="mailto:mrajanna@redhat.com" target="_blank"><font size="1" color="blue" face="sans-serif"><u>mrajanna@redhat.com</u></font></a><font size="1" face="sans-serif">></font><font size="3">
</font><font size="1" color="#5f5f5f" face="sans-serif"><br>
To: </font><font size="1" face="sans-serif">"Shaik
Salam" <</font><a href="mailto:shaik.salam@tcs.com" target="_blank"><font size="1" color="blue" face="sans-serif"><u>shaik.salam@tcs.com</u></font></a><font size="1" face="sans-serif">></font><font size="3">
</font><font size="1" color="#5f5f5f" face="sans-serif"><br>
Cc: </font><font size="1" face="sans-serif">"</font><a href="mailto:gluster-users@gluster.org" target="_blank"><font size="1" color="blue" face="sans-serif"><u>gluster-users@gluster.org</u></font></a><font size="1" face="sans-serif">
List" <</font><a href="mailto:gluster-users@gluster.org" target="_blank"><font size="1" color="blue" face="sans-serif"><u>gluster-users@gluster.org</u></font></a><font size="1" face="sans-serif">>,
"Michael Adam" <</font><a href="mailto:madam@redhat.com" target="_blank"><font size="1" color="blue" face="sans-serif"><u>madam@redhat.com</u></font></a><font size="1" face="sans-serif">></font><font size="3">
</font><font size="1" color="#5f5f5f" face="sans-serif"><br>
Date: </font><font size="1" face="sans-serif">01/24/2019
12:51 PM</font><font size="3"> </font><font size="1" color="#5f5f5f" face="sans-serif"><br>
Subject: </font><font size="1" face="sans-serif">Re:
Failed to provision volume with StorageClass "glusterfs-storage":
glusterfs: server busy</font><font size="3"> <br>
</font>
<hr noshade><font size="3"><br>
</font><font size="2" color="#ff8141"><b><br>
<br>
"External email. Open with Caution"</b></font><font size="3"> <br>
HI Shaik, <br>
<br>
can you provide me the outpout of </font><font size="3" color="red">$heketi-cli
server operations info</font><font size="3"> from heketi pod </font>
<p><tt><font size="3">as a workround you can try restarting the heketi pod.
This will cause the current operations to go stale, but other pending
pvcs may go to Bound state</font></tt><font size="3"> </font>
</p><p><tt><font size="3">Regards,</font></tt><font size="3"> </font>
</p><p><tt><font size="3">Madhu R</font></tt><font size="3"> <br>
<br>
On Thu, Jan 24, 2019 at 12:36 PM Shaik Salam <</font><a href="mailto:shaik.salam@tcs.com" target="_blank"><font size="3" color="blue"><u>shaik.salam@tcs.com</u></font></a><font size="3">>
wrote: </font><font size="2" face="sans-serif"><br>
H Madhu,</font><font size="3"> </font><font size="2" face="sans-serif"><br>
<br>
Could you please have look my issue If you have time (atleast workaround).</font><font size="3">
</font><font size="2" face="sans-serif"><br>
I am unable to send mail to "J</font><font size="1" face="sans-serif">ohn
Mulligan" <</font><a href="mailto:John_Mulligan@redhat.com" target="_blank"><font size="1" color="blue" face="sans-serif"><u>John_Mulligan@redhat.com</u></font></a><font size="1" face="sans-serif">></font><font size="2" face="sans-serif">"
who is currently handling issue</font><font size="3"> </font><font size="3" color="blue"><u><br>
</u></font><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1636912" target="_blank"><font size="2" color="blue" face="sans-serif"><u>https://bugzilla.redhat.com/show_bug.cgi?id=1636912</u></font></a><font size="3">
</font><font size="2" face="sans-serif"><br>
<br>
BR</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Salam</font><font size="3"> </font><font size="1" color="#5f5f5f" face="sans-serif"><br>
<br>
<br>
From: </font><font size="1" face="sans-serif">Shaik
Salam/HYD/TCS</font><font size="3"> </font><font size="1" color="#5f5f5f" face="sans-serif"><br>
To: </font><font size="1" face="sans-serif">"John
Mulligan" <</font><a href="mailto:John_Mulligan@redhat.com" target="_blank"><font size="1" color="blue" face="sans-serif"><u>John_Mulligan@redhat.com</u></font></a><font size="1" face="sans-serif">>,
"Michael Adam" <</font><a href="mailto:madam@redhat.com" target="_blank"><font size="1" color="blue" face="sans-serif"><u>madam@redhat.com</u></font></a><font size="1" face="sans-serif">>,
"Madhu Rajanna" <</font><a href="mailto:mrajanna@redhat.com" target="_blank"><font size="1" color="blue" face="sans-serif"><u>mrajanna@redhat.com</u></font></a><font size="1" face="sans-serif">></font><font size="3">
</font><font size="1" color="#5f5f5f" face="sans-serif"><br>
Cc: </font><font size="1" face="sans-serif">"</font><a href="mailto:gluster-users@gluster.org" target="_blank"><font size="1" color="blue" face="sans-serif"><u>gluster-users@gluster.org</u></font></a><font size="1" face="sans-serif">
List" <</font><a href="mailto:gluster-users@gluster.org" target="_blank"><font size="1" color="blue" face="sans-serif"><u>gluster-users@gluster.org</u></font></a><font size="1" face="sans-serif">></font><font size="3">
</font><font size="1" color="#5f5f5f" face="sans-serif"><br>
Date: </font><font size="1" face="sans-serif">01/24/2019
12:21 PM</font><font size="3"> </font><font size="1" color="#5f5f5f" face="sans-serif"><br>
Subject: </font><font size="1" face="sans-serif">Failed
to provision volume with StorageClass "glusterfs-storage": glusterfs:
server busy</font><font size="3"> </font>
</p><p>
</p><hr noshade><font size="2" face="sans-serif"><br>
<br>
</font><font size="3"> </font><font size="2" face="sans-serif"><br>
<br>
Hi All,</font><font size="3"> </font><font size="2" color="#2f2f2f" face="Segoe UI"><br>
<br>
We are facing also following issue on openshift origin while we are creating
pvc for pods.</font><font size="3"> </font><font size="2" color="#2f2f2f" face="Segoe UI"> (atlease
provide workaround to move further)</font><font size="3"> </font><font size="2" color="#2f2f2f" face="Segoe UI"><br>
<br>
Failed to provision volume with StorageClass "glusterfs-storage":
glusterfs: create volume err: error creating volume<br>
Failed to provision volume with StorageClass "glusterfs-storage":
glusterfs: create volume err: error creating volume Server busy. Retry
operation later..</font><font size="3"> </font><font size="2" color="#2f2f2f" face="Segoe UI"><br>
<br>
Please find heketidb dump and log</font><font size="3"> </font><font size="2" face="sans-serif"><br>
<br>
[negroni] Completed 429 Too Many Requests in 250.763µs</font><font size="3">
</font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds limit
(8)</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 169.08µs</font><font size="3">
</font><font size="2" face="sans-serif"><br>
[negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c</font><font size="3">
</font><font size="2" face="sans-serif"><br>
[negroni] Completed 404 Not Found in 148.125µs</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit
(8)</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 496.624µs</font><font size="3">
</font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds limit
(8)</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 101.673µs</font><font size="3">
</font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit
(8)</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 209.681µs</font><font size="3">
</font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds limit
(8)</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 103.595µs</font><font size="3">
</font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit
(8)</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 297.594µs</font><font size="3">
</font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds limit
(8)</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 96.75µs</font><font size="3">
</font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit
(8)</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 477.007µs</font><font size="3">
</font><font size="2" face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds limit
(8)</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 165.38µs</font><font size="3">
</font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit
(8)</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 488.253µs</font><font size="3">
</font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds limit
(8)</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 171.836µs</font><font size="3">
</font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit
(8)</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 208.59µs</font><font size="3">
</font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds limit
(8)</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Completed 429 Too Many Requests in 125.141µs</font><font size="3">
</font><font size="2" face="sans-serif"><br>
[negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606</font><font size="3">
</font><font size="2" face="sans-serif"><br>
[negroni] Completed 404 Not Found in 138.687µs</font><font size="3"> </font><font size="2" face="sans-serif"><br>
[negroni] Started POST /volumes</font><font size="3"> </font><font size="2" face="sans-serif"><br>
<br>
<br>
BR</font><font size="3"> </font><font size="2" face="sans-serif"><br>
Salam</font><font size="3"> <br>
=====-----=====-----=====<br>
Notice: The information contained in this e-mail<br>
message and/or attachments to it may contain <br>
confidential or privileged information. If you are <br>
not the intended recipient, any dissemination, use, <br>
review, distribution, printing or copying of the <br>
information contained in this e-mail message <br>
and/or attachments to it are strictly prohibited. If <br>
you have received this communication in error, <br>
please notify us by reply e-mail or telephone and <br>
immediately and permanently delete the message <br>
and any attachments. Thank you </font>
<p><font size="3"><br>
<br>
-- <br>
Madhu Rajanna <br>
Software Engineer </font><font size="3" color="blue"><u><br>
</u></font><a href="https://www.redhat.com/" target="_blank"><font size="1" color="#0082bf" face="sans-serif"><u>Red
Hat Bangalore, India</u></font></a><font size="3"> </font><font size="3" color="blue"><u><br>
</u></font><a href="mailto:mrajanna@redhat.com" target="_blank"><font size="1" color="#0082bf" face="sans-serif"><u>mrajanna@redhat.com</u></font></a><font size="1" color="#a2a2a2">
M: +91-9741133155 </font>
</p><p><a href="https://red.ht/sig" target="_blank"></a>
</p><p><font size="3"><br>
<br>
-- <br>
Madhu Rajanna <br>
Software Engineer </font><font size="3" color="blue"><u><br>
</u></font><a href="https://www.redhat.com/" target="_blank"><font size="1" color="#0082bf" face="sans-serif"><u>Red
Hat Bangalore, India</u></font></a><font size="3"> </font><font size="3" color="blue"><u><br>
</u></font><a href="mailto:mrajanna@redhat.com" target="_blank"><font size="1" color="#0082bf" face="sans-serif"><u>mrajanna@redhat.com</u></font></a><font size="1" color="#a2a2a2">
M: +91-9741133155 </font>
</p><p><a href="https://red.ht/sig" target="_blank"></a>
</p><p>
<br>
<br><font size="3">-- </font>
<br><font size="3">Madhu Rajanna</font>
<br><font size="3">Software Engineer</font>
<br><a href="https://www.redhat.com/" target="_blank"><font size="1" color="#0082bf" face="sans-serif">Red
Hat Bangalore, India</font></a>
<br><a href="mailto:mrajanna@redhat.com" target="_blank"><font size="1" color="#0082bf" face="sans-serif">mrajanna@redhat.com</font></a><font size="1" color="#a2a2a2">
M: +91-9741133155 </font>
</p><p><a href="https://red.ht/sig" target="_blank"></a>
</p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>Madhu Rajanna</div><div>Software Engineer<br></div><p style="font-weight:normal;margin:0px;font-size:10px;color:rgb(153,153,153)"><a style="color:rgb(0,136,206);font-size:10px;margin:0px;text-decoration:none;font-family:overpass,sans-serif" href="https://www.redhat.com" target="_blank">Red Hat <span>Bangalore, India</span></a></p>
<p style="font-weight:normal;margin:0px 0px 6px;font-size:10px;color:rgb(153,153,153)"><span style="margin:0px;padding:0px">
<a style="color:rgb(0,136,206);font-size:10px;margin:0px;text-decoration:none;font-family:overpass,sans-serif" href="mailto:mrajanna@redhat.com" target="_blank">mrajanna@redhat.com</a> </span>
<span href="tel:9845222159">M: +91-9741133155 </span>
</p>
<a href="https://red.ht/sig" target="_blank"> <img src="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" height="auto" width="90"></a></div></div>