<div dir="ltr">Looks like the requests to create volume has increased significantly. Heketi can handle 8 parallel volume create requests, and looks like there are 429 volume create requests pending.<div><br></div><div>I am not an expert in this. Added few more people in CC to help when they get to see this.</div><div><br></div><div>-Amar</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 24, 2019 at 11:28 AM Shaik Salam <<a href="mailto:shaik.salam@tcs.com">shaik.salam@tcs.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font size="2" face="sans-serif">Hi Surya,</font>
<br>
<br><font size="2" face="sans-serif">Could you please help us to resolve
below issue (at lease workaround for creating volume)</font>
<br><font size="2" face="sans-serif">Attached db dump and log. Please let
me know any other things need to check.</font>
<br><font size="2" face="sans-serif">Please guide us.</font>
<br>
<br><font size="2" face="sans-serif">BR</font>
<br><font size="2" face="sans-serif">Salam</font>
<br>
<br>
<br>
<br><font size="1" color="#5f5f5f" face="sans-serif">From:
</font><font size="1" face="sans-serif">Shaik Salam/HYD/TCS</font>
<br><font size="1" color="#5f5f5f" face="sans-serif">To:
</font><font size="1" face="sans-serif">"Amar Tumballi
Suryanarayan" <<a href="mailto:atumball@redhat.com" target="_blank">atumball@redhat.com</a>>, <a href="mailto:bugs@gluster.org" target="_blank">bugs@gluster.org</a>, "<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>
List" <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>></font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Cc:
</font><font size="1" face="sans-serif">"Murali Kottakota"
<<a href="mailto:murali.kottakota@tcs.com" target="_blank">murali.kottakota@tcs.com</a>>, "Sanju Rakonde" <<a href="mailto:srakonde@redhat.com" target="_blank">srakonde@redhat.com</a>></font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Date:
</font><font size="1" face="sans-serif">01/23/2019 06:19 PM</font>
<br><font size="1" color="#5f5f5f" face="sans-serif">Subject:
</font><font size="1" face="sans-serif"> Failed to provision
volume with StorageClass "glusterfs-storage": glusterfs: create
volume err: error creating volume </font>
<br>
<hr noshade>
<br>
<br>
<br>
<br>
<br><font size="2" color="#2f2f2f" face="Segoe UI">Hi,</font>
<br>
<br><font size="2" color="#2f2f2f" face="Segoe UI">We are facing also following
issue on openshift origin while we are creating pvc for pods.</font>
<br>
<br><font size="2" color="#2f2f2f" face="Segoe UI">Failed to provision volume
with StorageClass "glusterfs-storage": glusterfs: create volume
err: error creating volume<br>
Failed to provision volume with StorageClass "glusterfs-storage":
glusterfs: create volume err: error creating volume Server busy. Retry
operation later..</font>
<br>
<br><font size="2" color="#2f2f2f" face="Segoe UI">Please find heketidb dump
and log</font>
<br>
<br><font size="2" face="sans-serif">[negroni] Completed 429 Too Many Requests
in 250.763µs</font>
<br><font size="2" face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size="2" face="sans-serif">[heketi] WARNING 2019/01/23 12:07:49
operations in-flight (8) exceeds limit (8)</font>
<br><font size="2" face="sans-serif">[negroni] Completed 429 Too Many Requests
in 169.08µs</font>
<br><font size="2" face="sans-serif">[negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c</font>
<br><font size="2" face="sans-serif">[negroni] Completed 404 Not Found in
148.125µs</font>
<br><font size="2" face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size="2" face="sans-serif">[heketi] WARNING 2019/01/23 12:08:04
operations in-flight (8) exceeds limit (8)</font>
<br><font size="2" face="sans-serif">[negroni] Completed 429 Too Many Requests
in 496.624µs</font>
<br><font size="2" face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size="2" face="sans-serif">[heketi] WARNING 2019/01/23 12:08:04
operations in-flight (8) exceeds limit (8)</font>
<br><font size="2" face="sans-serif">[negroni] Completed 429 Too Many Requests
in 101.673µs</font>
<br><font size="2" face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size="2" face="sans-serif">[heketi] WARNING 2019/01/23 12:08:19
operations in-flight (8) exceeds limit (8)</font>
<br><font size="2" face="sans-serif">[negroni] Completed 429 Too Many Requests
in 209.681µs</font>
<br><font size="2" face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size="2" face="sans-serif">[heketi] WARNING 2019/01/23 12:08:19
operations in-flight (8) exceeds limit (8)</font>
<br><font size="2" face="sans-serif">[negroni] Completed 429 Too Many Requests
in 103.595µs</font>
<br><font size="2" face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size="2" face="sans-serif">[heketi] WARNING 2019/01/23 12:08:34
operations in-flight (8) exceeds limit (8)</font>
<br><font size="2" face="sans-serif">[negroni] Completed 429 Too Many Requests
in 297.594µs</font>
<br><font size="2" face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size="2" face="sans-serif">[heketi] WARNING 2019/01/23 12:08:34
operations in-flight (8) exceeds limit (8)</font>
<br><font size="2" face="sans-serif">[negroni] Completed 429 Too Many Requests
in 96.75µs</font>
<br><font size="2" face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size="2" face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size="2" face="sans-serif">[heketi] WARNING 2019/01/23 12:08:49
operations in-flight (8) exceeds limit (8)</font>
<br><font size="2" face="sans-serif">[negroni] Completed 429 Too Many Requests
in 477.007µs</font>
<br><font size="2" face="sans-serif">[heketi] WARNING 2019/01/23 12:08:49
operations in-flight (8) exceeds limit (8)</font>
<br><font size="2" face="sans-serif">[negroni] Completed 429 Too Many Requests
in 165.38µs</font>
<br><font size="2" face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size="2" face="sans-serif">[heketi] WARNING 2019/01/23 12:09:04
operations in-flight (8) exceeds limit (8)</font>
<br><font size="2" face="sans-serif">[negroni] Completed 429 Too Many Requests
in 488.253µs</font>
<br><font size="2" face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size="2" face="sans-serif">[heketi] WARNING 2019/01/23 12:09:04
operations in-flight (8) exceeds limit (8)</font>
<br><font size="2" face="sans-serif">[negroni] Completed 429 Too Many Requests
in 171.836µs</font>
<br><font size="2" face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size="2" face="sans-serif">[heketi] WARNING 2019/01/23 12:09:19
operations in-flight (8) exceeds limit (8)</font>
<br><font size="2" face="sans-serif">[negroni] Completed 429 Too Many Requests
in 208.59µs</font>
<br><font size="2" face="sans-serif">[negroni] Started POST /volumes</font>
<br><font size="2" face="sans-serif">[heketi] WARNING 2019/01/23 12:09:19
operations in-flight (8) exceeds limit (8)</font>
<br><font size="2" face="sans-serif">[negroni] Completed 429 Too Many Requests
in 125.141µs</font>
<br><font size="2" face="sans-serif">[negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606</font>
<br><font size="2" face="sans-serif">[negroni] Completed 404 Not Found in
138.687µs</font>
<br><font size="2" face="sans-serif">[negroni] Started POST /volumes</font>
<br>
<br>
<br><font size="2" face="sans-serif">BR</font>
<br><font size="2" face="sans-serif">Salam</font>
<br><p>=====-----=====-----=====<br>
Notice: The information contained in this e-mail<br>
message and/or attachments to it may contain <br>
confidential or privileged information. If you are <br>
not the intended recipient, any dissemination, use, <br>
review, distribution, printing or copying of the <br>
information contained in this e-mail message <br>
and/or attachments to it are strictly prohibited. If <br>
you have received this communication in error, <br>
please notify us by reply e-mail or telephone and <br>
immediately and permanently delete the message <br>
and any attachments. Thank you</p>
<p></p></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>