[Bugs] Unable to create new volume due to pending operations

[Shaik Salam]

Hi,

We have deployed glustrerfs as containers on openshift orgin.

We are unable to create new volume for opeshift pods observed following 
error.

Failed to provision volume with StorageClass "glusterfs-storage": 
glusterfs: create volume err: error creating volume 
Failed to provision volume with StorageClass "glusterfs-storage": 
glusterfs: create volume err: error creating volume Server busy. Retry 
operation later.. 

We could observe above issue due to pending operations on heketi db.
We have exported db and removed pending operations (volumes,bricks) on db, 
lvs from physical hosts and imported.
But heketi trying to delete volumes which are removed from heketi as part 
of pending operations in db and still we are unable to create volumes 
facing server busy.
Can you please let me know still heketi where it is getting volumes id 
which are not available on heketi deb??
why we are unable to create volumes ?? (no info on glusterd,glusterfsd 
logs)??


[negroni] Started POST /volumes 


[heketi] WARNING 2019/01/21 11:48:31 operations in-flight (8) exceeds 
limit (8) 

[negroni] Completed 429 Too Many Requests in 260.577µs 

[negroni] Started POST /volumes 

[heketi] WARNING 2019/01/21 11:48:39 operations in-flight (8) exceeds 
limit (8) 

[negroni] Completed 429 Too Many Requests in 221.477µs 

[negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 

[negroni] Completed 404 Not Found in 151.896µs 

[negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 

[negroni] Completed 404 Not Found in 125.387µs 

[negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c 

[negroni] Completed 404 Not Found in 168.23µs 

[negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 

[negroni] Completed 404 Not Found in 123.231µs 

[negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c 

[negroni] Completed 404 Not Found in 160.416µs 

[negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 

[negroni] Completed 404 Not Found in 124.439µs 

[negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c 

[negroni] Completed 404 Not Found in 126.748µs 

[negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606 

[negroni] Completed 404 Not Found in 138.377µs 

[negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c 

[negroni] Completed 404 Not Found in 138.477µs 

[negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c 

[negroni] Completed 404 Not Found in 267.79µs


BR
Salam
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/bugs/attachments/20190121/726cfc5c/attachment-0001.html>