[Gluster-users] Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: server busy
Madhu Rajanna
mrajanna at redhat.com
Thu Jan 24 08:03:28 UTC 2019
the heketi logs you have attached is not complete i believe, can you
povide the complete heketi logs
and also an we get the output of "ps aux" from the gluster pods ? I want to
see if any lvm commands or gluster commands are "stuck".
On Thu, Jan 24, 2019 at 1:16 PM Shaik Salam <shaik.salam at tcs.com> wrote:
> Hi Madhu.
>
> I tried lot of times restarted heketi pod but not resolved.
>
> sh-4.4# heketi-cli server operations info
> Operation Counts:
> Total: 0
> In-Flight: 0
> New: 0
> Stale: 0
>
> Now you can see all operations are zero. Now I try to create single volume
> below is observation in-flight reaching slowly to 8.
>
> sh-4.4# heketi-cli server operations infoCLI_SERVER=http://localhost:8080
> ; export HEKETI_CLI_USE Operation
> Counts:
> Total: 0
> In-Flight: 6
> New: 0
> Stale: 0
> sh-4.4# heketi-cli server operations info
> Operation Counts:
> Total: 0
> In-Flight: 7
> New: 0
> Stale: 0
> sh-4.4# heketi-cli server operations info
> Operation Counts:
> Total: 0
> In-Flight: 7
> New: 0
> Stale: 0
> sh-4.4# heketi-cli server operations info
> Operation Counts:
> Total: 0
> In-Flight: 7
> New: 0
> Stale: 0
> sh-4.4# heketi-cli server operations info
> Operation Counts:
> Total: 0
> In-Flight: 7
> New: 0
> Stale: 0
>
> [negroni] Completed 200 OK in 186.286µs
> [negroni] Started POST /volumes
> [negroni] Started GET /operations
> [negroni] Completed 200 OK in 166.294µs
> [negroni] Started GET /operations
> [negroni] Completed 200 OK in 186.411µs
> [negroni] Started GET /operations
> [negroni] Completed 200 OK in 179.796µs
> [negroni] Started POST /volumes
> [negroni] Started POST /volumes
> [negroni] Started POST /volumes
> [negroni] Started POST /volumes
> [negroni] Started GET /operations
> [negroni] Completed 200 OK in 131.108µs
> [negroni] Started POST /volumes
> [negroni] Started GET /operations
> [negroni] Completed 200 OK in 111.392µs
> [negroni] Started GET /operations
> [negroni] Completed 200 OK in 265.023µs
> [negroni] Started GET /operations
> [negroni] Completed 200 OK in 179.364µs
> [negroni] Started GET /operations
> [negroni] Completed 200 OK in 295.058µs
> [negroni] Started GET /operations
> [negroni] Completed 200 OK in 146.857µs
> [negroni] Started POST /volumes
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/24 07:43:36 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 403.166µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/24 07:43:51 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 193.554µs
>
>
> But for pod volume is not creating.
>
> 1:15:36 PM
> Warning
> Provisioning failed Failed to provision volume with StorageClass
> "glusterfs-storage": glusterfs: create volume err: error creating volume
> Server busy. Retry operation later..
> 9 times in the last 2 minutes
> 1:13:21 PM
> Warning
> Provisioning failed Failed to provision volume with StorageClass
> "glusterfs-storage": glusterfs: create volume err: error creating volume .
> 8 times in the last
>
>
>
>
> From: "Madhu Rajanna" <mrajanna at redhat.com>
> To: "Shaik Salam" <shaik.salam at tcs.com>
> Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org>,
> "Michael Adam" <madam at redhat.com>
> Date: 01/24/2019 12:51 PM
> Subject: Re: Failed to provision volume with StorageClass
> "glusterfs-storage": glusterfs: server busy
> ------------------------------
>
>
>
> *"External email. Open with Caution"*
> HI Shaik,
>
> can you provide me the outpout of $heketi-cli server operations info
> from heketi pod
>
> as a workround you can try restarting the heketi pod. This will cause the
> current operations to go stale, but other pending pvcs may go to Bound
> state
>
> Regards,
>
> Madhu R
>
> On Thu, Jan 24, 2019 at 12:36 PM Shaik Salam <*shaik.salam at tcs.com*
> <shaik.salam at tcs.com>> wrote:
> H Madhu,
>
> Could you please have look my issue If you have time (atleast workaround).
> I am unable to send mail to "John Mulligan" <*John_Mulligan at redhat.com*
> <John_Mulligan at redhat.com>>" who is currently handling issue
> *https://bugzilla.redhat.com/show_bug.cgi?id=1636912*
> <https://bugzilla.redhat.com/show_bug.cgi?id=1636912>
>
> BR
> Salam
>
>
> From: Shaik Salam/HYD/TCS
> To: "John Mulligan" <*John_Mulligan at redhat.com*
> <John_Mulligan at redhat.com>>, "Michael Adam" <*madam at redhat.com*
> <madam at redhat.com>>, "Madhu Rajanna" <*mrajanna at redhat.com*
> <mrajanna at redhat.com>>
> Cc: "*gluster-users at gluster.org* <gluster-users at gluster.org> List"
> <*gluster-users at gluster.org* <gluster-users at gluster.org>>
> Date: 01/24/2019 12:21 PM
> Subject: Failed to provision volume with StorageClass
> "glusterfs-storage": glusterfs: server busy
> ------------------------------
>
>
>
>
> Hi All,
>
> We are facing also following issue on openshift origin while we are
> creating pvc for pods. (atlease provide workaround to move further)
>
> Failed to provision volume with StorageClass "glusterfs-storage":
> glusterfs: create volume err: error creating volume
> Failed to provision volume with StorageClass "glusterfs-storage":
> glusterfs: create volume err: error creating volume Server busy. Retry
> operation later..
>
> Please find heketidb dump and log
>
> [negroni] Completed 429 Too Many Requests in 250.763µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 169.08µs
> [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c
> [negroni] Completed 404 Not Found in 148.125µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 496.624µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 101.673µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 209.681µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 103.595µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 297.594µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 96.75µs
> [negroni] Started POST /volumes
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 477.007µs
> [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 165.38µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 488.253µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 171.836µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 208.59µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 125.141µs
> [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606
> [negroni] Completed 404 Not Found in 138.687µs
> [negroni] Started POST /volumes
>
>
> BR
> Salam
> =====-----=====-----=====
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
>
>
> --
> Madhu Rajanna
> Software Engineer
> Red Hat Bangalore, India <https://www.redhat.com/>
> mrajanna at redhat.com M: +91-9741133155
>
> <https://red.ht/sig>
>
>
--
Madhu Rajanna
Software Engineer
Red Hat Bangalore, India <https://www.redhat.com>
mrajanna at redhat.com M: +91-9741133155
<https://red.ht/sig>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190124/90927d59/attachment.html>
More information about the Gluster-users
mailing list