[Gluster-users] Failed to provision volume with StorageClass "glusterfs-storage": glusterfs: server busy
Madhu Rajanna
mrajanna at redhat.com
Thu Jan 24 17:21:45 UTC 2019
Adding John who is having more idea about how to debug this one.
@Shaik Salam <shaik.salam at tcs.com> can you some more info on the hardware
on which you are running heketi (kernel details)
On Thu, Jan 24, 2019 at 7:42 PM Shaik Salam <shaik.salam at tcs.com> wrote:
> Hi Madhu,
>
> Sorry to disturb could you please provide atleast work around (to clear
> requests which stuck) to move further.
> We are also not able to find root cause from glusterd logs. Please find
> attachment.
>
> BR
> Salam
>
>
>
>
>
> From: Shaik Salam/HYD/TCS
> To: "Madhu Rajanna" <mrajanna at redhat.com>
> Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org>,
> "Michael Adam" <madam at redhat.com>
> Date: 01/24/2019 04:12 PM
> Subject: Re: Failed to provision volume with StorageClass
> "glusterfs-storage": glusterfs: server busy
> ------------------------------
>
>
> Hi Madhu,
>
> Please let me know If any other information required.
>
> BR
> Salam
>
>
>
>
> From: Shaik Salam/HYD/TCS
> To: "Madhu Rajanna" <mrajanna at redhat.com>
> Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org>,
> "Michael Adam" <madam at redhat.com>
> Date: 01/24/2019 03:23 PM
> Subject: Re: Failed to provision volume with StorageClass
> "glusterfs-storage": glusterfs: server busy
> ------------------------------
>
>
> Hi Madhu,
>
> This is complete one after restart of heketi pod and process log.
>
> BR
> Salam
>
> [attachment "heketi-pod-complete.log" deleted by Shaik Salam/HYD/TCS]
> [attachment "ps-aux.txt" deleted by Shaik Salam/HYD/TCS]
>
>
>
>
> From: "Madhu Rajanna" <mrajanna at redhat.com>
> To: "Shaik Salam" <shaik.salam at tcs.com>
> Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org>,
> "Michael Adam" <madam at redhat.com>
> Date: 01/24/2019 01:55 PM
> Subject: Re: Failed to provision volume with StorageClass
> "glusterfs-storage": glusterfs: server busy
> ------------------------------
>
>
>
> *"External email. Open with Caution"*
> the logs you provided is not complete, not able to figure out which
> command is struck, can you reattach the complete output of `ps aux` and
> also attach complete heketi logs.
>
> On Thu, Jan 24, 2019 at 1:41 PM Shaik Salam <*shaik.salam at tcs.com*
> <shaik.salam at tcs.com>> wrote:
> Hi Madhu,
>
> Please find requested info.
>
> BR
> Salam
>
>
>
>
>
> From: Madhu Rajanna <*mrajanna at redhat.com* <mrajanna at redhat.com>>
> To: Shaik Salam <*shaik.salam at tcs.com* <shaik.salam at tcs.com>>
> Cc: "*gluster-users at gluster.org* <gluster-users at gluster.org> List"
> <*gluster-users at gluster.org* <gluster-users at gluster.org>>, Michael Adam <
> *madam at redhat.com* <madam at redhat.com>>
> Date: 01/24/2019 01:33 PM
> Subject: Re: Failed to provision volume with StorageClass
> "glusterfs-storage": glusterfs: server busy
> ------------------------------
>
>
>
> * "External email. Open with Caution"*
> the heketi logs you have attached is not complete i believe, can you
> povide the complete heketi logs
> and also an we get the output of "ps aux" from the gluster pods ? I want
> to see if any lvm commands or gluster commands are "stuck".
>
>
> On Thu, Jan 24, 2019 at 1:16 PM Shaik Salam <*shaik.salam at tcs.com*
> <shaik.salam at tcs.com>> wrote:
> Hi Madhu.
>
> I tried lot of times restarted heketi pod but not resolved.
>
> sh-4.4# heketi-cli server operations info
> Operation Counts:
> Total: 0
> In-Flight: 0
> New: 0
> Stale: 0
>
> Now you can see all operations are zero. Now I try to create single volume
> below is observation in-flight reaching slowly to 8.
>
> sh-4.4# heketi-cli server operations infoCLI_SERVER=
> *http://localhost:8080* <http://localhost:8080/> ; export HEKETI_CLI_USE
> Operation Counts:
> Total: 0
> In-Flight: 6
> New: 0
> Stale: 0
> sh-4.4# heketi-cli server operations info
> Operation Counts:
> Total: 0
> In-Flight: 7
> New: 0
> Stale: 0
> sh-4.4# heketi-cli server operations info
> Operation Counts:
> Total: 0
> In-Flight: 7
> New: 0
> Stale: 0
> sh-4.4# heketi-cli server operations info
> Operation Counts:
> Total: 0
> In-Flight: 7
> New: 0
> Stale: 0
> sh-4.4# heketi-cli server operations info
> Operation Counts:
> Total: 0
> In-Flight: 7
> New: 0
> Stale: 0
>
> [negroni] Completed 200 OK in 186.286µs
> [negroni] Started POST /volumes
> [negroni] Started GET /operations
> [negroni] Completed 200 OK in 166.294µs
> [negroni] Started GET /operations
> [negroni] Completed 200 OK in 186.411µs
> [negroni] Started GET /operations
> [negroni] Completed 200 OK in 179.796µs
> [negroni] Started POST /volumes
> [negroni] Started POST /volumes
> [negroni] Started POST /volumes
> [negroni] Started POST /volumes
> [negroni] Started GET /operations
> [negroni] Completed 200 OK in 131.108µs
> [negroni] Started POST /volumes
> [negroni] Started GET /operations
> [negroni] Completed 200 OK in 111.392µs
> [negroni] Started GET /operations
> [negroni] Completed 200 OK in 265.023µs
> [negroni] Started GET /operations
> [negroni] Completed 200 OK in 179.364µs
> [negroni] Started GET /operations
> [negroni] Completed 200 OK in 295.058µs
> [negroni] Started GET /operations
> [negroni] Completed 200 OK in 146.857µs
> [negroni] Started POST /volumes
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/24 07:43:36 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 403.166µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/24 07:43:51 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 193.554µs
>
>
> But for pod volume is not creating.
> 1:15:36 PM
> Warning
> Provisioning failed Failed to provision volume with StorageClass
> "glusterfs-storage": glusterfs: create volume err: error creating volume
> Server busy. Retry operation later..
> 9 times in the last 2 minutes
> 1:13:21 PM
> Warning
> Provisioning failed Failed to provision volume with StorageClass
> "glusterfs-storage": glusterfs: create volume err: error creating volume .
> 8 times in the last
>
>
>
>
>
>
> From: "Madhu Rajanna" <*mrajanna at redhat.com* <mrajanna at redhat.com>>
> To: "Shaik Salam" <*shaik.salam at tcs.com* <shaik.salam at tcs.com>>
> Cc: "*gluster-users at gluster.org* <gluster-users at gluster.org> List"
> <*gluster-users at gluster.org* <gluster-users at gluster.org>>, "Michael Adam"
> <*madam at redhat.com* <madam at redhat.com>>
> Date: 01/24/2019 12:51 PM
> Subject: Re: Failed to provision volume with StorageClass
> "glusterfs-storage": glusterfs: server busy
> ------------------------------
>
>
>
> * "External email. Open with Caution"*
> HI Shaik,
>
> can you provide me the outpout of $heketi-cli server operations info
> from heketi pod
>
> as a workround you can try restarting the heketi pod. This will cause the
> current operations to go stale, but other pending pvcs may go to Bound
> state
>
> Regards,
>
> Madhu R
>
> On Thu, Jan 24, 2019 at 12:36 PM Shaik Salam <*shaik.salam at tcs.com*
> <shaik.salam at tcs.com>> wrote:
> H Madhu,
>
> Could you please have look my issue If you have time (atleast workaround).
> I am unable to send mail to "John Mulligan" <*John_Mulligan at redhat.com*
> <John_Mulligan at redhat.com>>" who is currently handling issue
> *https://bugzilla.redhat.com/show_bug.cgi?id=1636912*
> <https://bugzilla.redhat.com/show_bug.cgi?id=1636912>
>
> BR
> Salam
>
>
> From: Shaik Salam/HYD/TCS
> To: "John Mulligan" <*John_Mulligan at redhat.com*
> <John_Mulligan at redhat.com>>, "Michael Adam" <*madam at redhat.com*
> <madam at redhat.com>>, "Madhu Rajanna" <*mrajanna at redhat.com*
> <mrajanna at redhat.com>>
> Cc: "*gluster-users at gluster.org* <gluster-users at gluster.org> List"
> <*gluster-users at gluster.org* <gluster-users at gluster.org>>
> Date: 01/24/2019 12:21 PM
> Subject: Failed to provision volume with StorageClass
> "glusterfs-storage": glusterfs: server busy
>
> ------------------------------
>
>
>
>
> Hi All,
>
> We are facing also following issue on openshift origin while we are
> creating pvc for pods. (atlease provide workaround to move further)
>
> Failed to provision volume with StorageClass "glusterfs-storage":
> glusterfs: create volume err: error creating volume
> Failed to provision volume with StorageClass "glusterfs-storage":
> glusterfs: create volume err: error creating volume Server busy. Retry
> operation later..
>
> Please find heketidb dump and log
>
> [negroni] Completed 429 Too Many Requests in 250.763µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:07:49 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 169.08µs
> [negroni] Started DELETE /volumes/520bc5f4e1bfd029855a72f9ca7ebf6c
> [negroni] Completed 404 Not Found in 148.125µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 496.624µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:08:04 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 101.673µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 209.681µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:08:19 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 103.595µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 297.594µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:08:34 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 96.75µs
> [negroni] Started POST /volumes
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 477.007µs
> [heketi] WARNING 2019/01/23 12:08:49 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 165.38µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 488.253µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:09:04 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 171.836µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 208.59µs
> [negroni] Started POST /volumes
> [heketi] WARNING 2019/01/23 12:09:19 operations in-flight (8) exceeds
> limit (8)
> [negroni] Completed 429 Too Many Requests in 125.141µs
> [negroni] Started DELETE /volumes/99e87ecd0a816ac34ae5a04eabc1d606
> [negroni] Completed 404 Not Found in 138.687µs
> [negroni] Started POST /volumes
>
>
> BR
> Salam
> =====-----=====-----=====
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
>
>
> --
> Madhu Rajanna
> Software Engineer
> *Red Hat Bangalore, India* <https://www.redhat.com/>
> *mrajanna at redhat.com* <mrajanna at redhat.com> M: +91-9741133155
>
> <https://red.ht/sig>
>
>
>
> --
> Madhu Rajanna
> Software Engineer
> *Red Hat Bangalore, India* <https://www.redhat.com/>
> *mrajanna at redhat.com* <mrajanna at redhat.com> M: +91-9741133155
>
> <https://red.ht/sig>
>
>
>
> --
> Madhu Rajanna
> Software Engineer
> Red Hat Bangalore, India <https://www.redhat.com/>
> mrajanna at redhat.com M: +91-9741133155
>
> <https://red.ht/sig>
>
>
--
Madhu Rajanna
Software Engineer
Red Hat Bangalore, India <https://www.redhat.com>
mrajanna at redhat.com M: +91-9741133155
<https://red.ht/sig>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190124/4f425ffe/attachment.html>
More information about the Gluster-users
mailing list