[heketi-devel] Strange behaviour with Kubernetes StatefulSet and dynamic provisioning

[Christopher Schmidt]

Hi, has anyone had time to dig into that?

Is it a slowness or maybe a lock issue of Glusterd or an Heketi issue?

What can I do to find out what can help to increase glusterd performance?
Or can I somehow increase Heketi Timeouts?

best Christopher

Christopher Schmidt <fakod666 at gmail.com> schrieb am Fr., 18. Aug. 2017 um
15:31 Uhr:

> Ok, here are a couple of logs (see ZIP)
>
> Made two things:
> - Create two PVC manually and deleted them
> - Used the CoreOS Prometheus Operator, that creates a statefulset
>
> What I must say is, that creating a Volume takes some time. Replicated
> about a minute or so because there is always another transactions ongoing
>
> I've downgraded to HEKETI 4
> GlusterFS Pods 3.10.5
> K8s 1.6.8
> running on Container Linux
> I disabled all performance translators for every volume
>
> Thanks for helping...
> Christopher
>
> Humble Chirammal <hchiramm at redhat.com> schrieb am Fr., 18. Aug. 2017 um
> 08:39 Uhr:
>
>> Hi Christopher,
>>
>> I believe this is what happened here:
>>
>> heketi placed a request to gluster to create a volume which was
>> succesful, once its done, heketi will issue volume start command and this
>> command failed.  For heketi, this is 'one transaction' which is volume
>> creation and it "failed", so it would have tried to "delete the volume" (
>> this command would have failed in gluster too) created just now and will
>> also wipe the DB entry from heketi.
>>
>> Thats the reason for heketi not aware of this volume atm and volume still
>> exist in the backend.
>>
>> From the gluster error, it looks like "glusterd" is  busy atm to serve
>> the request: " Another transaction is in progress Please try again after
>> sometime.."
>>
>> I dont know what caused this ( may be many requests concurrent requests
>> in glustrd or slowness in executing gluster commands..etc) and the setup
>> much. Yeah, gluster and heketi logs could tell.
>>
>>
>>
>>
>> On Fri, Aug 18, 2017 at 11:55 AM, Luis Pabon <lpabon at chrysalix.org>
>> wrote:
>>
>>> That is strange. I wonder if the volume creation was successful. Can
>>> show attach the logs for Heketi?
>>>
>>> - Luis
>>>
>>> On Thu, Aug 17, 2017 at 8:31 AM, Christopher Schmidt <fakod666 at gmail.com
>>> > wrote:
>>>
>>>>
>>>> Hi all,
>>>>
>>>> I've just tried Prometheus on Kubernetes, using
>>>> - CoreOS Prometheus Operator (0.10.2)
>>>> - K8s 1.6.8
>>>> - GlusterFS 3.10.4 (disabled all performance translators)
>>>> - Heketi 5
>>>>
>>>> The Operator created a StatefulSet with a
>>>> VolumeClaimTemplate prometheus-prometheus-db-prometheus-prometheus-0
>>>> As you can see below, the claim has issues creating volumes.
>>>> But actually, the volumes are created. LoggedIn in my GlusterFS
>>>> Container I can see f.e. vol_9903142828b2954544430d0f9dd385bf but not in
>>>> Heketi
>>>>
>>>> Any idea how this can happen?
>>>>
>>>> best Christopher
>>>>
>>>>
>>>>
>>>>
>>>> > kubectl -n metrics describe pvc
>>>> prometheus-prometheus-db-prometheus-prometheus-0
>>>> Name: prometheus-prometheus-db-prometheus-prometheus-0
>>>> Namespace: metrics
>>>> StorageClass: distributed
>>>> Status: Pending
>>>> Volume:
>>>> Labels: app=prometheus
>>>> prometheus=prometheus
>>>> Annotations: volume.beta.kubernetes.io/storage-class=distributed
>>>> volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs
>>>> Capacity:
>>>> Access Modes:
>>>> Events:
>>>>   FirstSeen LastSeen Count From SubObjectPath Type Reason Message
>>>>   --------- -------- ----- ---- ------------- -------- ------ -------
>>>>   39s 39s 1 persistentvolume-controller Warning ProvisioningFailed Failed
>>>> to provision volume with StorageClass "distributed": glusterfs: create
>>>> volume err: error creating volume Unable to execute command on
>>>> glusterfs-vl1tl: volume start: vol_9903142828b2954544430d0f9dd385bf:
>>>> failed: Another transaction is in progress Please try again after sometime..
>>>>   28s 28s 1 persistentvolume-controller Warning ProvisioningFailed Failed
>>>> to provision volume with StorageClass "distributed": glusterfs: create
>>>> volume err: failed to create endpoint/service <nil>.
>>>>   15s 15s 1 persistentvolume-controller Warning ProvisioningFailed Failed
>>>> to provision volume with StorageClass "distributed": glusterfs: create
>>>> volume err: error creating volume Unable to execute command on
>>>> glusterfs-vl1tl: volume start: vol_f4e20daae202e6190085fd4bbbe837bb:
>>>> failed: Another transaction is in progress Please try again after sometime..
>>>>   0s 0s 1 persistentvolume-controller Warning ProvisioningFailed Failed
>>>> to provision volume with StorageClass "distributed": glusterfs: create
>>>> volume err: error creating volume Unable to execute command on
>>>> glusterfs-4sl4b: volume start: vol_8c913f988caa2328f57cc37e164d7d80:
>>>> failed: Another transaction is in progress Please try again after sometime..
>>>>
>>>> _______________________________________________
>>>> heketi-devel mailing list
>>>> heketi-devel at gluster.org
>>>> http://lists.gluster.org/mailman/listinfo/heketi-devel
>>>>
>>>>
>>>
>>> _______________________________________________
>>> heketi-devel mailing list
>>> heketi-devel at gluster.org
>>> http://lists.gluster.org/mailman/listinfo/heketi-devel
>>>
>>>
>>
>>
>> --
>> Cheers,
>> Humble
>>
>> Red Hat Storage Engineering
>> Mastering KVM Virtualization: http://amzn.to/2vFTXaW
>> Website: http://humblec.com
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/heketi-devel/attachments/20170828/4cdc3191/attachment-0001.html>