[heketi-devel] Strange behaviour with Kubernetes StatefulSet and dynamic provisioning

Fri Aug 18 13:31:26 UTC 2017

Ok, here are a couple of logs (see ZIP)

Made two things:
- Create two PVC manually and deleted them
- Used the CoreOS Prometheus Operator, that creates a statefulset

What I must say is, that creating a Volume takes some time. Replicated
about a minute or so because there is always another transactions ongoing

I've downgraded to HEKETI 4
GlusterFS Pods 3.10.5
K8s 1.6.8
running on Container Linux
I disabled all performance translators for every volume

Thanks for helping...
Christopher

Humble Chirammal <hchiramm at redhat.com> schrieb am Fr., 18. Aug. 2017 um
08:39 Uhr:

> Hi Christopher,
>
> I believe this is what happened here:
>
> heketi placed a request to gluster to create a volume which was succesful,
> once its done, heketi will issue volume start command and this command
> failed.  For heketi, this is 'one transaction' which is volume creation and
> it "failed", so it would have tried to "delete the volume" ( this command
> would have failed in gluster too) created just now and will also wipe the
> DB entry from heketi.
>
> Thats the reason for heketi not aware of this volume atm and volume still
> exist in the backend.
>
> From the gluster error, it looks like "glusterd" is  busy atm to serve the
> request: " Another transaction is in progress Please try again after
> sometime.."
>
> I dont know what caused this ( may be many requests concurrent requests in
> glustrd or slowness in executing gluster commands..etc) and the setup much.
> Yeah, gluster and heketi logs could tell.
>
>
>
>
> On Fri, Aug 18, 2017 at 11:55 AM, Luis Pabon <lpabon at chrysalix.org> wrote:
>
>> That is strange. I wonder if the volume creation was successful. Can show
>> attach the logs for Heketi?
>>
>> - Luis
>>
>> On Thu, Aug 17, 2017 at 8:31 AM, Christopher Schmidt <fakod666 at gmail.com>
>> wrote:
>>
>>>
>>> Hi all,
>>>
>>> I've just tried Prometheus on Kubernetes, using
>>> - CoreOS Prometheus Operator (0.10.2)
>>> - K8s 1.6.8
>>> - GlusterFS 3.10.4 (disabled all performance translators)
>>> - Heketi 5
>>>
>>> The Operator created a StatefulSet with a
>>> VolumeClaimTemplate prometheus-prometheus-db-prometheus-prometheus-0
>>> As you can see below, the claim has issues creating volumes.
>>> But actually, the volumes are created. LoggedIn in my GlusterFS
>>> Container I can see f.e. vol_9903142828b2954544430d0f9dd385bf but not in
>>> Heketi
>>>
>>> Any idea how this can happen?
>>>
>>> best Christopher
>>>
>>>
>>>
>>>
>>> > kubectl -n metrics describe pvc
>>> prometheus-prometheus-db-prometheus-prometheus-0
>>> Name: prometheus-prometheus-db-prometheus-prometheus-0
>>> Namespace: metrics
>>> StorageClass: distributed
>>> Status: Pending
>>> Volume:
>>> Labels: app=prometheus
>>> prometheus=prometheus
>>> Annotations: volume.beta.kubernetes.io/storage-class=distributed
>>> volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs
>>> Capacity:
>>> Access Modes:
>>> Events:
>>>   FirstSeen LastSeen Count From SubObjectPath Type Reason Message
>>>   --------- -------- ----- ---- ------------- -------- ------ -------
>>>   39s 39s 1 persistentvolume-controller Warning ProvisioningFailed Failed
>>> to provision volume with StorageClass "distributed": glusterfs: create
>>> volume err: error creating volume Unable to execute command on
>>> glusterfs-vl1tl: volume start: vol_9903142828b2954544430d0f9dd385bf:
>>> failed: Another transaction is in progress Please try again after sometime..
>>>   28s 28s 1 persistentvolume-controller Warning ProvisioningFailed Failed
>>> to provision volume with StorageClass "distributed": glusterfs: create
>>> volume err: failed to create endpoint/service <nil>.
>>>   15s 15s 1 persistentvolume-controller Warning ProvisioningFailed Failed
>>> to provision volume with StorageClass "distributed": glusterfs: create
>>> volume err: error creating volume Unable to execute command on
>>> glusterfs-vl1tl: volume start: vol_f4e20daae202e6190085fd4bbbe837bb:
>>> failed: Another transaction is in progress Please try again after sometime..
>>>   0s 0s 1 persistentvolume-controller Warning ProvisioningFailed Failed
>>> to provision volume with StorageClass "distributed": glusterfs: create
>>> volume err: error creating volume Unable to execute command on
>>> glusterfs-4sl4b: volume start: vol_8c913f988caa2328f57cc37e164d7d80:
>>> failed: Another transaction is in progress Please try again after sometime..
>>>
>>> _______________________________________________
>>> heketi-devel mailing list
>>> heketi-devel at gluster.org
>>> http://lists.gluster.org/mailman/listinfo/heketi-devel
>>>
>>>
>>
>> _______________________________________________
>> heketi-devel mailing list
>> heketi-devel at gluster.org
>> http://lists.gluster.org/mailman/listinfo/heketi-devel
>>
>>
>
>
> --
> Cheers,
> Humble
>
> Red Hat Storage Engineering
> Mastering KVM Virtualization: http://amzn.to/2vFTXaW
> Website: http://humblec.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/heketi-devel/attachments/20170818/31f7bac1/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: heketi.zip
Type: application/zip
Size: 11651 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/heketi-devel/attachments/20170818/31f7bac1/attachment-0001.zip>