[heketi-devel] Strange behaviour with Kubernetes StatefulSet and dynamic provisioning

Humble Chirammal hchiramm at redhat.com
Fri Aug 18 06:39:26 UTC 2017


Hi Christopher,

I believe this is what happened here:

heketi placed a request to gluster to create a volume which was succesful,
once its done, heketi will issue volume start command and this command
failed.  For heketi, this is 'one transaction' which is volume creation and
it "failed", so it would have tried to "delete the volume" ( this command
would have failed in gluster too) created just now and will also wipe the
DB entry from heketi.

Thats the reason for heketi not aware of this volume atm and volume still
exist in the backend.

>From the gluster error, it looks like "glusterd" is  busy atm to serve the
request: " Another transaction is in progress Please try again after
sometime.."

I dont know what caused this ( may be many requests concurrent requests in
glustrd or slowness in executing gluster commands..etc) and the setup much.
Yeah, gluster and heketi logs could tell.




On Fri, Aug 18, 2017 at 11:55 AM, Luis Pabon <lpabon at chrysalix.org> wrote:

> That is strange. I wonder if the volume creation was successful. Can show
> attach the logs for Heketi?
>
> - Luis
>
> On Thu, Aug 17, 2017 at 8:31 AM, Christopher Schmidt <fakod666 at gmail.com>
> wrote:
>
>>
>> Hi all,
>>
>> I've just tried Prometheus on Kubernetes, using
>> - CoreOS Prometheus Operator (0.10.2)
>> - K8s 1.6.8
>> - GlusterFS 3.10.4 (disabled all performance translators)
>> - Heketi 5
>>
>> The Operator created a StatefulSet with a VolumeClaimTemplate prometheus
>> -prometheus-db-prometheus-prometheus-0
>> As you can see below, the claim has issues creating volumes.
>> But actually, the volumes are created. LoggedIn in my GlusterFS Container
>> I can see f.e. vol_9903142828b2954544430d0f9dd385bf but not in Heketi
>>
>> Any idea how this can happen?
>>
>> best Christopher
>>
>>
>>
>>
>> > kubectl -n metrics describe pvc prometheus-prometheus-db-prome
>> theus-prometheus-0
>> Name: prometheus-prometheus-db-prometheus-prometheus-0
>> Namespace: metrics
>> StorageClass: distributed
>> Status: Pending
>> Volume:
>> Labels: app=prometheus
>> prometheus=prometheus
>> Annotations: volume.beta.kubernetes.io/storage-class=distributed
>> volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs
>> Capacity:
>> Access Modes:
>> Events:
>>   FirstSeen LastSeen Count From SubObjectPath Type Reason Message
>>   --------- -------- ----- ---- ------------- -------- ------ -------
>>   39s 39s 1 persistentvolume-controller Warning ProvisioningFailed Failed
>> to provision volume with StorageClass "distributed": glusterfs: create
>> volume err: error creating volume Unable to execute command on
>> glusterfs-vl1tl: volume start: vol_9903142828b2954544430d0f9dd385bf:
>> failed: Another transaction is in progress Please try again after sometime..
>>   28s 28s 1 persistentvolume-controller Warning ProvisioningFailed Failed
>> to provision volume with StorageClass "distributed": glusterfs: create
>> volume err: failed to create endpoint/service <nil>.
>>   15s 15s 1 persistentvolume-controller Warning ProvisioningFailed Failed
>> to provision volume with StorageClass "distributed": glusterfs: create
>> volume err: error creating volume Unable to execute command on
>> glusterfs-vl1tl: volume start: vol_f4e20daae202e6190085fd4bbbe837bb:
>> failed: Another transaction is in progress Please try again after sometime..
>>   0s 0s 1 persistentvolume-controller Warning ProvisioningFailed Failed
>> to provision volume with StorageClass "distributed": glusterfs: create
>> volume err: error creating volume Unable to execute command on
>> glusterfs-4sl4b: volume start: vol_8c913f988caa2328f57cc37e164d7d80:
>> failed: Another transaction is in progress Please try again after sometime..
>>
>> _______________________________________________
>> heketi-devel mailing list
>> heketi-devel at gluster.org
>> http://lists.gluster.org/mailman/listinfo/heketi-devel
>>
>>
>
> _______________________________________________
> heketi-devel mailing list
> heketi-devel at gluster.org
> http://lists.gluster.org/mailman/listinfo/heketi-devel
>
>


-- 
Cheers,
Humble

Red Hat Storage Engineering
Mastering KVM Virtualization: http://amzn.to/2vFTXaW
Website: http://humblec.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/heketi-devel/attachments/20170818/89650ace/attachment.html>


More information about the heketi-devel mailing list