[Gluster-devel] [Heketi] Block store related API design discussion

[Vijay Bellur]

On Tue, Sep 13, 2016 at 12:10 PM, Stephen Watt <swatt at redhat.com> wrote:

>
> Also, some important requirements to figure out/think about are:
>
> - How are you managing locking a block device against a container (or a
> host?)
> - Will your implementation work with OpenShift volume security for block
> devices (FSGroups + Recursive chown, chmod and SELinux labeling)
>
> If these aren't already figured out, would it be possible to create separate
> cards in your trello board so we can track the progress on the resolution of
> these two topics?
>

Tracking through Trello cards [1] and [2].

Thanks,
Vijay

[1] https://trello.com/c/LvfuP1cB/4-read-write-once-limit-a-block-device-to-a-single-container

[2] https://trello.com/c/Ne9NeU2y/43-review-openshift-volume-security-for-block-devices


> On Tue, Sep 13, 2016 at 11:06 AM, Luis Pabón <lpabon at redhat.com> wrote:
>>
>> Very good points.  Thanks Prasanna for putting this together.  I agree
>> with
>> your comments in that Heketi is the high level abstraction API and it
>> should have
>> an API similar of what is described by Prasanna.
>>
>> I definitely do not think any File Api should be available in Heketi,
>> because that is an implementation of the Block API.  The Heketi API should
>> be similar to something like OpenStack Cinder.
>>
>> I think that the actual management of the Volumes used for Block storage
>> and the files in them should be all managed by Heketi.  How they are
>> actually created is still to be determined, but we could have Heketi
>> create them, or have helper programs do that.
>>
>> We also need to document the exact workflow to enable a file in
>> a Gluster volume to be exposed as a block device.  This will help
>> determine where the creation of the file could take place.
>>
>> We can capture our decisions from these discussions in the
>> following page:
>>
>> https://github.com/heketi/heketi/wiki/Proposed-Changes
>>
>> - Luis
>>
>>
>> ----- Original Message -----
>> From: "Humble Chirammal" <hchiramm at redhat.com>
>> To: "Raghavendra Talur" <rtalur at redhat.com>
>> Cc: "Prasanna Kalever" <pkalever at redhat.com>, "gluster-devel"
>> <gluster-devel at gluster.org>, "Stephen Watt" <swatt at redhat.com>, "Luis Pabon"
>> <lpabon at redhat.com>, "Michael Adam" <madam at redhat.com>, "Ramakrishna
>> Yekulla" <rreddy at redhat.com>, "Mohamed Ashiq Liyazudeen"
>> <mliyazud at redhat.com>
>> Sent: Tuesday, September 13, 2016 2:23:39 AM
>> Subject: Re: [Gluster-devel] [Heketi] Block store related API design
>> discussion
>>
>>
>>
>>
>>
>> ----- Original Message -----
>> | From: "Raghavendra Talur" <rtalur at redhat.com>
>> | To: "Prasanna Kalever" <pkalever at redhat.com>
>> | Cc: "gluster-devel" <gluster-devel at gluster.org>, "Stephen Watt"
>> <swatt at redhat.com>, "Luis Pabon" <lpabon at redhat.com>,
>> | "Michael Adam" <madam at redhat.com>, "Humble Chirammal"
>> <hchiramm at redhat.com>, "Ramakrishna Yekulla"
>> | <rreddy at redhat.com>, "Mohamed Ashiq Liyazudeen" <mliyazud at redhat.com>
>> | Sent: Tuesday, September 13, 2016 11:08:44 AM
>> | Subject: Re: [Gluster-devel] [Heketi] Block store related API design
>> discussion
>> |
>> | On Mon, Sep 12, 2016 at 11:30 PM, Prasanna Kalever <pkalever at redhat.com>
>> | wrote:
>> |
>> | > Hi all,
>> | >
>> | > This mail is open for discussion on gluster block store integration
>> with
>> | > heketi and its REST API interface design constraints.
>> | >
>> | >
>> | >                          ___ Volume Request ...
>> | >                         |
>> | >                         |
>> | > PVC claim -> Heketi --->|
>> | >                         |
>> | >                         |
>> | >                         |
>> | >                         |
>> | >                         |                            __ BlockCreate
>> | >                         |                           |
>> | >                         |                           |__ BlockInfo
>> | >                         |                           |
>> | >                         |___ Block Request (APIS)-> |__ BlockResize
>> | >                                                     |
>> | >                                                     |__ BlockList
>> | >                                                     |
>> | >                                                     |__ BlockDelete
>> | >
>> | > Heketi will have block API and volume API, when user submit a
>> Persistent
>> | > volume claim, Kubernetes provisioner based on the storage class(from
>> PVC)
>> | > talks to heketi for storage, heketi intern calls block or volume API's
>> | > based on request.
>> | >
>> |
>> | This is probably wrong. It won't be Heketi calling block or volume APIs.
>> It
>> | would be Kubernetes calling block or volume API *of* Heketi.
>> |
>> |
>> | > With my limited understanding, heketi currently creates clusters from
>> | > provided nodes, creates volumes and handover them to the user.
>> | > For block related API's, it has to deal with files right ?
>> | >
>> | > Here is how block API's look like in short-
>> | > Create: heketi has to create file in the volume and export it as a
>> iscsi
>> | > target device and hand it over to user.
>> | > Info: show block stores information across all the clusters,
>> connection
>> | > info, size etc.
>> | > resize: resize the file in the volume, refresh connections from
>> initiator
>> | > side
>> | > List: List the connections
>> | > Delete: logout the connections and delete the file in the gluster
>> volume
>> | >
>> | > Couple of questions:
>> | > 1. Should Block API have sub API's such as FileCreate, FileList,
>> | > FileResize, File delete and etc then get it used in Block API as they
>> | > mostly deal with files.
>> | >
>> |
>> | IMO, Heketi should not expose any File related API. It should only have
>> | APIs to service request for block devices; how the block devices are
>> | created and modified is an implementation detail.
>> |
>> |
>> | > 2. How do we create the actual file in the volume, meaning using FUSE
>> | > mount (which may involve an extra process running) or gfapi, again if
>> gfapi
>> | > should we go with c API's, python bindings or go bindings ?
>> | >
>> | 3. Should we get targetcli related (LUN exporting) setup done from
>> heketi
>> | > or do we seek help from gdeploy for this ?
>> | >
>> |
>> | I would prefer to either have it in Heketi or in Kubernetes. If the API
>> in
>> | Heketi promises just the creation of block device, then the rest of the
>> | implementation should be in Kubernetes(the export part). If the API in
>> | Heketi promises create and export both, I would say Heketi should have
>> the
>> | implementation within itself.
>> |
>> |
>>
>> IMO, we should not think about how the clients ( ex: k8s) use it, because
>> there may be different clients.
>> We should concentrate mainly on 'in/out' of block API in Heketi.
>> Regardless of which client, the API should act the same way.
>>
>> --Humble
>
>