[Gluster-devel] [Heketi] Block store related API design discussion
Luis Pabón
lpabon at redhat.com
Tue Sep 20 13:32:08 UTC 2016
Awesome, Thanks guys.
- Luis
----- Original Message -----
From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
To: "Niels de Vos" <ndevos at redhat.com>
Cc: "Luis Pabón" <lpabon at redhat.com>, "gluster-devel" <gluster-devel at gluster.org>, "Stephen Watt" <swatt at redhat.com>, "Ramakrishna Yekulla" <rreddy at redhat.com>, "Humble Chirammal" <hchiramm at redhat.com>
Sent: Tuesday, September 20, 2016 5:53:30 AM
Subject: Re: [Gluster-devel] [Heketi] Block store related API design discussion
On Mon, Sep 19, 2016 at 9:22 PM, Niels de Vos <ndevos at redhat.com> wrote:
> On Mon, Sep 19, 2016 at 10:31:11AM -0400, Luis Pabón wrote:
> > Using qemu is interesting, but the I/O should be using the IO path of
> QEMU block API. If not,
> > TCMU would not know how to work with QEMU dynamic QCOW2 files.
> >
> > Now, if TCMU already has this, then that would be great!
>
> It has a qcow2 header, maybe you guys are lucky!
> https://github.com/open-iscsi/tcmu-runner/blob/master/qcow2.h
Sent the earlier mail before seeing this mail :-). So yes, what we
discussed is to see if this qemu in tcmu can internally use gfapi for doing
the operations or not is something we are trying to find out.
>
>
> Niels
>
> >
> > - Luis
> >
> > ----- Original Message -----
> > From: "Prasanna Kalever" <pkalever at redhat.com>
> > To: "Niels de Vos" <ndevos at redhat.com>
> > Cc: "Luis Pabón" <lpabon at redhat.com>, "Stephen Watt" <swatt at redhat.com>,
> "gluster-devel" <gluster-devel at gluster.org>, "Ramakrishna Yekulla" <
> rreddy at redhat.com>, "Humble Chirammal" <hchiramm at redhat.com>
> > Sent: Monday, September 19, 2016 7:13:36 AM
> > Subject: Re: [Gluster-devel] [Heketi] Block store related API design
> discussion
> >
> > On Mon, Sep 19, 2016 at 4:09 PM, Niels de Vos <ndevos at redhat.com> wrote:
> > >
> > > On Mon, Sep 19, 2016 at 03:34:29PM +0530, Prasanna Kalever wrote:
> > > > On Mon, Sep 19, 2016 at 10:13 AM, Niels de Vos <ndevos at redhat.com>
> wrote:
> > > > > On Tue, Sep 13, 2016 at 12:06:00PM -0400, Luis Pabón wrote:
> > > > >> Very good points. Thanks Prasanna for putting this together. I
> agree with
> > > > >> your comments in that Heketi is the high level abstraction API
> and it should have
> > > > >> an API similar of what is described by Prasanna.
> > > > >>
> > > > >> I definitely do not think any File Api should be available in
> Heketi,
> > > > >> because that is an implementation of the Block API. The Heketi
> API should
> > > > >> be similar to something like OpenStack Cinder.
> > > > >>
> > > > >> I think that the actual management of the Volumes used for Block
> storage
> > > > >> and the files in them should be all managed by Heketi. How they
> are
> > > > >> actually created is still to be determined, but we could have
> Heketi
> > > > >> create them, or have helper programs do that.
> > > > >
> > > > > Maybe a tool like qemu-img? If whatever iscsi service understand
> the
> > > > > format (at the very least 'raw'), you could get functionality like
> > > > > snapshots pretty simple.
> > > >
> > > > Niels,
> > > >
> > > > This is brilliant and subset of the Idea falls in one among my
> > > > thoughts, only concern is about building dependencies of qemu with
> > > > Heketi.
> > > > But at an advantage of easy and cool snapshots solution.
> > >
> > > And well tested as I understand that oVirt is moving to use qemu-img as
> > > well. Other tools are able to use the qcow2 format, maybe the iscsi
> > > servce that gets used does so too.
> > >
> > > Has there already been a decision on what Heketi will configure as
> iscsi
> > > service? I am aware of the tgt [1] and LIO/TCMU [2] projects.
> >
> > Niels,
> >
> > yes we will be using TCMU (Kernel Module) and TCMU-runner (user space
> > service) to expose file in Gluster volume as an iSCSI target.
> > more at [1], [2] & [3]
> >
> > [1] https://pkalever.wordpress.com/2016/06/23/gluster-
> solution-for-non-shared-persistent-storage-in-docker-container/
> > [2] https://pkalever.wordpress.com/2016/06/29/non-shared-
> persistent-gluster-storage-with-kubernetes/
> > [3] https://pkalever.wordpress.com/2016/08/16/read-write-
> once-persistent-storage-for-openshift-origin-using-gluster/
> >
> > --
> > Prasanna
> >
> > >
> > > Niels
> > >
> > > 1. http://stgt.sourceforge.net/
> > > 2. https://github.com/open-iscsi/tcmu-runner
> > > http://blog.gluster.org/2016/04/using-lio-with-gluster/
> > >
> > > >
> > > > --
> > > > Prasanna
> > > >
> > > > >
> > > > > Niels
> > > > >
> > > > >
> > > > >> We also need to document the exact workflow to enable a file in
> > > > >> a Gluster volume to be exposed as a block device. This will help
> > > > >> determine where the creation of the file could take place.
> > > > >>
> > > > >> We can capture our decisions from these discussions in the
> > > > >> following page:
> > > > >>
> > > > >> https://github.com/heketi/heketi/wiki/Proposed-Changes
> > > > >>
> > > > >> - Luis
> > > > >>
> > > > >>
> > > > >> ----- Original Message -----
> > > > >> From: "Humble Chirammal" <hchiramm at redhat.com>
> > > > >> To: "Raghavendra Talur" <rtalur at redhat.com>
> > > > >> Cc: "Prasanna Kalever" <pkalever at redhat.com>, "gluster-devel" <
> gluster-devel at gluster.org>, "Stephen Watt" <swatt at redhat.com>, "Luis
> Pabon" <lpabon at redhat.com>, "Michael Adam" <madam at redhat.com>,
> "Ramakrishna Yekulla" <rreddy at redhat.com>, "Mohamed Ashiq Liyazudeen" <
> mliyazud at redhat.com>
> > > > >> Sent: Tuesday, September 13, 2016 2:23:39 AM
> > > > >> Subject: Re: [Gluster-devel] [Heketi] Block store related API
> design discussion
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> ----- Original Message -----
> > > > >> | From: "Raghavendra Talur" <rtalur at redhat.com>
> > > > >> | To: "Prasanna Kalever" <pkalever at redhat.com>
> > > > >> | Cc: "gluster-devel" <gluster-devel at gluster.org>, "Stephen
> Watt" <swatt at redhat.com>, "Luis Pabon" <lpabon at redhat.com>,
> > > > >> | "Michael Adam" <madam at redhat.com>, "Humble Chirammal" <
> hchiramm at redhat.com>, "Ramakrishna Yekulla"
> > > > >> | <rreddy at redhat.com>, "Mohamed Ashiq Liyazudeen" <
> mliyazud at redhat.com>
> > > > >> | Sent: Tuesday, September 13, 2016 11:08:44 AM
> > > > >> | Subject: Re: [Gluster-devel] [Heketi] Block store related API
> design discussion
> > > > >> |
> > > > >> | On Mon, Sep 12, 2016 at 11:30 PM, Prasanna Kalever <
> pkalever at redhat.com>
> > > > >> | wrote:
> > > > >> |
> > > > >> | > Hi all,
> > > > >> | >
> > > > >> | > This mail is open for discussion on gluster block store
> integration with
> > > > >> | > heketi and its REST API interface design constraints.
> > > > >> | >
> > > > >> | >
> > > > >> | > ___ Volume Request ...
> > > > >> | > |
> > > > >> | > |
> > > > >> | > PVC claim -> Heketi --->|
> > > > >> | > |
> > > > >> | > |
> > > > >> | > |
> > > > >> | > |
> > > > >> | > | __
> BlockCreate
> > > > >> | > | |
> > > > >> | > | |__
> BlockInfo
> > > > >> | > | |
> > > > >> | > |___ Block Request (APIS)-> |__
> BlockResize
> > > > >> | > |
> > > > >> | > |__
> BlockList
> > > > >> | > |
> > > > >> | > |__
> BlockDelete
> > > > >> | >
> > > > >> | > Heketi will have block API and volume API, when user submit a
> Persistent
> > > > >> | > volume claim, Kubernetes provisioner based on the storage
> class(from PVC)
> > > > >> | > talks to heketi for storage, heketi intern calls block or
> volume API's
> > > > >> | > based on request.
> > > > >> | >
> > > > >> |
> > > > >> | This is probably wrong. It won't be Heketi calling block or
> volume APIs. It
> > > > >> | would be Kubernetes calling block or volume API *of* Heketi.
> > > > >> |
> > > > >> |
> > > > >> | > With my limited understanding, heketi currently creates
> clusters from
> > > > >> | > provided nodes, creates volumes and handover them to the user.
> > > > >> | > For block related API's, it has to deal with files right ?
> > > > >> | >
> > > > >> | > Here is how block API's look like in short-
> > > > >> | > Create: heketi has to create file in the volume and export it
> as a iscsi
> > > > >> | > target device and hand it over to user.
> > > > >> | > Info: show block stores information across all the clusters,
> connection
> > > > >> | > info, size etc.
> > > > >> | > resize: resize the file in the volume, refresh connections
> from initiator
> > > > >> | > side
> > > > >> | > List: List the connections
> > > > >> | > Delete: logout the connections and delete the file in the
> gluster volume
> > > > >> | >
> > > > >> | > Couple of questions:
> > > > >> | > 1. Should Block API have sub API's such as FileCreate,
> FileList,
> > > > >> | > FileResize, File delete and etc then get it used in Block API
> as they
> > > > >> | > mostly deal with files.
> > > > >> | >
> > > > >> |
> > > > >> | IMO, Heketi should not expose any File related API. It should
> only have
> > > > >> | APIs to service request for block devices; how the block
> devices are
> > > > >> | created and modified is an implementation detail.
> > > > >> |
> > > > >> |
> > > > >> | > 2. How do we create the actual file in the volume, meaning
> using FUSE
> > > > >> | > mount (which may involve an extra process running) or gfapi,
> again if gfapi
> > > > >> | > should we go with c API's, python bindings or go bindings ?
> > > > >> | >
> > > > >> | 3. Should we get targetcli related (LUN exporting) setup done
> from heketi
> > > > >> | > or do we seek help from gdeploy for this ?
> > > > >> | >
> > > > >> |
> > > > >> | I would prefer to either have it in Heketi or in Kubernetes. If
> the API in
> > > > >> | Heketi promises just the creation of block device, then the
> rest of the
> > > > >> | implementation should be in Kubernetes(the export part). If the
> API in
> > > > >> | Heketi promises create and export both, I would say Heketi
> should have the
> > > > >> | implementation within itself.
> > > > >> |
> > > > >> |
> > > > >>
> > > > >> IMO, we should not think about how the clients ( ex: k8s) use it,
> because there may be different clients.
> > > > >> We should concentrate mainly on 'in/out' of block API in Heketi.
> Regardless of which client, the API should act the same way.
> > > > >>
> > > > >> --Humble
> > > > >> _______________________________________________
> > > > >> Gluster-devel mailing list
> > > > >> Gluster-devel at gluster.org
> > > > >> http://www.gluster.org/mailman/listinfo/gluster-devel
> > > > >
> > > > > _______________________________________________
> > > > > Gluster-devel mailing list
> > > > > Gluster-devel at gluster.org
> > > > > http://www.gluster.org/mailman/listinfo/gluster-devel
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
--
Pranith
More information about the Gluster-devel
mailing list