[Gluster-devel] Proposal for 3.5: Make adding custom translators easy

[Jeff Darcy]

On 08/12/2013 01:52 PM, Justin Clift wrote:
> For 3.5, I'd like to propose we add a way for people to easily add custom
> translators they've written. (using C, or Glupy, or whatever)
>
> I'm not actually sure (yet) how we'd go about doing it, so would like to
> discuss it first before writing things up. :)
>
> As a first thought, maybe we could provide some CLI tooling to enable this:
>
> eg: To add a native server side translator written using Glupy, after the
> marker translator:
>
> $ sudo gluster volume $VOLNAME add-translator $TRANSLATOR_NAME \
> $TRANSLATOR_TYPE $FOLLOWING_TRANSLATOR $PATH
>
> Which could be:
>
> $ sudo gluster volume examplevolume add-translator mycustomone server marker
> /home/jc/mycustomone.py
>
> The CLI would see it's a .py file (therefore Glupy), so it would: * Copy the
> .py file to the Glupy sub-dir * Distribute the .py file to the other servers
> of the volume * Ensure the translator is incorporated into the native
> server .vol file after the "marker" translator
>
> It would also need a matching "remove-translator" command.
>
> The downside though... this approach seems like a fairly complicated way of
> doing things.  Not really in line with the concept of keeping Gluster simple
> for admins.
>
> Maybe we should just have a new install Gluster sub-dir instead, which takes
> a .so/.py translator file, and an XML fragment to say what to do with it.
> No CLI.  (would suite deployment via packaging instead)
>
> Thoughts?

To start with, you can include me in the "anything but XML" camp.  ;)

I think we can make things pretty straightforward for the simple one-in one-out
case, like most of the performance translators.  The configuration data you
mention looks pretty good, except that I'd add an extra "above" or "below"
qualifier for maximum control over where new translators are instantiated in
volfiles.  I'd also recommend that the translator itself - not the
configuration snippet - contain version information to ensure that we don't
combine incompatible translators.  That's trivial in Python, and pretty easy
using dlsym and "well known" symbol names in C.

Things get a lot hairier if you want to add support for options that aren't
constant values, or for fan-out translators like DHT and AFR.  Then again,
maybe we should just fall back to the volfile-generation hook feature for that.