[Gluster-devel] Glusterd2 - Some anticipated changes to glusterfs source

[Kaushal M]

I've created an issue [1] to request the changes to the xlators.

I've also posted a patch for review [2] which adds the new fields to
the xlator options. The patch is on the experimental branch for now,
but I could just as well post it on master. It doesn't affect any
operations of GD1 or xlators yet.

~kaushal

[1]: https://github.com/gluster/glusterfs/issues/302
[2]: https://review.gluster.org/18050

On Thu, Aug 17, 2017 at 12:46 PM, Kaushal M <kshlmster at gmail.com> wrote:
> On Thu, Aug 3, 2017 at 2:12 PM, Milind Changire <mchangir at redhat.com> wrote:
>>
>>
>> On Thu, Aug 3, 2017 at 12:56 PM, Kaushal M <kshlmster at gmail.com> wrote:
>>>
>>> On Thu, Aug 3, 2017 at 2:14 AM, Niels de Vos <ndevos at redhat.com> wrote:
>>> > On Wed, Aug 02, 2017 at 05:03:35PM +0530, Prashanth Pai wrote:
>>> >> Hi all,
>>> >>
>>> >> The ongoing work on glusterd2 necessitates following non-breaking and
>>> >> non-exhaustive list of changes to glusterfs source code:
>>> >>
>>> >> Port management
>>> >> - Remove hard-coding of glusterd's port as 24007 in clients and
>>> >> elsewhere.
>>> >>   Glusterd2 can be configured to listen to clients on any port (still
>>> >> defaults to
>>> >>   24007 though)
>>> >> - Let the bricks and daemons choose any available port and if needed
>>> >> report
>>> >>   the port used to glusterd during the "sign in" process. Prasanna has
>>> >> a
>>> >> patch
>>> >>   to do this.
>>> >> - Glusterd <--> brick (or any other local daemon) communication should
>>> >>   always happen over Unix Domain Socket. Currently glusterd and brick
>>> >>   process communicates over UDS and also port 24007. This will allow us
>>> >>   to set better authentication and rules for port 24007 as it shall
>>> >> only be
>>> >> used
>>> >>   by clients.
>>> >
>>> > I prefer this last point to be configurable. At least for debugging we
>>> > should be able to capture network traces and display the communication
>>> > in Wireshark. Defaulting to UNIX Domain Sockets is fine though.
>>>
>>> This is the communication between GD2 and bricks, of which there is
>>> not a lot happening, and not much to capture.
>>> But I agree, it will be nice to have this configurable.
>>>
>>
>> Could glusterd start attempting port binding at 24007 and progress on to
>> higher port numbers until successful and register the bound port number with
>> rpcbind ? This way the setup will be auto-configurable and admins need not
>> scratch their heads to decide upon one port number. Gluster clients could
>> always talk to rpcbind on the nodes to get glusterd service port whenever a
>> reconnect is required.
>
> 24007 has always been used as the GlusterD port. There was a plan to
> have it registered with IANA as well.
> Having a well defined port is useful to allow proper firewall rules to be setup.
>
>>
>>>
>>> >
>>> >
>>> >> Changes to xlator options
>>> >> - Xlator authors do not have to modify glusterd2 code to expose new
>>> >> xlator
>>> >>   options. IOW, glusterd2 will not contain the "glusterd_volopt_map"
>>> >> table.
>>> >>   Most of its fields will be moved to the xlator itself. Glusterd2 can
>>> >> load
>>> >>   xlator's shared object and read it's volume_options table. This also
>>> >> means
>>> >>   xlators have to adhere to some naming conventions for options.
>>> >> - Add following additional fields (names are indicative) to
>>> >> volume_option_t:
>>> >>     - Tag: This is to enable users to list only options having a
>>> >> certain
>>> >> tag.
>>> >>              IOW, it allows us to filter "volume set help" like output.
>>> >>              Example of tags: debug, perf, network etc.
>>> >>     - Opversion: The minimum (or a range) op-version required by the
>>> >> xlator.
>>> >>     - Configurable: A bool to indicate whether this option is
>>> >> user-configurable.
>>> >>                           This may also be clubbed with DOC/NO_DOC
>>> >> functionality.
>>> >
>>> > This is something I have been thinking about to do as well. libgfapi
>>> > users would like to list all the valid options before mounting (and
>>> > receiving the .vol file) is done. Similar to how many mount options are
>>> > set over FUSE, the options should be available through libgfapi.
>>> > Hardcoding the options is just wrong, inspecting the available xlators
>>> > (.so files) seems to make more sense. Each option would have to describe
>>> > if it can be client-side so that we can apply some resonable filters by
>>> > default.
>>> >
>>>
>>> Looks like we'd missed this. All the fields available in the vol opt
>>> map will move to xlator option tables, including the client flag.
>>>
>>> > A GitHub Issue with this feature request is at
>>> > https://github.com/gluster/glusterfs/issues/263. I appreciate additional
>>> > comments and ideas about it :-)
>>> >
>>>
>>> We need to open an issue for our requested changes are well, which
>>> will be a superset of this request. We'll make sure to mention this
>>> feature request in it.
>>> Or we could use a single issue as a tracker for all the xlator option
>>> changes, in which case I'd prefer we update the existing issue.
>>>
>>> >
>>> >> - Xlators like AFR, changelog require non-static information such as
>>> >> brick
>>> >> path
>>> >>   to be present in it's options in the volfile. Currently, xlator
>>> >> authors
>>> >> have
>>> >>   to modify glusterd code to get it.
>>> >>   This can rather be indicated by the xlator itself using
>>> >> templates/placehoders.
>>> >>   For example, "changelog-dir" can be set in xlator's option as as
>>> >>   <<brick-path>>/.glusterfs/changelogs and then glusterd2 will ensure
>>> >> to
>>> >> replace
>>> >>   <<brick-path>> with actual path during volfile generation.
>>> >
>>> > I suggest to stick with whatever is a common syntax for other
>>> > configuration files that uses placeholders. Maybe just {variable} or
>>> > $VARIABLE, the <<variable>> looks a bit awkward.
>>>
>>> The exact syntax for these variables hasn't been decided yet. But I'm
>>> leaning towards '{{ variable }}' used in the Go template package,
>>> which is what we'll mostly end up using to implement this
>>> functionality.
>>>
>>> >
>>> >
>>> >> We'd like to hear your thoughts, suggestions and comments to these
>>> >> proposed
>>> >> changes.
>>> >
>>> > Thanks for sharing!
>>> > Niels
>>> > _______________________________________________
>>> > Gluster-devel mailing list
>>> > Gluster-devel at gluster.org
>>> > http://lists.gluster.org/mailman/listinfo/gluster-devel
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>>
>>
>>
>> --
>> Milind
>>