[Gluster-devel] Glusterd2 - Some anticipated changes to glusterfs source

[Kaushal M]

On Thu, Aug 3, 2017 at 2:12 PM, Milind Changire <mchangir at redhat.com> wrote:
>
>
> On Thu, Aug 3, 2017 at 12:56 PM, Kaushal M <kshlmster at gmail.com> wrote:
>>
>> On Thu, Aug 3, 2017 at 2:14 AM, Niels de Vos <ndevos at redhat.com> wrote:
>> > On Wed, Aug 02, 2017 at 05:03:35PM +0530, Prashanth Pai wrote:
>> >> Hi all,
>> >>
>> >> The ongoing work on glusterd2 necessitates following non-breaking and
>> >> non-exhaustive list of changes to glusterfs source code:
>> >>
>> >> Port management
>> >> - Remove hard-coding of glusterd's port as 24007 in clients and
>> >> elsewhere.
>> >>   Glusterd2 can be configured to listen to clients on any port (still
>> >> defaults to
>> >>   24007 though)
>> >> - Let the bricks and daemons choose any available port and if needed
>> >> report
>> >>   the port used to glusterd during the "sign in" process. Prasanna has
>> >> a
>> >> patch
>> >>   to do this.
>> >> - Glusterd <--> brick (or any other local daemon) communication should
>> >>   always happen over Unix Domain Socket. Currently glusterd and brick
>> >>   process communicates over UDS and also port 24007. This will allow us
>> >>   to set better authentication and rules for port 24007 as it shall
>> >> only be
>> >> used
>> >>   by clients.
>> >
>> > I prefer this last point to be configurable. At least for debugging we
>> > should be able to capture network traces and display the communication
>> > in Wireshark. Defaulting to UNIX Domain Sockets is fine though.
>>
>> This is the communication between GD2 and bricks, of which there is
>> not a lot happening, and not much to capture.
>> But I agree, it will be nice to have this configurable.
>>
>
> Could glusterd start attempting port binding at 24007 and progress on to
> higher port numbers until successful and register the bound port number with
> rpcbind ? This way the setup will be auto-configurable and admins need not
> scratch their heads to decide upon one port number. Gluster clients could
> always talk to rpcbind on the nodes to get glusterd service port whenever a
> reconnect is required.

24007 has always been used as the GlusterD port. There was a plan to
have it registered with IANA as well.
Having a well defined port is useful to allow proper firewall rules to be setup.

>
>>
>> >
>> >
>> >> Changes to xlator options
>> >> - Xlator authors do not have to modify glusterd2 code to expose new
>> >> xlator
>> >>   options. IOW, glusterd2 will not contain the "glusterd_volopt_map"
>> >> table.
>> >>   Most of its fields will be moved to the xlator itself. Glusterd2 can
>> >> load
>> >>   xlator's shared object and read it's volume_options table. This also
>> >> means
>> >>   xlators have to adhere to some naming conventions for options.
>> >> - Add following additional fields (names are indicative) to
>> >> volume_option_t:
>> >>     - Tag: This is to enable users to list only options having a
>> >> certain
>> >> tag.
>> >>              IOW, it allows us to filter "volume set help" like output.
>> >>              Example of tags: debug, perf, network etc.
>> >>     - Opversion: The minimum (or a range) op-version required by the
>> >> xlator.
>> >>     - Configurable: A bool to indicate whether this option is
>> >> user-configurable.
>> >>                           This may also be clubbed with DOC/NO_DOC
>> >> functionality.
>> >
>> > This is something I have been thinking about to do as well. libgfapi
>> > users would like to list all the valid options before mounting (and
>> > receiving the .vol file) is done. Similar to how many mount options are
>> > set over FUSE, the options should be available through libgfapi.
>> > Hardcoding the options is just wrong, inspecting the available xlators
>> > (.so files) seems to make more sense. Each option would have to describe
>> > if it can be client-side so that we can apply some resonable filters by
>> > default.
>> >
>>
>> Looks like we'd missed this. All the fields available in the vol opt
>> map will move to xlator option tables, including the client flag.
>>
>> > A GitHub Issue with this feature request is at
>> > https://github.com/gluster/glusterfs/issues/263. I appreciate additional
>> > comments and ideas about it :-)
>> >
>>
>> We need to open an issue for our requested changes are well, which
>> will be a superset of this request. We'll make sure to mention this
>> feature request in it.
>> Or we could use a single issue as a tracker for all the xlator option
>> changes, in which case I'd prefer we update the existing issue.
>>
>> >
>> >> - Xlators like AFR, changelog require non-static information such as
>> >> brick
>> >> path
>> >>   to be present in it's options in the volfile. Currently, xlator
>> >> authors
>> >> have
>> >>   to modify glusterd code to get it.
>> >>   This can rather be indicated by the xlator itself using
>> >> templates/placehoders.
>> >>   For example, "changelog-dir" can be set in xlator's option as as
>> >>   <<brick-path>>/.glusterfs/changelogs and then glusterd2 will ensure
>> >> to
>> >> replace
>> >>   <<brick-path>> with actual path during volfile generation.
>> >
>> > I suggest to stick with whatever is a common syntax for other
>> > configuration files that uses placeholders. Maybe just {variable} or
>> > $VARIABLE, the <<variable>> looks a bit awkward.
>>
>> The exact syntax for these variables hasn't been decided yet. But I'm
>> leaning towards '{{ variable }}' used in the Go template package,
>> which is what we'll mostly end up using to implement this
>> functionality.
>>
>> >
>> >
>> >> We'd like to hear your thoughts, suggestions and comments to these
>> >> proposed
>> >> changes.
>> >
>> > Thanks for sharing!
>> > Niels
>> > _______________________________________________
>> > Gluster-devel mailing list
>> > Gluster-devel at gluster.org
>> > http://lists.gluster.org/mailman/listinfo/gluster-devel
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
>
>
> --
> Milind
>