[Gluster-devel] How do we identify peers?

[Anand Avati]

Using UUIDs as the identifier for all internal management communication is
a good idea. The CLI commands could still use hosts or IPs, but that should
be translated to UUIDs as an alias as superficially as possible (ideally in
the CLI itself, glusterd only communicates by UUID). The translation of
UUID to host IPs should really be dynamic/discovery based. Each host must
present all its IPs to all its peers. Volgen and protocol/client must be
enhanced to accept multiple IPs for each brick and auto select the
right/routed address at runtime (or accept a preferred interface as input).

We should even start identifying bricks by UUID and self-discovered
(independent of host and backend mount-point). This way if an EBS volume is
detached and attached to a different node at a different backend mount,
configuration must get auto reconfigured (either completely using udev, or
with partial/limited input from admin). Converting to UUID would be partial
if bricks are not pulled in to the scheme.

Avati


On Thu, Jun 27, 2013 at 2:13 AM, Kaushal M <kshlmster at gmail.com> wrote:

> On Thu, Jun 27, 2013 at 2:30 AM, Joe Julian <joe at julianfamily.org> wrote:
> > On 06/25/2013 11:37 PM, Kaushal M wrote:
> >>
> >> Forgot the link to the bug,
> >> [1] https://bugzilla.redhat.com/show_bug.cgi?id=890502
> >> Comment 2 in particular shows all the described problems
> >>
> >> On Wed, Jun 26, 2013 at 12:06 PM, Kaushal M <kshlmster at gmail.com>
> wrote:
> >>>
> >>> Hi all,
> >>> We have a problem with how we identify peers in GlusterFS which we
> >>> need to solve.
> >>>
> >>> Peers in GlusterFS are currently identified using two keys.
> >>>   - the address of the peer (hostname/ip)
> >>>   - a UUID
> >>> The address is used as an identifier in the gluster cli commands and
> >>> is the key visible to the users of glusterfs. The UUID is used
> >>> internally by glusterd to identify peers.
> >>> This approach works well when we are using glusterfs on systems with
> >>> single network interfaces and when we are using a hostname as the
> >>> address. We run into problems when we multiple network interfaces are
> >>> involved or if hostname and ips are mixed in gluster cli commands.
> >>> Bug-890502 [1] is a good example of the problems we have. This bug
> >>> involves usage of multiple interfaces as well as mixing of
> >>> hostname/ip.
> >>>
> >>> We need to come up with a good solution to this which would solve this.
> >>>
> >>> Some of the solutions that have been thought of are below.
> >>>
> >>> 1. Continue with the current implementation of using UUID internally
> >>> and addresses externally, but provide a good translation interface
> >>> which will translate a given address into UUID correctly. The
> >>> translation interface might involve some network requests which could
> >>> increase the time taken for command execution.
> >>>
> >>> 2. Use the UUID itself both internally and externally, and have a
> >>> command for associating a list of addresses with a UUID. The downside
> >>> is that typing out UUIDs is unwieldy and users will need users to
> >>> execute more commands.
> >>>
> >>> 3. Similar to 2, but use an alias for the peer externally instead of
> >>> the UUID. This will need an additional mechanism to specify the alias
> >>> for a peer. This approach will need users one additional command
> >>> compared to approach 2, but the other commands will be simpler to
> >>> type.
> >>>
> >>>
> >>>
> >>> What are your thoughts about the problem and the proposed solutions?
> >>> If anyone has any questions regarding the problem or the solutions, or
> >>> if there is a better solution, please sound off.
> >>>
> > Here's my opinion on that from pre-acquisition:
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=765437
> >
> > [FEAT] Use uuid in volume info file for servers instead of hostname or ip
> > address
> >
> > In the volume info file, if the servers were referenced by uuid if the
> > hostnames or ip addresses of the hosts change, the bricks would still
> > reference the correct machine.
> >
>
> This is what we want to do, stop using addresses to identify peers.
> We use UUIDs internally, but its usage is inconsistent. There are
> places where we use UUID, as in the peer commands, and places where we
> use addresses, as in the volume commands. Externally at least, we've
> always used addresses, but it has drawbacks which we are trying to
> fix.
> We want to have a consistent way to identify peers everywhere. UUIDs
> do this, but are a pain to use with the commands.
> We could still continue using addresses with commands and have  a
> robust translation layer which will translate any given address to a
> UUID internally. Or we could have a short and unique alias for each
> peer, which has a 1-1 mapping to UUID, which will be used with
> commands. When using UUIDs directly or using aliases we need a way to
> associate addresses with peers, which Justin's concept does.
>
> I'm right now leaning towards a hybrid approach of using a translation
> layer and using connection groups. We could use any of the addresses
> associated with a peer, with the commands. Internally, we will be
> maintaining a list of addresses associated with peers which will make
> the internal translation easier. We will use UUIDs exclusively
> internally, this includes the volfiles and other generated config
> files.
> I'll write a detailed description of this and send it here for comments
> soon.
>
> - Kaushal
>
> > In the IRC channel [on 2011-10-05], we had someone whose network had been
> > renumbered. He had created his volumes using IP addresses instead of
> > hostnames. He was able to peer probe which changed the host information
> to
> > accurately reflect the new ip addresses, but the volumes still pointed at
> > the old addresses. They could not run replace-brick to update the
> addresses
> > of their bricks.
> >
> > If the bricks were referenced by host uuid, the peer probe would have
> > allowed the bricks to resolve to the new information without further
> > changes.
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> https://lists.nongnu.org/mailman/listinfo/gluster-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20130627/ce723645/attachment-0001.html>