[Bugs] [Bug 1310445] New: Gluster not resolving hosts with IPv6 only lookups

Sun Feb 21 16:19:32 UTC 2016

https://bugzilla.redhat.com/show_bug.cgi?id=1310445

            Bug ID: 1310445
           Summary: Gluster not resolving hosts with IPv6 only lookups
           Product: GlusterFS
           Version: 3.7.7
         Component: glusterd
          Keywords: Reopened
          Severity: medium
          Priority: medium
          Assignee: bugs at gluster.org
          Reporter: amukherj at redhat.com
                CC: amukherj at redhat.com, anders.blomdell at control.lth.se,
                    anoopcs at redhat.com, bloch at verdurin.com,
                    bugs at gluster.org, gluster-bugs at redhat.com,
                    joachim at tingvold.com, joe at julianfamily.org,
                    kaushal at redhat.com, kwade at redhat.com,
                    nithind1988 at yahoo.in, ueberall at projektzentrisch.de,
                    zhougulf at gmail.com
        Depends On: 1117886
            Blocks: 1117822 (glusterfs-3.6.0)

+++ This bug was initially created as a clone of Bug #1117886 +++

+++ This bug was initially created as a clone of Bug #922801 +++

Description of problem:

  As reported on IRC by alex88, gluster isn't resolving hosts that have
  only IPv6 DNS entries (ie AAAA instead of A).

    [2013-03-18 10:23:42.211253] I [glusterfsd.c:1666:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.1
    [2013-03-18 10:23:42.213462] E [common-utils.c:125:gf_resolve_ip6]
0-resolver: getaddrinfo failed (Name or service not known)
    [2013-03-18 10:23:42.213537] E
[name.c:243:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution
failed on host storage.site.com
    [2013-03-18 10:23:42.213609] E [glusterfsd-mgmt.c:1787:mgmt_rpc_notify]
0-glusterfsd-mgmt: failed to connect with remote-host: Success
    [2013-03-18 10:23:42.213649] I [glusterfsd-mgmt.c:1790:mgmt_rpc_notify]
0-glusterfsd-mgmt: -1 connect attempts left
    [2013-03-18 10:23:42.213846] W [glusterfsd.c:831:cleanup_and_exit]
(-->/usr/sbin/glusterfs(glusterfs_mgmt_init+0x1ff) [0x7f38a41a41af]
(-->/usr/lib/libgfrpc.so.0(rpc_clnt_start+0x12) [0x7f38a3af0922]
(-->/usr/sbin/glusterfs(+0xd486) [0x7f38a41a4486]))) 0-: received signum (1),
shutting down
    [2013-03-18 10:23:42.213905] I [fuse-bridge.c:4648:fini] 0-fuse: Unmounting
'/mnt/site-development/'.

Version-Release number of selected component (if applicable):

  glusterfs 3.3.1 built on Feb 21 2013 03:24:40
  "got by the ubuntu ppa"

How reproducible:

  Every time.

Steps to Reproduce:

  1. Install gluster.
  2. Attempt to use a host that only has AAAA DNS records.
  3. Fails here.

--- Additional comment from Joe Julian on 2014-04-18 20:15:50 EDT ---

We need IPv6 for 3.6. Can we make this a priority for this cycle?

--- Additional comment from Justin Clift on 2014-04-18 21:59:30 EDT ---

It'll probably depend on the code for "Better Peer Identification" getting in
first:

http://www.gluster.org/community/documentation/index.php/Features/Better_peer_identification

If that gets in with sufficient lead time, it might be possible to add IPv6
support for 3.6 as well.  Probably need to ask Kaushal if he thinks its
feasible.

--- Additional comment from Anders Blomdell on 2014-06-11 06:44:36 EDT ---

This is still a problem with 3.5.0, it looks like almost all ipv6 support got
ripped out with commit 3dc56cbd16b1074d7ca1a4fe4c5bf44400eb63ff :-(

--- Additional comment from Adam Huffman on 2014-06-12 08:19:18 EDT ---

I see similar problems in a dual-stack setup.

--- Additional comment from Anders Blomdell on 2014-06-12 10:16:26 EDT ---

Reverting 3dc56cbd16b1074d7ca1a4fe4c5bf44400eb63ff (with one small manual
merge) made IPv6 work for me. :-)

Let's hope it gets into 3.6.

--- Additional comment from Anders Blomdell on 2014-07-09 11:20:53 EDT ---

Reinstating ipv6 support by reverting 3dc56cbd16b1074d7ca1a4fe4c5bf44400eb63ff

Make sure to check for regressions in
https://bugzilla.redhat.com/show_bug.cgi?id=764655

--- Additional comment from Anand Avati on 2014-07-10 14:37:12 EDT ---

REVIEW: http://review.gluster.org/8292 (Reinstating ipv6 support by reverting
3dc56cbd16b1074d7ca1a4fe4c5bf44400eb63ff) posted (#2) for review on master by
Anders Blomdell (anders.blomdell at control.lth.se)

--- Additional comment from Anand Avati on 2014-07-10 14:39:17 EDT ---

REVIEW: http://review.gluster.org/8292 (Reinstate ipv6 support) posted (#3) for
review on master by Anders Blomdell (anders.blomdell at control.lth.se)

--- Additional comment from Anand Avati on 2014-07-11 12:02:43 EDT ---

REVIEW: http://review.gluster.org/8292 (Reinstate ipv6 support) posted (#4) for
review on master by Anders Blomdell (anders.blomdell at control.lth.se)

--- Additional comment from Gulf Zhou on 2014-12-12 03:20:36 EST ---

(In reply to Anders Blomdell from comment #1)
> Reinstating ipv6 support by reverting
> 3dc56cbd16b1074d7ca1a4fe4c5bf44400eb63ff
>     
> Make sure to check for regressions in
> https://bugzilla.redhat.com/show_bug.cgi?id=764655

Have this been done on 3.5 branch? Thanks.

--- Additional comment from Anders Blomdell on 2014-12-12 04:05:10 EST ---

(In reply to Gulf Zhou from comment #5)
> Have this been done on 3.5 branch? Thanks.

No, and probably shouldn't. The problem was more complex than I anticipated,
the problem occurs in multiple places, and the lookup of hosts and connecting
to them is unfortunately decoupled, which means that replacing gethostbyname
with getaddrinfo will (always?) return the IPv6 address, which means that hosts
without IPv6 connectivity will be hosed (this was the AFAICT the
[unexplained/unexplored] reason for removing IPv6 support in the first place.

My current plan is to wait for:

http://supercolony.gluster.org/pipermail/gluster-devel/2014-December/043180.html

to get some momentum before I look into it again, since it looks like it has
the potential to provide the right decoupling for network connections.

--- Additional comment from Gulf Zhou on 2014-12-14 20:18:09 EST ---

Thanks, Anders. Read through the thread. The IPv6 support is very important
feature for us. How will this be prioritized? Don't have the feeling when it
can be solved. Any comments?

--- Additional comment from Gulf Zhou on 2015-03-12 21:20:57 EDT ---

Hi, Anders

   Is there any update for the IPv6 support fix? 

   Thanks.

--- Additional comment from Anders Blomdell on 2015-03-13 08:10:51 EDT ---

Waiting for 4.6 (or 4.7) to get stable enough in my lab before I continue with
IPv6, :-/

--- Additional comment from Anders Blomdell on 2015-03-13 08:12:25 EDT ---

I meant: 3.6 or 3.7

--- Additional comment from Nithin on 2015-06-11 14:33:11 EDT ---

Hi Niels & Anders,

Can I contribute to this bug fix ? I've worked on Gluster IPv6 functionality
bugs in 3.3.2 in my past organization and was able to successfully bring up
gluster on IPv6 link local addresses as well.

Please find my work in progress patch. I was successfully able to create
volumes with 3 peers and add bricks. I'll continue testing other basic
functionality and see when needs to be modified.

Brief info about the patch:
Here I'm trying to use "transport.address-family" option in
/etc/glusterfs/glusterd.vol file and then propagate the same to server and
client vol files and their translators.

In this way when user mentions "transport.address-family inet6" in its
glusterd.vol file, all glusterd servers open AF_INET6 sockets and then the same
information is stored in glusterd_volinfo and used when generating vol config
files.

-thanks
Nithin

--- Additional comment from Nithin on 2015-06-11 14:34:32 EDT ---

--- Additional comment from Joachim Tingvold on 2015-08-12 13:42:53 EDT ---

Hi,

So, trying to get glusterfs working in an IPv6-only environment. Tried two
freshly installed CentOS-boxes, both having only IPv6-adresses and quad-A
DNS-records.

Tried latest glusterfs 3.5, 3.6 and 3.7. In all of them, glusterd by-default
only listens to IPv4 (tcp 24007), hence initial 'probe' doesn't work.

Am I understanding this bug correctly that _all_ IPv6-support has been
(temporarily) removed until this bug is resolved?

--- Additional comment from Nithin on 2015-08-15 11:29:23 EDT ---

(In reply to Joachim Tingvold from comment #13)
> Hi,
> 
> So, trying to get glusterfs working in an IPv6-only environment. Tried two
> freshly installed CentOS-boxes, both having only IPv6-adresses and quad-A
> DNS-records.
> 
> Tried latest glusterfs 3.5, 3.6 and 3.7. In all of them, glusterd by-default
> only listens to IPv4 (tcp 24007), hence initial 'probe' doesn't work.
> 
> Am I understanding this bug correctly that _all_ IPv6-support has been
> (temporarily) removed until this bug is resolved?

Yes, basically this bug is to fix issues that are breaking IPv6 functionality.
Most of the support code is already there but is broken probably due to lack of
testing focus on IPv6.

--- Additional comment from Gulf Zhou on 2015-08-18 02:51:09 EDT ---

Hi, Nithin

     Are the patch you proposed merged back to the glusterfs branches? Or is
there any plan to merge it? 

     Thanks.

--- Additional comment from Nithin on 2015-08-19 04:06:43 EDT ---

Hi Gulf,

I have the patch that is almost same as attached in one of my above comments.
Basic functionality is working, i'm trying to validate if there are any errors
and I'm yet to run regression. Once done, I'll raise a review.

--- Additional comment from Anand Avati on 2015-08-22 07:24:47 EDT ---

REVIEW: http://review.gluster.org/11988 (glusterd: transport address-family
inet6 not working) posted (#1) for review on master by Nithin Kumar D
(nithind1988 at yahoo.in)

--- Additional comment from Nithin on 2015-08-23 01:48:39 EDT ---

--- Additional comment from Nithin on 2015-08-23 01:49:12 EDT ---

--- Additional comment from Anand Avati on 2015-08-27 14:23:24 EDT ---

REVIEW: http://review.gluster.org/11988 (glusterd: transport address-family
inet6 not working) posted (#2) for review on master by Nithin Kumar D
(nithind1988 at yahoo.in)

--- Additional comment from Nithin on 2015-08-27 14:27:49 EDT ---

--- Additional comment from Nithin on 2015-08-27 14:28:17 EDT ---

--- Additional comment from Nithin on 2015-08-27 14:28:46 EDT ---

--- Additional comment from Vijay Bellur on 2015-09-09 20:48:57 EDT ---

REVIEW: http://review.gluster.org/11988 (glusterd: transport address-family
inet6 not working) posted (#3) for review on master by Nithin Kumar D
(nithind1988 at yahoo.in)

--- Additional comment from Vijay Bellur on 2015-09-13 12:08:14 EDT ---

REVIEW: http://review.gluster.org/11988 (glusterd: Bug fixes for IPv6 support)
posted (#4) for review on master by Nithin Kumar D (nithind1988 at yahoo.in)

--- Additional comment from Nithin on 2015-09-13 12:23:56 EDT ---

--- Additional comment from Nithin on 2015-09-13 12:24:38 EDT ---

--- Additional comment from Vijay Bellur on 2015-09-15 02:24:06 EDT ---

REVIEW: http://review.gluster.org/11988 (glusterd: Bug fixes for IPv6 support)
posted (#5) for review on master by Vijay Bellur (vbellur at redhat.com)

--- Additional comment from Kaleb KEITHLEY on 2015-10-22 11:40:20 EDT ---

pre-release version is ambiguous and about to be removed as a choice.

If you believe this is still a bug, please change the status back to NEW and
choose the appropriate, applicable version for it.

--- Additional comment from Vijay Bellur on 2015-11-11 01:26:34 EST ---

REVIEW: http://review.gluster.org/11988 (glusterd: Bug fixes for IPv6 support)
posted (#6) for review on master by Nithin Kumar D (nithind1988 at yahoo.in)

--- Additional comment from Vijay Bellur on 2015-11-15 12:06:16 EST ---

REVIEW: http://review.gluster.org/11988 (glusterd: Bug fixes for IPv6 support)
posted (#7) for review on master by Nithin Kumar D (nithind1988 at yahoo.in)

--- Additional comment from Vijay Bellur on 2015-12-03 19:18:58 EST ---

REVIEW: http://review.gluster.org/11988 (glusterd: Bug fixes for IPv6 support)
posted (#8) for review on master by Nithin Kumar D (nithind1988 at yahoo.in)

--- Additional comment from Vijay Bellur on 2015-12-06 04:34:08 EST ---

REVIEW: http://review.gluster.org/11988 (glusterd: Bug fixes for IPv6 support)
posted (#9) for review on master by Nithin Kumar D (nithind1988 at yahoo.in)

--- Additional comment from Vijay Bellur on 2016-01-31 11:02:26 EST ---

REVIEW: http://review.gluster.org/11988 (glusterd: Bug fixes for IPv6 support)
posted (#10) for review on master by Nithin Kumar D (nithind1988 at yahoo.in)

--- Additional comment from Anoop C S on 2016-02-17 03:59:52 EST ---

Re-opening the BZ because we have patches posted which has not yet merged.

--- Additional comment from Vijay Bellur on 2016-02-20 12:16:48 EST ---

COMMIT: http://review.gluster.org/11988 committed in master by Jeff Darcy
(jdarcy at redhat.com) 
------
commit 46bd29e0f2a7fc9278068a06d12066d614f365ec
Author: Nithin D <nithind1988 at yahoo.in>
Date:   Sun Nov 15 22:14:43 2015 +0530

    glusterd: Bug fixes for IPv6 support

    Problem:
    Glusterd not working using ipv6 transport. The idea is with proper
glusterd.vol configuration,
    1. glusterd needs to listen on default port (240007) as IPv6 TCP listner.
    2. Volume creation/deletion/mounting/add-bricks/delete-bricks/peer-probe
       needs to work using ipv6 addresses.
    3. Bricks needs to listen on ipv6 addresses.
    All the above functionality is needed to say that glusterd supports ipv6
transport and this is broken.

    Fix:
    When "option transport.address-family inet6" option is present in
glusterd.vol
    file, it is made sure that glusterd creates listeners using ipv6 sockets
only and also the same information is saved
    inside brick volume files used by glusterfsd brick process when they are
starting.

    Tests Run:
    Regression tests using ./run-tests.sh
        IPv4: Ran manually till tests/basic/rpm.t .
        IPv6: (Need to add the above mentioned config and also add an entry for
"hostname ::1" in /etc/hosts)
            Started failing at ./tests/basic/glusterd/arbiter-volume-probe.t
and ran successfully till here

    Unit Tests using Ipv6
        peer probe
        add-bricks
        remove-bricks
        create volume
        replace-bricks
        start volume
        stop volume
        delete volume

    Change-Id: Iebc96e6cce748b5924ce5da17b0114600ec70a6e
    BUG: 1117886
    Signed-off-by: Nithin D <nithind1988 at yahoo.in>
    Reviewed-on: http://review.gluster.org/11988
    Smoke: Gluster Build System <jenkins at build.gluster.com>
    NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.com>
    Reviewed-by: Atin Mukherjee <amukherj at redhat.com>
    Reviewed-by: Jeff Darcy <jdarcy at redhat.com>

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1117822
[Bug 1117822] Tracker bug for GlusterFS 3.6.0
https://bugzilla.redhat.com/show_bug.cgi?id=1117886
[Bug 1117886] Gluster not resolving hosts with IPv6 only lookups
-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.