[Bugs] [Bug 1437332] New: auth failure after upgrade to GlusterFS 3.10

Thu Mar 30 06:03:54 UTC 2017

https://bugzilla.redhat.com/show_bug.cgi?id=1437332

            Bug ID: 1437332
           Summary: auth failure after upgrade to GlusterFS 3.10
           Product: Red Hat Gluster Storage
           Version: 3.3
         Component: protocol
          Keywords: Triaged
          Severity: urgent
          Assignee: prasanna.kalever at redhat.com
          Reporter: amukherj at redhat.com
        QA Contact: sbhaloth at redhat.com
                CC: amukherj at redhat.com, bordas.csaba at gmail.com,
                    bugs at gluster.org, hiscal at 126.com, michalon at igbmc.fr,
                    news at ascora.de, rhs-bugs at redhat.com,
                    rkavunga at redhat.com, storage-qa-internal at redhat.com
        Depends On: 1429117, 1433815

+++ This bug was initially created as a clone of Bug #1433815 +++

+++ This bug was initially created as a clone of Bug #1429117 +++

Description of problem:
We enabled the IP based auth feature with
gluster volume set store_temp auth.allow xxx.xxx.xxx...
This worked fine up to GlusterFS 3.9. After upgrading to 3.10, we noticed that
we cannot mount any volume from a remove client anymore.
Looking at the brick logs we found:

[2017-03-04 15:56:17.469490] I [MSGID: 115091]
[server-handshake.c:659:server_setvolume] 0-store_temp-server: Failed to get
client opversion
[2017-03-04 15:56:17.469520] E [MSGID: 115004]
[authenticate.c:224:gf_authenticate] 0-auth: no authentication module is
interested in accepting remote-client (null)
[2017-03-04 15:56:17.469602] E [MSGID: 115001]
[server-handshake.c:718:server_setvolume] 0-store_temp-server: Cannot
authenticate client from
backupserver-9596-2017/03/04-15:56:17:438653-store_temp-client-2-0-0 3.9.1
[Permission denied]
[2017-03-04 15:56:28.472405] I [MSGID: 115036] [server.c:559:server_rpc_notify]
0-store_temp-server: disconnecting connection from
backupserver-9596-2017/03/04-15:56:17:438653-store_temp-client-2-0-0
[2017-03-04 15:56:28.472518] I [MSGID: 101055] [client_t.c:436:gf_client_unref]
0-store_temp-server: Shutting down connection
backupserver-9596-2017/03/04-15:56:17:438653-store_temp-client-2-0-0

This problem exists even when creating completely new volumes. We already
restarted and even rebooted all GlusterFS peers and the clients as well. All
peers and all clients have been upgraded to 3.10

Version-Release number of selected component (if applicable):
3.10

How reproducible:
-Create a new volume 
-enable auth.allow based on IPs

Steps to Reproduce:
1. gluster volume create store_temp disperse 3 redundancy 1 ...
2. gluster volume set store_temp auth.allow xxx.xxx.xxx.xxx
3. gluster volume start store_temp
4. gluster mount ... (on a client)

Actual results:
-error message at clients "failed to set the volume [Permission denied]"
-error message at server: "no authentication module is interested in accepting
remote-client (null)"

Expected results:
successful mount

Additional info:
Ubuntu 16.04

--- Additional comment from Jiffin on 2017-03-07 07:29:04 EST ---

Can you provide entire logs including bricks,glusterd and glusterfs client.
Also it will be easier if can take the tcdump from server and client

--- Additional comment from Jonathan Michalon on 2017-03-07 09:11:19 EST ---

I am stumbling on the same problem.
Setting log level to DEBUG (gluster volume set volname
diagnostics.brick-log-level DEBUG) I got this first interesting stuff:
  allowed = "192.168.122.186", received addr = "R"
Then some time afterwards:
  allowed = "192.168.122.186", received addr = "m"

So it was looking like we were reading some random memory. And indeed looking
into source code, between 3.9 and 3.10 the big switch/case filling peer_addr
disappeared in /xlators/protocol/auth/addr/src/addr.c 
I think this is enough to tell that there is some problem here :)

--- Additional comment from Atin Mukherjee on 2017-03-13 02:00:47 EDT ---

auth failures need not be in glusterd, moving this to core component.

--- Additional comment from Yong on 2017-03-19 03:35:18 EDT ---

I have the same issue, I think this is critical, please help

--- Additional comment from Worker Ant on 2017-03-19 21:00:07 EDT ---

REVIEW: https://review.gluster.org/16920 (protocol : fix auth-allow regression)
posted (#1) for review on master by Atin Mukherjee (amukherj at redhat.com)

--- Additional comment from Worker Ant on 2017-03-20 16:03:19 EDT ---

REVIEW: https://review.gluster.org/16920 (protocol : fix auth-allow regression)
posted (#2) for review on master by Atin Mukherjee (amukherj at redhat.com)

--- Additional comment from Worker Ant on 2017-03-24 14:22:14 EDT ---

REVIEW: https://review.gluster.org/16920 (protocol : fix auth-allow regression)
posted (#3) for review on master by Atin Mukherjee (amukherj at redhat.com)

--- Additional comment from Worker Ant on 2017-03-27 13:11:09 EDT ---

REVIEW: https://review.gluster.org/16920 (protocol : fix auth-allow regression)
posted (#4) for review on master by Atin Mukherjee (amukherj at redhat.com)

--- Additional comment from Worker Ant on 2017-03-27 13:24:54 EDT ---

REVIEW: https://review.gluster.org/16920 (protocol : fix auth-allow regression)
posted (#5) for review on master by Atin Mukherjee (amukherj at redhat.com)

--- Additional comment from Worker Ant on 2017-03-28 01:38:03 EDT ---

REVIEW: https://review.gluster.org/16920 (protocol : fix auth-allow regression)
posted (#6) for review on master by Atin Mukherjee (amukherj at redhat.com)

--- Additional comment from Worker Ant on 2017-03-28 16:04:33 EDT ---

REVIEW: https://review.gluster.org/16920 (protocol : fix auth-allow regression)
posted (#7) for review on master by Jeff Darcy (jeff at pl.atyp.us)

--- Additional comment from Worker Ant on 2017-03-28 17:51:31 EDT ---

REVIEW: https://review.gluster.org/16920 (protocol : fix auth-allow regression)
posted (#8) for review on master by Jeff Darcy (jeff at pl.atyp.us)

--- Additional comment from Worker Ant on 2017-03-28 18:05:24 EDT ---

REVIEW: https://review.gluster.org/16920 (protocol : fix auth-allow regression)
posted (#9) for review on master by Jeff Darcy (jeff at pl.atyp.us)

--- Additional comment from Worker Ant on 2017-03-28 18:24:02 EDT ---

REVIEW: https://review.gluster.org/16920 (protocol : fix auth-allow regression)
posted (#10) for review on master by Jeff Darcy (jeff at pl.atyp.us)

--- Additional comment from Worker Ant on 2017-03-30 01:57:02 EDT ---

COMMIT: https://review.gluster.org/16920 committed in master by Atin Mukherjee
(amukherj at redhat.com) 
------
commit 0bd58241143e91b683a3e5c4335aabf9eed537fe
Author: Atin Mukherjee <amukherj at redhat.com>
Date:   Mon Mar 20 05:15:25 2017 +0530

    protocol : fix auth-allow regression

    One of the brick multiplexing patches (commit 1a95fc3) had some changes
    in gf_auth () & server_setvolume () functions which caused auth-allow
    feature to be broken. mount doesn't succeed even if it's part of the
    auth-allow list. This fix does the following:

    1. Reintroduce the peer-info data back in gf_auth () so that fnmatch has
    valid input and it can decide on the result.

    2. config-params dict should capture key values pairs for all the bricks
    in case brick multiplexing is on. In case brick multiplexing isn't
    enabled, then config-params should carry attributes from protocol/server
    such that all rpc auth related attributes stay in tact in the
    dictionary.

    Change-Id: I007c4c6d78620a896b8858a29459a77de8b52412
    BUG: 1433815
    Signed-off-by: Atin Mukherjee <amukherj at redhat.com>
    Reviewed-on: https://review.gluster.org/16920
    Tested-by: Jeff Darcy <jeff at pl.atyp.us>
    Smoke: Gluster Build System <jenkins at build.gluster.org>
    NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
    Reviewed-by: Jeff Darcy <jeff at pl.atyp.us>
    Reviewed-by: MOHIT AGRAWAL <moagrawa at redhat.com>

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1429117
[Bug 1429117] auth failure after upgrade to GlusterFS 3.10
https://bugzilla.redhat.com/show_bug.cgi?id=1433815
[Bug 1433815] auth failure after upgrade to GlusterFS 3.10
-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=XLzNEyKfvo&a=cc_unsubscribe