[Bugs] [Bug 1253542] New: rebalance stuck at 0 byte when auth.allow is set

Fri Aug 14 04:38:07 UTC 2015

https://bugzilla.redhat.com/show_bug.cgi?id=1253542

            Bug ID: 1253542
           Summary: rebalance stuck at 0 byte when auth.allow is set
           Product: GlusterFS
           Version: 3.7.3
         Component: glusterd
          Severity: high
          Priority: medium
          Assignee: bugs at gluster.org
          Reporter: nbalacha at redhat.com
                CC: bhubbard at redhat.com, bkunal at redhat.com,
                    bugs at gluster.org, cbuissar at redhat.com,
                    gluster-bugs at redhat.com, racpatel at redhat.com,
                    rgowdapp at redhat.com, vagarwal at redhat.com
        Depends On: 1213893, 1248415

+++ This bug was initially created as a clone of Bug #1248415 +++

Description of problem:

When setting auth.allow, rebalance will get stuck unless the IPs of the gluster
nodes themselves are included.
The rebalance will be kept as 'in progress', but will be kept at 0 Byte.

                                    Node Rebalanced-files          size      
scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------  
-----------   -----------   -----------         ------------     --------------
                               localhost                0        0Bytes        
    0             0             0          in progress               0.00
                             node3                0        0Bytes             0
            0             0          in progress               0.00
                             node4                0        0Bytes             0
            0             0          in progress               0.00
                         192.168.100.206                0        0Bytes        
    0             0             0          in progress               0.00
volume rebalance: thingluster: success:

On the bricks logs, we can see the authentication being prevented :
[2015-04-21 13:43:03.131329] E [server-handshake.c:589:server_setvolume]
0-thingluster-server: Cannot authenticate client from
<server>-2015/04/21-13:42:58:108057-thingluster-client-0-0-0 3.6.0.53
[2015-04-21 13:43:08.419405] E [authenticate.c:239:gf_authenticate] 0-auth: no
authentication module is interested in accepting remote-client (null)

Version-Release number of selected component (if applicable): tested on 3.0u3
and 3.0u4

How reproducible: 100%/easy

Steps to Reproduce:
1. set auth.allow to some client IP
2. mount and move files
3. start rebalance

Actual results:
rebalance is hung, authentication errors are shown in the brick logs

Expected results:
Rebalance should still work if we restrict auth.allow.

Additional info:
Workaround : add all the IPs of the gluster nodes in auth.allow.

--- Additional comment from Cedric Buissart on 2015-04-21 09:55:45 EDT ---

And the rebalance-<volume>.log :

[2015-04-21 13:43:08.412805] W [client-handshake.c:1108:client_setvolume_cbk]
0-thingluster-client-3: failed to set the volume (Permission denied)
[2015-04-21 13:43:08.412821] W [client-handshake.c:1134:client_setvolume_cbk]
0-thingluster-client-3: failed to get 'process-uuid' from reply dict
[2015-04-21 13:43:08.412828] E [client-handshake.c:1140:client_setvolume_cbk]
0-thingluster-client-3: SETVOLUME on remote-host failed: Authentication failed
[2015-04-21 13:43:08.412834] I [client-handshake.c:1225:client_setvolume_cbk]
0-thingluster-client-3: sending AUTH_FAILED even

--- Additional comment from Anand Avati on 2015-08-11 05:59:59 EDT ---

COMMIT: http://review.gluster.org/11819 committed in master by Atin Mukherjee
(amukherj at redhat.com) 
------
commit 8603ac80218d6161416fd0f3cca511ba09aef2c8
Author: N Balachandran <nbalacha at redhat.com>
Date:   Mon Aug 3 13:57:37 2015 +0530

    glusterd/rebalance: trusted rebalance volfile

    Creating the client volfiles with GF_CLIENT_OTHER
    overwrites the trusted rebalance volfile and causes rebalance
    to fail if auth.allow is set.
    Now, we always set the value of trusted-client to GF_CLIENT_TRUSTED
    for rebalance volfiles.

    Change-Id: I95eb510256d18dfa9048f96a1aeb71cca4811811
    BUG: 1248415
    Signed-off-by: N Balachandran <nbalacha at redhat.com>
    Reviewed-on: http://review.gluster.org/11819
    Tested-by: Gluster Build System <jenkins at build.gluster.com>
    Tested-by: NetBSD Build System <jenkins at build.gluster.org>
    Reviewed-by: Avra Sengupta <asengupt at redhat.com>
    Reviewed-by: Rajesh Joseph <rjoseph at redhat.com>
    Reviewed-by: Atin Mukherjee <amukherj at redhat.com>

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1213893
[Bug 1213893] rebalance stuck at 0 byte when auth.allow is set
https://bugzilla.redhat.com/show_bug.cgi?id=1248415
[Bug 1248415] rebalance stuck at 0 byte when auth.allow is set
-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.