[Gluster-users] Adding a new node receives State: Peer Rejected (Connected)

Thu Oct 18 11:06:03 UTC 2018

Hi all,

I am testing to extend my gluster cluster and add a new node in glusterfs
3.12.14 version on CentOS Linux release 7.5.1804 and receives
State: Peer Rejected (Connected)

[root at serverc ~]# gluster peer status
Number of Peers: 2

Hostname: servera
Uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549
State: Peer Rejected (Connected)

Hostname: serverb.lab.local
Uuid: e88e454a-f85a-472a-8920-a541c8615d03
State: Peer Rejected (Connected)

[root at servera ~]# gluster peer status
Number of Peers: 2

Hostname: serverb.lab.local
Uuid: e88e454a-f85a-472a-8920-a541c8615d03
State: Peer in Cluster (Connected)

Hostname: serverc
Uuid: 4dc48498-bb24-446e-a3c2-e2855beaa02e
State: Peer Rejected (Connected)

[root at servera ~]# gluster volume info mastervol

Volume Name: mastervol
Type: Distribute
Volume ID: 22fd33cf-ee41-4eef-94c3-a1627258b0e0
Status: Started
Snapshot Count: 0
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: servera:/bricks/brick-a1/brick
Brick2: serverb:/bricks/brick-b1/brick
Options Reconfigured:
server.ssl: on
client.ssl: on
auth.ssl-allow: servera,serverb,workstation,servere,serverc
features.scrub-throttle: aggressive
features.scrub-freq: hourly
features.scrub: Active
features.bitrot: on
features.hard-timeout: 1s
features.soft-timeout: 5s
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
storage.batch-fsync-delay-usec: 0
server.allow-insecure: on
performance.stat-prefetch: off
transport.address-family: inet
nfs.disable: off
performance.client-io-threads: on
cluster.rebal-throttle: lazy
geo-replication.indexing: on
geo-replication.ignore-pid-check: on
changelog.changelog: on
changelog.rollover-time: 5
cluster.enable-shared-storage: enable
auto-delete: enable
snap-activate-on-create: enable

root at serverc ~]# systemctl stop glusterd
[root at serverc ~]# cd /var/lib/glusterd/
[root at serverc glusterd]# ll
total 8
drwxr-xr-x. 2 root root   6 Oct 18 11:19 bitd
-rw-------. 1 root root  66 Oct 18 11:25 glusterd.info
drwxr-xr-x. 2 root root  35 Oct 18 11:22 glustershd
drwxr-xr-x. 2 root root   6 Oct 18 11:21 groups
drwxr-xr-x. 3 root root  15 Oct 18 11:21 hooks
drwxr-xr-x. 2 root root   6 Oct 18 11:21 nfs
-rw-------. 1 root root 157 Oct 18 11:22 options
drwxr-xr-x. 2 root root  94 Oct 18 11:25 peers
drwxr-xr-x. 2 root root  24 Oct 18 11:22 quotad
drwxr-xr-x. 2 root root   6 Oct 18 11:19 scrub
-rw-r--r--. 1 root root   0 Oct 18 11:01 secure-access
drwxr-xr-x. 2 root root  31 Oct 18 11:22 snaps
drwxr-xr-x. 2 root root   6 Oct 18 11:19 ss_brick
drwxr-xr-x. 3 root root  63 Oct 18 11:22 trash
drwxr-xr-x. 5 root root  68 Oct 18 11:22 vols
[root at serverc glusterd]# rm -rf bitd glustershd groups hooks nfs options
peers quotad scrub snaps ss_brick trash vols
[root at serverc glusterd]# ll
total 4
-rw-------. 1 root root 66 Oct 18 11:25 glusterd.info
-rw-r--r--. 1 root root  0 Oct 18 11:01 secure-access
[root at serverc glusterd]#
[root at serverc glusterd]# systemctl start glusterd
[root at serverc glusterd]# gluster peer staus
unrecognized word: staus (position 1)
[root at serverc glusterd]# gluster peer status
Number of Peers: 0

[root at serverc glusterd]# gluster peer probe servera
peer probe: success.
[root at serverc glusterd]# gluster peer probe serverb
peer probe: success.
[root at serverc glusterd]# gluster peer status
Number of Peers: 2

Hostname: servera
Uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549
State: Accepted peer request (Connected)

Hostname: serverb
Uuid: e88e454a-f85a-472a-8920-a541c8615d03
State: Accepted peer request (Connected)

[root at serverc glusterd]# systemctl restart glusterd
[root at serverc glusterd]# gluster peer status
Number of Peers: 2

Hostname: servera
Uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549
State: Sent and Received peer request (Connected)

Hostname: serverb
Uuid: e88e454a-f85a-472a-8920-a541c8615d03
State: Sent and Received peer request (Connected)

[root at serverc glusterd]#
[root at serverc glusterd]# gluster peer status
Number of Peers: 2

Hostname: servera
Uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549
State: Peer in Cluster (Connected)

Hostname: serverb.lab.local
Uuid: e88e454a-f85a-472a-8920-a541c8615d03
State: Peer in Cluster (Connected)
Other names:
serverb.lab.local
[root at serverc glusterd]# gluster peer status
Number of Peers: 2

Hostname: servera
Uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549
State: Peer in Cluster (Connected)

Hostname: serverb.lab.local
Uuid: e88e454a-f85a-472a-8920-a541c8615d03
State: Peer Rejected (Connected)
Other names:
serverb.lab.local

[2018-10-18 10:44:06.424524] I [MSGID: 101190]
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 1
[2018-10-18 10:44:06.430245] I [MSGID: 106544]
[glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID:
4dc48498-bb24-446e-a3c2-e2855beaa02e
[2018-10-18 10:44:06.440524] I [MSGID: 106511]
[glusterd-rpc-ops.c:262:__glusterd_probe_cbk] 0-management: Received probe
resp from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549, host: servera
[2018-10-18 10:44:06.440557] I [MSGID: 106511]
[glusterd-rpc-ops.c:422:__glusterd_probe_cbk] 0-glusterd: Received resp to
probe req
[2018-10-18 10:44:06.448906] I [MSGID: 106511]
[glusterd-rpc-ops.c:262:__glusterd_probe_cbk] 0-management: Received probe
resp from uuid: e88e454a-f85a-472a-8920-a541c8615d03, host: serverb
[2018-10-18 10:44:06.448941] I [MSGID: 106511]
[glusterd-rpc-ops.c:422:__glusterd_probe_cbk] 0-glusterd: Received resp to
probe req
[2018-10-18 10:44:08.785483] I [MSGID: 106487]
[glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
The message "I [MSGID: 106487]
[glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req" repeated 2 times between [2018-10-18
10:44:08.785483] and [2018-10-18 10:44:14.411018]
[2018-10-18 10:44:17.014898] I [MSGID: 106493]
[glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC
from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549, host: servera, port: 0
[2018-10-18 10:44:17.017750] I [MSGID: 106492]
[glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd:
Received friend update from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549
[2018-10-18 10:44:17.019991] I [MSGID: 106502]
[glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management:
Received my uuid as Friend
[2018-10-18 10:44:17.020433] I [MSGID: 106493]
[glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management:
Received ACC from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549
[2018-10-18 10:44:17.023102] I [MSGID: 106493]
[glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC
from uuid: e88e454a-f85a-472a-8920-a541c8615d03, host: serverb, port: 0
[2018-10-18 10:44:17.030879] I [MSGID: 106493]
[glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management:
Received ACC from uuid: e88e454a-f85a-472a-8920-a541c8615d03
[2018-10-18 10:44:17.104044] I [MSGID: 106487]
[glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2018-10-18 10:44:18.502357] I [MSGID: 106163]
[glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack]
0-management: using the op-version 31202
[2018-10-18 10:44:18.506876] I [MSGID: 106493]
[glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management:
Received ACC from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549
[2018-10-18 10:44:18.531884] I [MSGID: 106490]
[glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd:
Received probe from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549
[2018-10-18 10:44:18.542340] I [MSGID: 106105]
[glusterd-snapshot-utils.c:1389:glusterd_is_peer_snap_conflicting]
0-management: Snapshot Scheduled-serenity-replvol_GMT-2018.10.14-18.10.01
from peer servera missing on localhost
[2018-10-18 10:44:18.542494] I [MSGID: 106105]
[glusterd-snapshot-utils.c:1389:glusterd_is_peer_snap_conflicting]
0-management: Snapshot Scheduled-serenity-replvol_GMT-2018.10.14-18.12.01
from peer servera missing on localhost
[2018-10-18 10:44:18.542561] I [MSGID: 106105]
[glusterd-snapshot-utils.c:1389:glusterd_is_peer_snap_conflicting]
0-management: Snapshot Scheduled-serenity-replvol_GMT-2018.10.14-18.14.01
from peer servera missing on localhost
[2018-10-18 10:44:18.542634] I [MSGID: 106105]
[glusterd-snapshot-utils.c:1389:glusterd_is_peer_snap_conflicting]
0-management: Snapshot Scheduled-serenity-replvol_GMT-2018.10.14-18.16.01
from peer servera missing on localhost
[2018-10-18 10:44:18.542712] I [MSGID: 106105]
[glusterd-snapshot-utils.c:1389:glusterd_is_peer_snap_conflicting]
0-management: Snapshot Scheduled-serenity-replvol_GMT-2018.10.14-18.18.01
from peer servera missing on localhost
[2018-10-18 10:44:18.545596] I [MSGID: 106493]
[glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd:
Responded to servera (0), ret: 0, op_ret: 0
[2018-10-18 10:44:18.550763] I [MSGID: 106492]
[glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd:
Received friend update from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549
[2018-10-18 10:44:18.552681] I [MSGID: 106502]
[glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management:
Received my uuid as Friend
[2018-10-18 10:44:18.579609] W [MSGID: 101095]
[xlator.c:162:xlator_volopt_dynload] 0-xlator:
/usr/lib64/glusterfs/3.12.14/xlator/nfs/server.so: cannot open shared
object file: No such file or directory
The message "W [MSGID: 101095] [xlator.c:162:xlator_volopt_dynload]
0-xlator: /usr/lib64/glusterfs/3.12.14/xlator/nfs/server.so: cannot open
shared object file: No such file or directory" repeated 5 times between
[2018-10-18 10:44:18.579609] and [2018-10-18 10:44:18.605927]
[2018-10-18 10:44:18.607656] E [MSGID: 106049]
[glusterd-snapshot-utils.c:1529:glusterd_gen_snap_volfiles] 0-management:
Parent volinfo not found for 1ec76d1dbc1c4209b774b13c07b7b337 volume of
snap Scheduled-serenity-replvol_GMT-2018.10.14-18.10.01
[2018-10-18 10:44:18.607689] E [MSGID: 106069]
[glusterd-snapshot-utils.c:1678:glusterd_import_friend_snap] 0-management:
Failed to generate snap vol files for snap
Scheduled-serenity-replvol_GMT-2018.10.14-18.10.01
[2018-10-18 10:44:18.607819] E [MSGID: 106100]
[glusterd-snapshot-utils.c:2106:glusterd_update_snaps_synctask]
0-management: Failed to import snap
Scheduled-serenity-replvol_GMT-2018.10.14-18.10.01 from peer servera
[2018-10-18 10:44:18.664472] I [rpc-clnt.c:1044:rpc_clnt_connection_init]
0-snapd: setting frame-timeout to 600
[2018-10-18 10:44:18.664613] I [socket.c:4315:socket_init] 0-snapd: SSL
support for glusterd is ENABLED
[2018-10-18 10:44:18.664776] E [socket.c:4383:socket_init] 0-snapd: failed
to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-10-18 10:44:18.665723] E [MSGID: 106029]
[glusterd-volgen.c:974:volgen_write_volfile] 0-management: file creation
failed [No such file or directory]
[2018-10-18 10:44:18.665753] E [MSGID: 106069]
[glusterd-volgen.c:1020:volgen_write_volfile] 0-management: failed to
create volfile /var/lib/glusterd/vols/replvol/replvol-snapd.vol
[2018-10-18 10:44:18.665775] E [MSGID: 106114]
[glusterd-snapd-svc.c:175:glusterd_snapdsvc_manager] 0-management: Couldn't
create snapd volfile for volume: replvol
[2018-10-18 10:44:18.693863] W [MSGID: 101095]
[xlator.c:162:xlator_volopt_dynload] 0-xlator:
/usr/lib64/glusterfs/3.12.14/xlator/nfs/server.so: cannot open shared
object file: No such file or directory
[2018-10-18 10:44:18.712003] I [rpc-clnt.c:1044:rpc_clnt_connection_init]
0-nfs: setting frame-timeout to 600
[2018-10-18 10:44:18.712208] I [socket.c:4315:socket_init] 0-nfs: SSL
support for glusterd is ENABLED
[2018-10-18 10:44:18.712392] E [socket.c:4383:socket_init] 0-nfs: failed to
open /etc/ssl/dhparam.pem, DH ciphers are disabled
The message "W [MSGID: 101095] [xlator.c:162:xlator_volopt_dynload]
0-xlator: /usr/lib64/glusterfs/3.12.14/xlator/nfs/server.so: cannot open
shared object file: No such file or directory" repeated 5 times between
[2018-10-18 10:44:18.693863] and [2018-10-18 10:44:18.710617]
[2018-10-18 10:44:18.713052] I [MSGID: 106132]
[glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already
stopped
[2018-10-18 10:44:18.713091] I [MSGID: 106568]
[glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is
stopped
[2018-10-18 10:44:18.713106] I [MSGID: 106600]
[glusterd-nfs-svc.c:82:glusterd_nfssvc_manager] 0-management: nfs/server.so
xlator is not installed
[2018-10-18 10:44:18.713147] I [rpc-clnt.c:1044:rpc_clnt_connection_init]
0-glustershd: setting frame-timeout to 600
[2018-10-18 10:44:18.713228] I [socket.c:4315:socket_init] 0-glustershd:
SSL support for glusterd is ENABLED
[2018-10-18 10:44:18.713343] E [socket.c:4383:socket_init] 0-glustershd:
failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-10-18 10:44:18.715967] W [MSGID: 101095]
[xlator.c:162:xlator_volopt_dynload] 0-xlator:
/usr/lib64/glusterfs/3.12.14/xlator/nfs/server.so: cannot open shared
object file: No such file or directory
[2018-10-18 10:44:18.718305] W [MSGID: 101095]
[xlator.c:162:xlator_volopt_dynload] 0-xlator:
/usr/lib64/glusterfs/3.12.14/xlator/nfs/server.so: cannot open shared
object file: No such file or directory
[2018-10-18 10:44:18.720532] I [MSGID: 106132]
[glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: glustershd
already stopped
[2018-10-18 10:44:18.720636] I [MSGID: 106568]
[glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd
service is stopped
[2018-10-18 10:44:18.720722] I [MSGID: 106567]
[glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting
glustershd service
[2018-10-18 10:44:19.736067] I [rpc-clnt.c:1044:rpc_clnt_connection_init]
0-quotad: setting frame-timeout to 600
[2018-10-18 10:44:19.736205] I [socket.c:4315:socket_init] 0-quotad: SSL
support for glusterd is ENABLED
[2018-10-18 10:44:19.736312] E [socket.c:4383:socket_init] 0-quotad: failed
to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-10-18 10:44:19.737481] I [MSGID: 106132]
[glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already
stopped
[2018-10-18 10:44:19.737504] I [MSGID: 106568]
[glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is
stopped
[2018-10-18 10:44:19.737533] I [MSGID: 106567]
[glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting quotad
service
[2018-10-18 10:44:20.746148] I [rpc-clnt.c:1044:rpc_clnt_connection_init]
0-bitd: setting frame-timeout to 600
[2018-10-18 10:44:20.746274] I [socket.c:4315:socket_init] 0-bitd: SSL
support for glusterd is ENABLED
[2018-10-18 10:44:20.746388] E [socket.c:4383:socket_init] 0-bitd: failed
to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-10-18 10:44:20.747092] I [MSGID: 106132]
[glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already
stopped
[2018-10-18 10:44:20.747120] I [MSGID: 106568]
[glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is
stopped
[2018-10-18 10:44:20.747159] I [rpc-clnt.c:1044:rpc_clnt_connection_init]
0-scrub: setting frame-timeout to 600
[2018-10-18 10:44:20.747239] I [socket.c:4315:socket_init] 0-scrub: SSL
support for glusterd is ENABLED
[2018-10-18 10:44:20.747323] E [socket.c:4383:socket_init] 0-scrub: failed
to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-10-18 10:44:20.747955] I [MSGID: 106132]
[glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already
stopped
[2018-10-18 10:44:20.747978] I [MSGID: 106568]
[glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is
stopped
[2018-10-18 10:44:20.748550] I [MSGID: 106493]
[glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management:
Received ACC from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549
[2018-10-18 10:44:20.786840] I [MSGID: 106163]
[glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack]
0-management: using the op-version 31202
[2018-10-18 10:44:20.836517] I [MSGID: 106490]
[glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd:
Received probe from uuid: e88e454a-f85a-472a-8920-a541c8615d03
[2018-10-18 10:44:20.839057] E [MSGID: 106010]
[glusterd-utils.c:3383:glusterd_compare_friend_volume] 0-management:
Version of Cksums replvol differ. local cksum = 3139705045, remote cksum =
3139442900 on peer serverb.lab.local
[2018-10-18 10:44:20.839450] I [MSGID: 106493]
[glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd:
Responded to serverb.lab.local (0), ret: 0, op_ret: -1
[2018-10-18 10:44:22.822516] I [MSGID: 106487]
[glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[root at serverc glusterd]#
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181018/bb6c765f/attachment.html>