<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi all,</div><div><br></div><div>I am testing to extend my gluster cluster and add a new node in
glusterfs 3.12.14
version on CentOS Linux release 7.5.1804 and receives <br></div><div>State: Peer Rejected (Connected)</div><div><br></div><div><br>[root@serverc ~]# gluster peer status<br>Number of Peers: 2<br><br>Hostname: servera<br>Uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549<br>State: Peer Rejected (Connected)<br><br>Hostname: serverb.lab.local<br>Uuid: e88e454a-f85a-472a-8920-a541c8615d03<br>State: Peer Rejected (Connected)<br></div><div><br></div><div><br></div><div>[root@servera ~]# gluster peer status<br>Number of Peers: 2<br><br>Hostname: serverb.lab.local<br>Uuid: e88e454a-f85a-472a-8920-a541c8615d03<br>State: Peer in Cluster (Connected)<br><br>Hostname: serverc<br>Uuid: 4dc48498-bb24-446e-a3c2-e2855beaa02e<br>State: Peer Rejected (Connected)<br></div><div><br></div><div><br></div><div><br>[root@servera ~]# gluster volume info mastervol<br><br>Volume Name: mastervol<br>Type: Distribute<br>Volume ID: 22fd33cf-ee41-4eef-94c3-a1627258b0e0<br>Status: Started<br>Snapshot Count: 0<br>Number of Bricks: 2<br>Transport-type: tcp<br>Bricks:<br>Brick1: servera:/bricks/brick-a1/brick<br>Brick2: serverb:/bricks/brick-b1/brick<br>Options Reconfigured:<br>server.ssl: on<br>client.ssl: on<br>auth.ssl-allow: servera,serverb,workstation,servere,serverc<br>features.scrub-throttle: aggressive<br>features.scrub-freq: hourly<br>features.scrub: Active<br>features.bitrot: on<br>features.hard-timeout: 1s<br>features.soft-timeout: 5s<br>features.quota-deem-statfs: on<br>features.inode-quota: on<br>features.quota: on<br>storage.batch-fsync-delay-usec: 0<br>server.allow-insecure: on<br>performance.stat-prefetch: off<br>transport.address-family: inet<br>nfs.disable: off<br>performance.client-io-threads: on<br>cluster.rebal-throttle: lazy<br>geo-replication.indexing: on<br>geo-replication.ignore-pid-check: on<br>changelog.changelog: on<br>changelog.rollover-time: 5<br>cluster.enable-shared-storage: enable<br>auto-delete: enable<br>snap-activate-on-create: enable</div><div><br></div><div><br></div><div>root@serverc ~]# systemctl stop glusterd<br>[root@serverc ~]# cd /var/lib/glusterd/<br>[root@serverc glusterd]# ll<br>total 8<br>drwxr-xr-x. 2 root root 6 Oct 18 11:19 bitd<br>-rw-------. 1 root root 66 Oct 18 11:25 <a href="http://glusterd.info">glusterd.info</a><br>drwxr-xr-x. 2 root root 35 Oct 18 11:22 glustershd<br>drwxr-xr-x. 2 root root 6 Oct 18 11:21 groups<br>drwxr-xr-x. 3 root root 15 Oct 18 11:21 hooks<br>drwxr-xr-x. 2 root root 6 Oct 18 11:21 nfs<br>-rw-------. 1 root root 157 Oct 18 11:22 options<br>drwxr-xr-x. 2 root root 94 Oct 18 11:25 peers<br>drwxr-xr-x. 2 root root 24 Oct 18 11:22 quotad<br>drwxr-xr-x. 2 root root 6 Oct 18 11:19 scrub<br>-rw-r--r--. 1 root root 0 Oct 18 11:01 secure-access<br>drwxr-xr-x. 2 root root 31 Oct 18 11:22 snaps<br>drwxr-xr-x. 2 root root 6 Oct 18 11:19 ss_brick<br>drwxr-xr-x. 3 root root 63 Oct 18 11:22 trash<br>drwxr-xr-x. 5 root root 68 Oct 18 11:22 vols<br>[root@serverc glusterd]# rm -rf bitd glustershd groups hooks nfs options peers quotad scrub snaps ss_brick trash vols<br>[root@serverc glusterd]# ll<br>total 4<br>-rw-------. 1 root root 66 Oct 18 11:25 <a href="http://glusterd.info">glusterd.info</a><br>-rw-r--r--. 1 root root 0 Oct 18 11:01 secure-access<br>[root@serverc glusterd]#<br>[root@serverc glusterd]# systemctl start glusterd<br>[root@serverc glusterd]# gluster peer staus<br>unrecognized word: staus (position 1)<br>[root@serverc glusterd]# gluster peer status<br>Number of Peers: 0<br></div><div><br></div><div><br>[root@serverc glusterd]# gluster peer probe servera<br>peer probe: success.<br>[root@serverc glusterd]# gluster peer probe serverb<br>peer probe: success.<br>[root@serverc glusterd]# gluster peer status<br>Number of Peers: 2<br><br>Hostname: servera<br>Uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549<br>State: Accepted peer request (Connected)<br><br>Hostname: serverb<br>Uuid: e88e454a-f85a-472a-8920-a541c8615d03<br>State: Accepted peer request (Connected)<br></div><div><br></div><div><br></div><div>[root@serverc glusterd]# systemctl restart glusterd<br>[root@serverc glusterd]# gluster peer status<br>Number of Peers: 2<br><br>Hostname: servera<br>Uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549<br>State: Sent and Received peer request (Connected)<br><br>Hostname: serverb<br>Uuid: e88e454a-f85a-472a-8920-a541c8615d03<br>State: Sent and Received peer request (Connected)<br><br>[root@serverc glusterd]#<br>[root@serverc glusterd]# gluster peer status<br>Number of Peers: 2<br><br>Hostname: servera<br>Uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549<br>State: Peer in Cluster (Connected)<br><br>Hostname: serverb.lab.local<br>Uuid: e88e454a-f85a-472a-8920-a541c8615d03<br>State: Peer in Cluster (Connected)<br>Other names:<br>serverb.lab.local<br>[root@serverc glusterd]# gluster peer status<br>Number of Peers: 2<br><br>Hostname: servera<br>Uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549<br>State: Peer in Cluster (Connected)<br><br>Hostname: serverb.lab.local<br>Uuid: e88e454a-f85a-472a-8920-a541c8615d03<br>State: Peer Rejected (Connected)<br>Other names:<br>serverb.lab.local</div><div><br></div><div>[2018-10-18 10:44:06.424524] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1<br>[2018-10-18 10:44:06.430245] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 4dc48498-bb24-446e-a3c2-e2855beaa02e<br>[2018-10-18 10:44:06.440524] I [MSGID: 106511] [glusterd-rpc-ops.c:262:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549, host: servera<br>[2018-10-18 10:44:06.440557] I [MSGID: 106511] [glusterd-rpc-ops.c:422:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req<br>[2018-10-18 10:44:06.448906] I [MSGID: 106511] [glusterd-rpc-ops.c:262:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: e88e454a-f85a-472a-8920-a541c8615d03, host: serverb<br>[2018-10-18 10:44:06.448941] I [MSGID: 106511] [glusterd-rpc-ops.c:422:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req<br>[2018-10-18 10:44:08.785483] I [MSGID: 106487] [glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>The message "I [MSGID: 106487] [glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 2 times between [2018-10-18 10:44:08.785483] and [2018-10-18 10:44:14.411018]<br>[2018-10-18 10:44:17.014898] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549, host: servera, port: 0<br>[2018-10-18 10:44:17.017750] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549<br>[2018-10-18 10:44:17.019991] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend<br>[2018-10-18 10:44:17.020433] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549<br>[2018-10-18 10:44:17.023102] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: e88e454a-f85a-472a-8920-a541c8615d03, host: serverb, port: 0<br>[2018-10-18 10:44:17.030879] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: e88e454a-f85a-472a-8920-a541c8615d03<br>[2018-10-18 10:44:17.104044] I [MSGID: 106487] [glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>[2018-10-18 10:44:18.502357] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202<br>[2018-10-18 10:44:18.506876] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549<br>[2018-10-18 10:44:18.531884] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549<br>[2018-10-18 10:44:18.542340] I [MSGID: 106105] [glusterd-snapshot-utils.c:1389:glusterd_is_peer_snap_conflicting] 0-management: Snapshot Scheduled-serenity-replvol_GMT-2018.10.14-18.10.01 from peer servera missing on localhost<br>[2018-10-18 10:44:18.542494] I [MSGID: 106105] [glusterd-snapshot-utils.c:1389:glusterd_is_peer_snap_conflicting] 0-management: Snapshot Scheduled-serenity-replvol_GMT-2018.10.14-18.12.01 from peer servera missing on localhost<br>[2018-10-18 10:44:18.542561] I [MSGID: 106105] [glusterd-snapshot-utils.c:1389:glusterd_is_peer_snap_conflicting] 0-management: Snapshot Scheduled-serenity-replvol_GMT-2018.10.14-18.14.01 from peer servera missing on localhost<br>[2018-10-18 10:44:18.542634] I [MSGID: 106105] [glusterd-snapshot-utils.c:1389:glusterd_is_peer_snap_conflicting] 0-management: Snapshot Scheduled-serenity-replvol_GMT-2018.10.14-18.16.01 from peer servera missing on localhost<br>[2018-10-18 10:44:18.542712] I [MSGID: 106105] [glusterd-snapshot-utils.c:1389:glusterd_is_peer_snap_conflicting] 0-management: Snapshot Scheduled-serenity-replvol_GMT-2018.10.14-18.18.01 from peer servera missing on localhost<br>[2018-10-18 10:44:18.545596] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to servera (0), ret: 0, op_ret: 0<br>[2018-10-18 10:44:18.550763] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549<br>[2018-10-18 10:44:18.552681] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend<br>[2018-10-18 10:44:18.579609] W [MSGID: 101095] [xlator.c:162:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.12.14/xlator/nfs/server.so: cannot open shared object file: No such file or directory<br>The message "W [MSGID: 101095] [xlator.c:162:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.12.14/xlator/nfs/server.so: cannot open shared object file: No such file or directory" repeated 5 times between [2018-10-18 10:44:18.579609] and [2018-10-18 10:44:18.605927]<br>[2018-10-18 10:44:18.607656] E [MSGID: 106049] [glusterd-snapshot-utils.c:1529:glusterd_gen_snap_volfiles] 0-management: Parent volinfo not found for 1ec76d1dbc1c4209b774b13c07b7b337 volume of snap Scheduled-serenity-replvol_GMT-2018.10.14-18.10.01<br>[2018-10-18 10:44:18.607689] E [MSGID: 106069] [glusterd-snapshot-utils.c:1678:glusterd_import_friend_snap] 0-management: Failed to generate snap vol files for snap Scheduled-serenity-replvol_GMT-2018.10.14-18.10.01<br>[2018-10-18 10:44:18.607819] E [MSGID: 106100] [glusterd-snapshot-utils.c:2106:glusterd_update_snaps_synctask] 0-management: Failed to import snap Scheduled-serenity-replvol_GMT-2018.10.14-18.10.01 from peer servera<br>[2018-10-18 10:44:18.664472] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600<br>[2018-10-18 10:44:18.664613] I [socket.c:4315:socket_init] 0-snapd: SSL support for glusterd is ENABLED<br>[2018-10-18 10:44:18.664776] E [socket.c:4383:socket_init] 0-snapd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-10-18 10:44:18.665723] E [MSGID: 106029] [glusterd-volgen.c:974:volgen_write_volfile] 0-management: file creation failed [No such file or directory]<br>[2018-10-18 10:44:18.665753] E [MSGID: 106069] [glusterd-volgen.c:1020:volgen_write_volfile] 0-management: failed to create volfile /var/lib/glusterd/vols/replvol/replvol-snapd.vol<br>[2018-10-18 10:44:18.665775] E [MSGID: 106114] [glusterd-snapd-svc.c:175:glusterd_snapdsvc_manager] 0-management: Couldn't create snapd volfile for volume: replvol<br>[2018-10-18 10:44:18.693863] W [MSGID: 101095] [xlator.c:162:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.12.14/xlator/nfs/server.so: cannot open shared object file: No such file or directory<br>[2018-10-18 10:44:18.712003] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600<br>[2018-10-18 10:44:18.712208] I [socket.c:4315:socket_init] 0-nfs: SSL support for glusterd is ENABLED<br>[2018-10-18 10:44:18.712392] E [socket.c:4383:socket_init] 0-nfs: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>The message "W [MSGID: 101095] [xlator.c:162:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.12.14/xlator/nfs/server.so: cannot open shared object file: No such file or directory" repeated 5 times between [2018-10-18 10:44:18.693863] and [2018-10-18 10:44:18.710617]<br>[2018-10-18 10:44:18.713052] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: nfs already stopped<br>[2018-10-18 10:44:18.713091] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped<br>[2018-10-18 10:44:18.713106] I [MSGID: 106600] [glusterd-nfs-svc.c:82:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed<br>[2018-10-18 10:44:18.713147] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600<br>[2018-10-18 10:44:18.713228] I [socket.c:4315:socket_init] 0-glustershd: SSL support for glusterd is ENABLED<br>[2018-10-18 10:44:18.713343] E [socket.c:4383:socket_init] 0-glustershd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-10-18 10:44:18.715967] W [MSGID: 101095] [xlator.c:162:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.12.14/xlator/nfs/server.so: cannot open shared object file: No such file or directory<br>[2018-10-18 10:44:18.718305] W [MSGID: 101095] [xlator.c:162:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.12.14/xlator/nfs/server.so: cannot open shared object file: No such file or directory<br>[2018-10-18 10:44:18.720532] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: glustershd already stopped<br>[2018-10-18 10:44:18.720636] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped<br>[2018-10-18 10:44:18.720722] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service<br>[2018-10-18 10:44:19.736067] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600<br>[2018-10-18 10:44:19.736205] I [socket.c:4315:socket_init] 0-quotad: SSL support for glusterd is ENABLED<br>[2018-10-18 10:44:19.736312] E [socket.c:4383:socket_init] 0-quotad: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-10-18 10:44:19.737481] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: quotad already stopped<br>[2018-10-18 10:44:19.737504] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped<br>[2018-10-18 10:44:19.737533] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting quotad service<br>[2018-10-18 10:44:20.746148] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600<br>[2018-10-18 10:44:20.746274] I [socket.c:4315:socket_init] 0-bitd: SSL support for glusterd is ENABLED<br>[2018-10-18 10:44:20.746388] E [socket.c:4383:socket_init] 0-bitd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-10-18 10:44:20.747092] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: bitd already stopped<br>[2018-10-18 10:44:20.747120] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped<br>[2018-10-18 10:44:20.747159] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600<br>[2018-10-18 10:44:20.747239] I [socket.c:4315:socket_init] 0-scrub: SSL support for glusterd is ENABLED<br>[2018-10-18 10:44:20.747323] E [socket.c:4383:socket_init] 0-scrub: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-10-18 10:44:20.747955] I [MSGID: 106132] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: scrub already stopped<br>[2018-10-18 10:44:20.747978] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped<br>[2018-10-18 10:44:20.748550] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: fee14355-9dd7-4996-ba08-ee6d17cbf549<br>[2018-10-18 10:44:20.786840] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31202<br>[2018-10-18 10:44:20.836517] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: e88e454a-f85a-472a-8920-a541c8615d03<br>[2018-10-18 10:44:20.839057] E [MSGID: 106010] [glusterd-utils.c:3383:glusterd_compare_friend_volume] 0-management: Version of Cksums replvol differ. local cksum = 3139705045, remote cksum = 3139442900 on peer serverb.lab.local<br>[2018-10-18 10:44:20.839450] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to serverb.lab.local (0), ret: 0, op_ret: -1<br>[2018-10-18 10:44:22.822516] I [MSGID: 106487] [glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>[root@serverc glusterd]#<br><br></div></div></div></div></div></div></div></div></div></div>