[Gluster-users] 答复: 答复: 答复: Gluster peer rejected and failed to start
Joe Julian
joe at julianfamily.org
Fri Jun 5 02:44:11 UTC 2015
You cannot join a trusted pool, you must be invited.
In other words, a server inside the trusted pool must probe the new server, and that new server must not be part of another pool.
On June 4, 2015 7:24:28 PM PDT, "vyyy杨雨阳" <yuyangyang at Ctrip.com> wrote:
>Can anyone help me?
>
>New nodes still can't join the cluster.
>The new nodes can peer each other, just not exist cluster nodes.
>I can not find any network problem, there were no firewall, port 24007
>is connected, there were all at the same subnet...
>
>Thanks
>
>
>
>[root at SVR6993HW2285 glusterfs]# more etc-glusterfs-glusterd.vol.log
>[2015-06-05 01:59:02.276107] I [MSGID: 100030] [glusterfsd.c:2018:main]
>0-/usr/sbin/glusterd: Started running /usr/sbin/glust
>erd version 3.6.3 (args: /usr/sbin/glusterd
>--pid-file=/var/run/glusterd.pid)
>[2015-06-05 01:59:02.318477] I [glusterd.c:1214:init] 0-management:
>Maximum allowed open file descriptors set to 65536
>[2015-06-05 01:59:02.318524] I [glusterd.c:1259:init] 0-management:
>Using /var/lib/glusterd as working directory
>[2015-06-05 01:59:02.329603] E [rpc-transport.c:266:rpc_transport_load]
>0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-trans
>port/rdma.so: cannot open shared object file: No such file or directory
>[2015-06-05 01:59:02.329625] W [rpc-transport.c:270:rpc_transport_load]
>0-rpc-transport: volume 'rdma.management': transport-
>type 'rdma' is not valid or not found on this machine
>[2015-06-05 01:59:02.329637] W [rpcsvc.c:1524:rpcsvc_transport_create]
>0-rpc-service: cannot create listener, initing the tra
>nsport failed
>[2015-06-05 01:59:02.343346] I
>[glusterd.c:413:glusterd_check_gsync_present] 0-glusterd:
>geo-replication module not installed
> in the system
>[2015-06-05 01:59:02.351861] I
>[glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd:
>retrieved op-version: 2
>[2015-06-05 01:59:02.363138] I
>[glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list]
>0-management: No missed snap
>s list.
>Final graph:
>+------------------------------------------------------------------------------+
> 1: volume management
> 2: type mgmt/glusterd
> 3: option rpc-auth.auth-glusterfs on
> 4: option rpc-auth.auth-unix on
> 5: option rpc-auth.auth-null on
> 6: option transport.socket.listen-backlog 128
> 7: option ping-timeout 30
> 8: option transport.socket.read-fail-log off
> 9: option transport.socket.keepalive-interval 2
> 10: option transport.socket.keepalive-time 10
> 11: option transport-type rdma
> 12: option working-directory /var/lib/glusterd
> 13: end-volume
> 14:
>+------------------------------------------------------------------------------+
>[2015-06-05 02:00:43.025378] I [glusterd.c:146:glusterd_uuid_init]
>0-management: retrieved UUID: 1b98af34-08f9-492f-b54e-7608
>2588d63b
>[2015-06-05 02:00:43.025462] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>[2015-06-05 02:00:44.025069] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>[2015-06-05 02:00:44.599862] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:00:46.029823] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:00:46.701845] I
>[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd:
>Received get vol req
>[2015-06-05 02:01:43.119612] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>[2015-06-05 02:01:44.165527] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>[2015-06-05 02:01:44.851903] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:01:46.190401] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:01:46.722394] I
>[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd:
>Received get vol req
>[2015-06-05 02:01:58.719968] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:02:32.584999] I
>[glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd:
>Received CLI probe req glust
>erfs05.sh2.ctripcorp.com 24007
>[2015-06-05 02:02:32.586281] I
>[glusterd-handler.c:3165:glusterd_probe_begin] 0-glusterd: Unable to
>find peerinfo for host: g
>lusterfs05.sh2.ctripcorp.com (24007)
>[2015-06-05 02:02:32.602192] I
>[rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting
>frame-timeout to 600
>[2015-06-05 02:02:32.606053] I
>[glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect
>returned 0
>[2015-06-05 02:02:32.620958] E
>[glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk]
>0-management: failed to get the
>'versions' from peer (10.8.230.209:24007)
>[2015-06-05 02:02:32.621020] I [MSGID: 106004]
>[glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management: Peer
>000000
>00-0000-0000-0000-000000000000, in Establishing Connection state, has
>disconnected from glusterd.
>[2015-06-05 02:02:43.264141] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>[2015-06-05 02:02:44.312918] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>[2015-06-05 02:02:44.781434] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:02:46.176888] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:02:46.799916] I
>[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd:
>Received get vol req
>[2015-06-05 02:03:42.720994] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>[2015-06-05 02:03:43.771987] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>[2015-06-05 02:03:44.659775] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:03:45.956807] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:03:46.688787] I
>[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd:
>Received get vol req
>[2015-06-05 02:04:42.891131] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>[2015-06-05 02:04:43.957423] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>[2015-06-05 02:04:44.995638] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:04:46.062171] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:04:46.957327] I
>[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd:
>Received get vol req
>[2015-06-05 02:05:42.966651] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>[2015-06-05 02:05:44.008498] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>[2015-06-05 02:05:45.049936] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:05:46.087007] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:05:47.029325] I
>[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd:
>Received get vol req
>[2015-06-05 02:06:42.874089] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>[2015-06-05 02:06:43.910998] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>[2015-06-05 02:06:44.932802] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:06:46.053731] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:06:47.055859] I
>[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd:
>Received get vol req
>[2015-06-05 02:07:43.099647] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>[2015-06-05 02:07:44.146454] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>[2015-06-05 02:07:45.218982] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:07:46.246303] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:07:47.106954] I
>[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd:
>Received get vol req
>[2015-06-05 02:08:43.226090] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>[2015-06-05 02:08:44.292041] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>[2015-06-05 02:08:45.351830] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:08:45.688349] I
>[glusterd-handler.c:1225:__glusterd_handle_cli_list_friends]
>0-glusterd: Received cli list req
>[2015-06-05 02:08:46.651768] I
>[glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd:
>Received get vol req
>[2015-06-05 02:09:42.748628] W
>[glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx
>modification failed
>
>
>
>Best Regards
>
>
>-----邮件原件-----
>发件人: vyyy杨雨阳
>发送时间: Thursday, June 04, 2015 11:00 AM
>收件人: 'Atin Mukherjee'; Gluster-users at gluster.org
>主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to
>start
>
>
>The problem 2 has resolved.
>
>From glusterd log file, I find Cksums error as following:
>[2015-06-04 01:25:03.137798] E [MSGID: 106010]
>[glusterd-utils.c:3297:glusterd_compare_friend_volume] 0-management:
>Version of Cksums JQStore2 differ. local cksum = 2617028277, remote
>cksum = 2210669223 on peer glusterfs09.sh2.ctripcorp.com
>
>The solution for this problem was to do the following, on the BAD node:
>/etc/init.d/glusterd stop
>rsync -havP --delete 172.26.177.115:/var/lib/glusterd/vols/
>/var/lib/glusterd/vols/
>/etc/init.d/glusterd start
>
>
>
>The problem 1 is still under inverstigate:
>
>New nodes can not join the cluster, log file shown: failed to get the
>'versions' from peer
>I have checked the file /var/lib/glusterd/ glusterd.info, all nodes are
>" operating-version=2"
>
>
> [2015-06-04 02:13:49.199416] E
>[glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk]
>0-management: failed to get the
> 'versions' from peer (10.8.230.211:24007)
> [2015-06-04 02:14:07.112577] I
>[glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd:
>Received CLI probe req glust
> erfs05.sh2.ctripcorp.com 24007
> [2015-06-04 02:14:07.113778] I
>[glusterd-handler.c:3165:glusterd_probe_begin] 0-glusterd: Unable to
>find peerinfo for host: g
> lusterfs05.sh2.ctripcorp.com (24007)
> [2015-06-04 02:14:07.131023] I
>[rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting
>frame-timeout to 600
> [2015-06-04 02:14:07.134829] I
>[glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect
>returned 0
> [
>
>
>
>
>Best Regards
>杨雨阳
>Site Operations Center
>Ext.15554 Seat.1#3F049
>We are the F.I.R.S.T.
>Fast, Innovation, Reliability , Secure, Team
>
>
>-----邮件原件-----
>发件人: vyyy杨雨阳
>发送时间: Wednesday, June 03, 2015 5:37 PM
>收件人: 'Atin Mukherjee'; Gluster-users at gluster.org
>主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to
>start
>
>Hi Atin and everybody,
>
> I have forced peer detach the news nodes ,The peer status seems even
>strange, glusterfs05 and glusterfs10 are peer in cluster, Can you help
>me? Any advice is appreciated.
>
>Glusterfs05
>
> [root at sh02svr5956 ~]# gluster peer status
> Number of Peers: 5
>
> Hostname: glusterfs06.sh2.ctripcorp.com
> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
> State: Peer Rejected (Connected)
>
> Hostname: glusterfs07.sh2.ctripcorp.com
> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
> State: Peer Rejected (Connected)
>
> Hostname: glusterfs08.sh2.ctripcorp.com
> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
> State: Peer Rejected (Connected)
>
> Hostname: glusterfs10.sh2.ctripcorp.com
> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
> State: Peer in Cluster (Connected)
>
> Hostname: glusterfs09.sh2.ctripcorp.com
> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
> State: Peer Rejected (Connected)
> [root at sh02svr5956 ~]#
>
>
> Glusterfs06
>
> [root at SH02SVR5955 ~]# gluster peer status
> Number of Peers: 5
>
> Hostname: glusterfs10.sh2.ctripcorp.com
> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
> State: Peer Rejected (Connected)
>
> Hostname: glusterfs08.sh2.ctripcorp.com
> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
> State: Peer in Cluster (Connected)
>
> Hostname: glusterfs07.sh2.ctripcorp.com
> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
> State: Peer in Cluster (Connected)
>
> Hostname: 10.8.230.209
> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
> State: Peer Rejected (Connected)
>
> Hostname: glusterfs09.sh2.ctripcorp.com
> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
> State: Peer in Cluster (Connected)
> [root at SH02SVR5955 ~]#
>
>
>Glusterfs07
>
>[root at SH02SVR5954 ~]# gluster peer status
>Number of Peers: 5
>
>Hostname: glusterfs10.sh2.ctripcorp.com
>Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>State: Peer Rejected (Connected)
>
>Hostname: 10.8.230.209
>Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>State: Peer Rejected (Connected)
>
>Hostname: glusterfs06.sh2.ctripcorp.com
>Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>State: Peer in Cluster (Connected)
>
>Hostname: glusterfs08.sh2.ctripcorp.com
>Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>State: Peer in Cluster (Connected)
>
>Hostname: glusterfs09.sh2.ctripcorp.com
>Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>State: Peer in Cluster (Connected)
>[root at SH02SVR5954 ~]#
>
>
>
>Glusterfs08
>
>[root at SH02SVR5953 ~]# gluster peer status
>Number of Peers: 5
>
>Hostname: glusterfs07.sh2.ctripcorp.com
>Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>State: Peer in Cluster (Connected)
>
>Hostname: glusterfs10.sh2.ctripcorp.com
>Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>State: Peer Rejected (Connected)
>
>Hostname: glusterfs06.sh2.ctripcorp.com
>Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>State: Peer in Cluster (Connected)
>
>Hostname: 10.8.230.209
>Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>State: Peer Rejected (Connected)
>
>Hostname: glusterfs09.sh2.ctripcorp.com
>Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>State: Peer in Cluster (Connected)
>[root at SH02SVR5953 ~]#
>
>
>[root at SH02SVR5952 ~]# gluster peer status
>Number of Peers: 5
>
>Hostname: glusterfs06.sh2.ctripcorp.com
>Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>State: Peer in Cluster (Connected)
>
>Hostname: glusterfs07.sh2.ctripcorp.com
>Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>State: Peer in Cluster (Connected)
>
>Hostname: glusterfs08.sh2.ctripcorp.com
>Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>State: Peer in Cluster (Connected)
>
>Hostname: 10.8.230.209
>Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>State: Peer Rejected (Connected)
>
>Hostname: glusterfs10.sh2.ctripcorp.com
>Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>State: Peer Rejected (Connected)
>[root at SH02SVR5952 ~]#
>
>
>Glusterfs10
>
>[root at SH02SVR5951 ~]# gluster peer status
>Number of Peers: 6
>
>Hostname: glusterfs07.sh2.ctripcorp.com
>Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>State: Peer Rejected (Connected)
>
>Hostname: glusterfs11.sh2.ctripcorp.com
>Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
>State: Peer Rejected (Connected)
>
>Hostname: glusterfs09.sh2.ctripcorp.com
>Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>State: Peer Rejected (Connected)
>
>Hostname: glusterfs06.sh2.ctripcorp.com
>Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>State: Peer Rejected (Connected)
>
>Hostname: 10.8.230.209
>Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>State: Peer in Cluster (Connected)
>
>Hostname: glusterfs08.sh2.ctripcorp.com
>Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>State: Peer Rejected (Connected)
>[root at SH02SVR5951 ~]#
>
>
>
>
>
>
>
>Best Regards
>杨雨阳
>Site Operations Center
>Ext.15554 Seat.1#3F049
>We are the F.I.R.S.T.
>Fast, Innovation, Reliability , Secure, Team
>
>
>
>-----邮件原件-----
>发件人: vyyy杨雨阳
>发送时间: Wednesday, June 03, 2015 2:55 PM
>收件人: 'Atin Mukherjee'
>主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to
>start
>
>Can I delete the files in /var/lib/glusterd/peers which state is not =3
>on all nodes, then restart glusterd?
>
>
>[root at sh02svr5956 peers]# more 0475e11a-3c43-4b76-99dd-1d2e814c5f15
>uuid=0475e11a-3c43-4b76-99dd-1d2e814c5f15
>state=6
>hostname1=glusterfs12.sh2.ctripcorp.com
>
>
>
>
>Best Regards
>Yang
>
>
>-----邮件原件-----
>发件人: vyyy杨雨阳
>发送时间: Wednesday, June 03, 2015 2:39 PM
>收件人: 'Atin Mukherjee'
>主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to
>start
>
>Since it's intranet, You can only control my desktop, and then access
>these node.
>Should I apply a webex room?
>
>
>Best Regards
>杨雨阳
>Site Operations Center
>Ext.15554 Seat.1#3F049
>We are the F.I.R.S.T.
>Fast, Innovation, Reliability , Secure, Team
>
>
>
>-----邮件原件-----
>发件人: Atin Mukherjee [mailto:amukherj at redhat.com]
>发送时间: Wednesday, June 03, 2015 2:36 PM
>收件人: vyyy杨雨阳
>主题: Re: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to
>start
>
>
>Is it possible to share your ip/user/passwd for few of these nodes?
>
>On 06/03/2015 11:50 AM, vyyy杨雨阳 wrote:
>> Can you remote control my desktop (like webex)?
>> The nodes host bricks. And on the new nodes, I did "clean
>/var/lib/glusterd and restart" several times, but have no effect.
>> Is there any other work around?
>>
>>
>> Best Regards
>> Yang
>>
>> -----邮件原件-----
>> 发件人: Atin Mukherjee [mailto:amukherj at redhat.com]
>> 发送时间: Wednesday, June 03, 2015 12:33 PM
>> 收件人: vyyy杨雨阳; Gluster-users at gluster.org
>> 主题: Re: 答复: [Gluster-users] Gluster peer rejected and failed to start
>>
>> I can't comment on what went wrong till I can have visibility on the
>> entire cluster. Does the node which fails to start host any brick? If
>> not then work around could be to clean /var/lib/glusterd and restart
>> glusterd, that should sync up data and bring this node in
>consistency.
>>
>> HTH,
>> Atin
>>
>> On 06/03/2015 08:34 AM, vyyy杨雨阳 wrote:
>>> I found that /var/lib/glusterd/peers can't include node's own peer,
>I delete this file. Glusterd started finally.
>>> But the peer status still not normal.
>>> Seems that glusterfs10 leave the cluster and join the new nodes
>alone.
>>>
>>> Peer status on glusterfs10
>>> [root at SH02SVR5951 ~]# gluster peer status
>>> Number of Peers: 7
>>>
>>> Hostname: glusterfs07.sh2.ctripcorp.com
>>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>>> State: Peer Rejected (Connected)
>>>
>>> Hostname: glusterfs11.sh2.ctripcorp.com
>>> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
>>> State: Accepted peer request (Connected)
>>>
>>> Hostname: glusterfs09.sh2.ctripcorp.com
>>> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>>> State: Peer Rejected (Connected)
>>>
>>> Hostname: glusterfs06.sh2.ctripcorp.com
>>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>>> State: Peer Rejected (Connected)
>>>
>>> Hostname: 10.8.230.209
>>> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>>> State: Peer Rejected (Connected)
>>>
>>> Hostname: glusterfs08.sh2.ctripcorp.com
>>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>>> State: Peer Rejected (Connected)
>>>
>>> Hostname: glusterfs12.sh2.ctripcorp.com
>>> Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15
>>> State: Peer in Cluster (Connected)
>>>
>>>
>>>
>>> Peer status on glusterfs10
>>>
>>> [root at SVR6993HW2285 ~]# gluster peer status
>>> Number of Peers: 6
>>>
>>> Hostname: glusterfs05.sh2.ctripcorp.com
>>> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>>> State: Peer Rejected (Connected)
>>>
>>> Hostname: glusterfs09.sh2.ctripcorp.com
>>> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>>> State: Peer Rejected (Connected)
>>>
>>> Hostname: glusterfs07.sh2.ctripcorp.com
>>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>>> State: Peer Rejected (Connected)
>>>
>>> Hostname: glusterfs08.sh2.ctripcorp.com
>>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>>> State: Peer Rejected (Connected)
>>>
>>> Hostname: glusterfs06.sh2.ctripcorp.com
>>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>>> State: Peer Rejected (Connected)
>>>
>>> Hostname: glusterfs10.sh2.ctripcorp.com
>>> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>>> State: Peer in Cluster (Connected)
>>>
>>>
>>> Peer status on glusterfs05, this is the node that I tried to peer
>probe new nodes (glusterfs11 to glusterfs14)
>>>
>>> [root at sh02svr5956 ~]# gluster peer status
>>> Number of Peers: 9
>>>
>>> Hostname: glusterfs06.sh2.ctripcorp.com
>>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>>> State: Peer in Cluster (Connected)
>>>
>>> Hostname: glusterfs07.sh2.ctripcorp.com
>>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>>> State: Peer in Cluster (Connected)
>>>
>>> Hostname: glusterfs08.sh2.ctripcorp.com
>>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>>> State: Peer in Cluster (Connected)
>>>
>>> Hostname: glusterfs10.sh2.ctripcorp.com
>>> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>>> State: Peer Rejected (Connected)
>>>
>>> Hostname: glusterfs09.sh2.ctripcorp.com
>>> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>>> State: Peer in Cluster (Connected)
>>>
>>> Hostname: glusterfs11.sh2.ctripcorp.com
>>> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
>>> State: Peer Rejected (Connected)
>>>
>>> Hostname: glusterfs12.sh2.ctripcorp.com
>>> Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15
>>> State: Peer in Cluster (Connected)
>>>
>>> Hostname: glusterfs13.sh2.ctripcorp.com
>>> Uuid: 43e96d31-6c6d-4096-832a-440e544554ca
>>> State: Peer Rejected (Connected)
>>>
>>> Hostname: glusterfs14.sh2.ctripcorp.com
>>> Uuid: 45c11af9-7850-4598-ad62-4bb343b62524
>>> State: Peer Rejected (Connected)
>>> [root at sh02svr5956 ~]#
>>>
>>>
>>>
>>>
>>> Best Regards
>>> Yang
>>>
>>>
>>> -----邮件原件-----
>>> 发件人: vyyy杨雨阳
>>> 发送时间: Wednesday, June 03, 2015 10:04 AM
>>> 收件人: 'Atin Mukherjee'; 'Gluster-users at gluster.org'
>>> 主题: [Gluster-users] Gluster peer rejected and failed to start
>>>
>>> I am still messed, can you help me figure out problem 2 (A node of
>cluster failed to start glusterd) priority
>>>
>>> The glusterd log on the node gluster10 shown ' resolve brick failed
>in restore'
>>> I checked /var/lib/glusterd/vols, seems ok
>>>
>>> Glusterd log file Error message:
>>>
>>> [2015-06-03 01:31:32.564010] E
>[glusterd-store.c:4244:glusterd_resolve_all_bricks] 0-glusterd: resolve
>brick failed in restore
>>> [2015-06-03 01:31:32.564036] E [xlator.c:425:xlator_init]
>0-management: Initialization of volume 'management' failed, review
>>> your volfile again
>>>
>>>
>>>
>>>
>>> Best Regards
>>> Yang
>>>
>>> -----邮件原件-----
>>> 发件人: vyyy杨雨阳
>>> 发送时间: Tuesday, June 02, 2015 6:27 PM
>>> 收件人: 'Atin Mukherjee'; Gluster-users at gluster.org
>>> 主题: 答复: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed
>to start
>>>
>>> Gluster11 to gluster14 are new nodes I tried to join exit cluster
>but failed
>>>
>>>
>>> On gluster12:
>>> [root at SVR6994HW2285 ~]# gluster peer status
>>> Number of Peers: 1
>>>
>>> Hostname: glusterfs05.sh2.ctripcorp.com
>>> Uuid: 04f22ee8-8e0
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150604/eb213d49/attachment.html>
More information about the Gluster-users
mailing list