[Gluster-users] 答复: 回复： 答复: 答复: 答复: Gluster peer rejected and failed to start
Atin Mukherjee
amukherj at redhat.com
Fri Jun 5 03:57:29 UTC 2015
Give me the glusterd log of 10.8.230.209.
~Atin
On 06/05/2015 08:34 AM, vyyy杨雨阳 wrote:
> Selinux is disabled on all server
>
> # SELINUX= can take one of these three values:
> # enforcing - SELinux security policy is enforced.
> # permissive - SELinux prints warnings instead of enforcing.
> # disabled - SELinux is fully disabled.
> SELINUX=disabled
>
>
>
> Best Regards
> 杨雨阳
>
> Site Operations Center
> Ext.15554 Seat.1#3F049
> We are the F.I.R.S.T.
> Fast, Innovation, Reliability , Secure, Team
>
>
>
> 发件人: huangql [mailto:huangql at ihep.ac.cn]
> 发送时间: Friday, June 05, 2015 10:57 AM
> 收件人: Atin Mukherjee; Gluster-users at gluster.org; vyyy杨雨阳
> 主题: 回复：[Gluster-users] 答复: 答复: 答复: Gluster peer rejected and failed to start
>
>
> Maybe you can check selinux on the server side. Make sure selinux is disabled.
>
> 原始邮件
> 发件人: vyyy杨雨阳<yuyangyang at Ctrip.com<mailto:yuyangyang at Ctrip.com>>
> 收件人: Atin Mukherjee<amukherj at redhat.com<mailto:amukherj at redhat.com>>; Gluster-users at gluster.org<Gluster-users at gluster.org<mailto:Gluster-users at gluster.org%3cGluster-users at gluster.org>>
> 发送时间: 2015年6月4日(周四) 21:24
> 主题: [Gluster-users] 答复: 答复: 答复: Gluster peer rejected and failed to start
>
>
> Can anyone help me?
>
>
>
> New nodes still can't join the cluster.
>
> The new nodes can peer each other, just not exist cluster nodes.
>
> I can not find any network problem, there were no firewall, port 24007 is connected, there were all at the same subnet...
>
>
>
> Thanks
>
>
>
>
>
>
>
> [root at SVR6993HW2285 glusterfs]# more etc-glusterfs-glusterd.vol.log
>
> [2015-06-05 01:59:02.276107] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glust
>
> erd version 3.6.3 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
>
> [2015-06-05 01:59:02.318477] I [glusterd.c:1214:init] 0-management: Maximum allowed open file descriptors set to 65536
>
> [2015-06-05 01:59:02.318524] I [glusterd.c:1259:init] 0-management: Using /var/lib/glusterd as working directory
>
> [2015-06-05 01:59:02.329603] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-trans
>
> port/rdma.so: cannot open shared object file: No such file or directory
>
> [2015-06-05 01:59:02.329625] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-
>
> type 'rdma' is not valid or not found on this machine
>
> [2015-06-05 01:59:02.329637] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the tra
>
> nsport failed
>
> [2015-06-05 01:59:02.343346] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed
>
> in the system
>
> [2015-06-05 01:59:02.351861] I [glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2
>
> [2015-06-05 01:59:02.363138] I [glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snap
>
> s list.
>
> Final graph:
>
> +------------------------------------------------------------------------------+
>
> 1: volume management
>
> 2: type mgmt/glusterd
>
> 3: option rpc-auth.auth-glusterfs on
>
> 4: option rpc-auth.auth-unix on
>
> 5: option rpc-auth.auth-null on
>
> 6: option transport.socket.listen-backlog 128
>
> 7: option ping-timeout 30
>
> 8: option transport.socket.read-fail-log off
>
> 9: option transport.socket.keepalive-interval 2
>
> 10: option transport.socket.keepalive-time 10
>
> 11: option transport-type rdma
>
> 12: option working-directory /var/lib/glusterd
>
> 13: end-volume
>
> 14:
>
> +------------------------------------------------------------------------------+
>
> [2015-06-05 02:00:43.025378] I [glusterd.c:146:glusterd_uuid_init] 0-management: retrieved UUID: 1b98af34-08f9-492f-b54e-7608
>
> 2588d63b
>
> [2015-06-05 02:00:43.025462] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
> [2015-06-05 02:00:44.025069] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
> [2015-06-05 02:00:44.599862] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:00:46.029823] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:00:46.701845] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-06-05 02:01:43.119612] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
> [2015-06-05 02:01:44.165527] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
> [2015-06-05 02:01:44.851903] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:01:46.190401] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:01:46.722394] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-06-05 02:01:58.719968] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:02:32.584999] I [glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glust
>
> erfs05.sh2.ctripcorp.com 24007
>
> [2015-06-05 02:02:32.586281] I [glusterd-handler.c:3165:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: g
>
> lusterfs05.sh2.ctripcorp.com (24007)
>
> [2015-06-05 02:02:32.602192] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>
> [2015-06-05 02:02:32.606053] I [glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect returned 0
>
> [2015-06-05 02:02:32.620958] E [glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk] 0-management: failed to get the
>
> 'versions' from peer (10.8.230.209:24007)
>
> [2015-06-05 02:02:32.621020] I [MSGID: 106004] [glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management: Peer 000000
>
> 00-0000-0000-0000-000000000000, in Establishing Connection state, has disconnected from glusterd.
>
> [2015-06-05 02:02:43.264141] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
> [2015-06-05 02:02:44.312918] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
> [2015-06-05 02:02:44.781434] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:02:46.176888] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:02:46.799916] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-06-05 02:03:42.720994] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
> [2015-06-05 02:03:43.771987] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
> [2015-06-05 02:03:44.659775] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:03:45.956807] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:03:46.688787] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-06-05 02:04:42.891131] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
> [2015-06-05 02:04:43.957423] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
> [2015-06-05 02:04:44.995638] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:04:46.062171] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:04:46.957327] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-06-05 02:05:42.966651] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
> [2015-06-05 02:05:44.008498] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
> [2015-06-05 02:05:45.049936] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:05:46.087007] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:05:47.029325] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-06-05 02:06:42.874089] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
> [2015-06-05 02:06:43.910998] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
> [2015-06-05 02:06:44.932802] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:06:46.053731] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:06:47.055859] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-06-05 02:07:43.099647] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
> [2015-06-05 02:07:44.146454] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
> [2015-06-05 02:07:45.218982] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:07:46.246303] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:07:47.106954] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-06-05 02:08:43.226090] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
> [2015-06-05 02:08:44.292041] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
> [2015-06-05 02:08:45.351830] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:08:45.688349] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-06-05 02:08:46.651768] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-06-05 02:09:42.748628] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
>
>
>
>
>
>
> Best Regards
>
>
>
>
>
> -----邮件原件-----
>
> 发件人: vyyy杨雨阳
>
> 发送时间: Thursday, June 04, 2015 11:00 AM
>
> 收件人: 'Atin Mukherjee'; Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
>
> 主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start
>
>
>
>
>
> The problem 2 has resolved.
>
>
>
> From glusterd log file, I find Cksums error as following:
>
> [2015-06-04 01:25:03.137798] E [MSGID: 106010] [glusterd-utils.c:3297:glusterd_compare_friend_volume] 0-management: Version of Cksums JQStore2 differ. local cksum = 2617028277, remote cksum = 2210669223 on peer glusterfs09.sh2.ctripcorp.com
>
>
>
> The solution for this problem was to do the following, on the BAD node:
>
> /etc/init.d/glusterd stop
>
> rsync -havP --delete 172.26.177.115:/var/lib/glusterd/vols/ /var/lib/glusterd/vols/
>
> /etc/init.d/glusterd start
>
>
>
>
>
>
>
> The problem 1 is still under inverstigate:
>
>
>
> New nodes can not join the cluster, log file shown: failed to get the 'versions' from peer
>
> I have checked the file /var/lib/glusterd/ glusterd.info, all nodes are " operating-version=2"
>
>
>
>
>
> [2015-06-04 02:13:49.199416] E [glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk] 0-management: failed to get the
>
> 'versions' from peer (10.8.230.211:24007)
>
> [2015-06-04 02:14:07.112577] I [glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glust
>
> erfs05.sh2.ctripcorp.com 24007
>
> [2015-06-04 02:14:07.113778] I [glusterd-handler.c:3165:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: g
>
> lusterfs05.sh2.ctripcorp.com (24007)
>
> [2015-06-04 02:14:07.131023] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>
> [2015-06-04 02:14:07.134829] I [glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect returned 0
>
> [
>
>
>
>
>
>
>
>
>
> Best Regards
>
> 杨雨阳
>
> Site Operations Center
>
> Ext.15554 Seat.1#3F049
>
> We are the F.I.R.S.T.
>
> Fast, Innovation, Reliability , Secure, Team
>
>
>
>
>
> -----邮件原件-----
>
> 发件人: vyyy杨雨阳
>
> 发送时间: Wednesday, June 03, 2015 5:37 PM
>
> 收件人: 'Atin Mukherjee'; Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
>
> 主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start
>
>
>
> Hi Atin and everybody,
>
>
>
> I have forced peer detach the news nodes ,The peer status seems even strange, glusterfs05 and glusterfs10 are peer in cluster, Can you help me? Any advice is appreciated.
>
>
>
> Glusterfs05
>
>
>
> [root at sh02svr5956 ~]# gluster peer status
>
> Number of Peers: 5
>
>
>
> Hostname: glusterfs06.sh2.ctripcorp.com
>
> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>
> State: Peer Rejected (Connected)
>
>
>
> Hostname: glusterfs07.sh2.ctripcorp.com
>
> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>
> State: Peer Rejected (Connected)
>
>
>
> Hostname: glusterfs08.sh2.ctripcorp.com
>
> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>
> State: Peer Rejected (Connected)
>
>
>
> Hostname: glusterfs10.sh2.ctripcorp.com
>
> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>
> State: Peer in Cluster (Connected)
>
>
>
> Hostname: glusterfs09.sh2.ctripcorp.com
>
> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>
> State: Peer Rejected (Connected)
>
> [root at sh02svr5956 ~]#
>
>
>
>
>
> Glusterfs06
>
>
>
> [root at SH02SVR5955 ~]# gluster peer status
>
> Number of Peers: 5
>
>
>
> Hostname: glusterfs10.sh2.ctripcorp.com
>
> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>
> State: Peer Rejected (Connected)
>
>
>
> Hostname: glusterfs08.sh2.ctripcorp.com
>
> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>
> State: Peer in Cluster (Connected)
>
>
>
> Hostname: glusterfs07.sh2.ctripcorp.com
>
> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>
> State: Peer in Cluster (Connected)
>
>
>
> Hostname: 10.8.230.209
>
> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>
> State: Peer Rejected (Connected)
>
>
>
> Hostname: glusterfs09.sh2.ctripcorp.com
>
> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>
> State: Peer in Cluster (Connected)
>
> [root at SH02SVR5955 ~]#
>
>
>
>
>
> Glusterfs07
>
>
>
> [root at SH02SVR5954 ~]# gluster peer status
>
> Number of Peers: 5
>
>
>
> Hostname: glusterfs10.sh2.ctripcorp.com
>
> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>
> State: Peer Rejected (Connected)
>
>
>
> Hostname: 10.8.230.209
>
> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>
> State: Peer Rejected (Connected)
>
>
>
> Hostname: glusterfs06.sh2.ctripcorp.com
>
> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>
> State: Peer in Cluster (Connected)
>
>
>
> Hostname: glusterfs08.sh2.ctripcorp.com
>
> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>
> State: Peer in Cluster (Connected)
>
>
>
> Hostname: glusterfs09.sh2.ctripcorp.com
>
> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>
> State: Peer in Cluster (Connected)
>
> [root at SH02SVR5954 ~]#
>
>
>
>
>
>
>
> Glusterfs08
>
>
>
> [root at SH02SVR5953 ~]# gluster peer status
>
> Number of Peers: 5
>
>
>
> Hostname: glusterfs07.sh2.ctripcorp.com
>
> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>
> State: Peer in Cluster (Connected)
>
>
>
> Hostname: glusterfs10.sh2.ctripcorp.com
>
> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>
> State: Peer Rejected (Connected)
>
>
>
> Hostname: glusterfs06.sh2.ctripcorp.com
>
> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>
> State: Peer in Cluster (Connected)
>
>
>
> Hostname: 10.8.230.209
>
> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>
> State: Peer Rejected (Connected)
>
>
>
> Hostname: glusterfs09.sh2.ctripcorp.com
>
> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>
> State: Peer in Cluster (Connected)
>
> [root at SH02SVR5953 ~]#
>
>
>
>
>
> [root at SH02SVR5952 ~]# gluster peer status
>
> Number of Peers: 5
>
>
>
> Hostname: glusterfs06.sh2.ctripcorp.com
>
> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>
> State: Peer in Cluster (Connected)
>
>
>
> Hostname: glusterfs07.sh2.ctripcorp.com
>
> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>
> State: Peer in Cluster (Connected)
>
>
>
> Hostname: glusterfs08.sh2.ctripcorp.com
>
> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>
> State: Peer in Cluster (Connected)
>
>
>
> Hostname: 10.8.230.209
>
> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>
> State: Peer Rejected (Connected)
>
>
>
> Hostname: glusterfs10.sh2.ctripcorp.com
>
> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>
> State: Peer Rejected (Connected)
>
> [root at SH02SVR5952 ~]#
>
>
>
>
>
> Glusterfs10
>
>
>
> [root at SH02SVR5951 ~]# gluster peer status
>
> Number of Peers: 6
>
>
>
> Hostname: glusterfs07.sh2.ctripcorp.com
>
> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>
> State: Peer Rejected (Connected)
>
>
>
> Hostname: glusterfs11.sh2.ctripcorp.com
>
> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
>
> State: Peer Rejected (Connected)
>
>
>
> Hostname: glusterfs09.sh2.ctripcorp.com
>
> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>
> State: Peer Rejected (Connected)
>
>
>
> Hostname: glusterfs06.sh2.ctripcorp.com
>
> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>
> State: Peer Rejected (Connected)
>
>
>
> Hostname: 10.8.230.209
>
> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>
> State: Peer in Cluster (Connected)
>
>
>
> Hostname: glusterfs08.sh2.ctripcorp.com
>
> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>
> State: Peer Rejected (Connected)
>
> [root at SH02SVR5951 ~]#
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Best Regards
>
> 杨雨阳
>
> Site Operations Center
>
> Ext.15554 Seat.1#3F049
>
> We are the F.I.R.S.T.
>
> Fast, Innovation, Reliability , Secure, Team
>
>
>
>
>
>
>
> -----邮件原件-----
>
> 发件人: vyyy杨雨阳
>
> 发送时间: Wednesday, June 03, 2015 2:55 PM
>
> 收件人: 'Atin Mukherjee'
>
> 主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start
>
>
>
> Can I delete the files in /var/lib/glusterd/peers which state is not =3 on all nodes, then restart glusterd?
>
>
>
>
>
> [root at sh02svr5956 peers]# more 0475e11a-3c43-4b76-99dd-1d2e814c5f15
>
> uuid=0475e11a-3c43-4b76-99dd-1d2e814c5f15
>
> state=6
>
> hostname1=glusterfs12.sh2.ctripcorp.com
>
>
>
>
>
>
>
>
>
> Best Regards
>
> Yang
>
>
>
>
>
> -----邮件原件-----
>
> 发件人: vyyy杨雨阳
>
> 发送时间: Wednesday, June 03, 2015 2:39 PM
>
> 收件人: 'Atin Mukherjee'
>
> 主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start
>
>
>
> Since it's intranet, You can only control my desktop, and then access these node.
>
> Should I apply a webex room?
>
>
>
>
>
> Best Regards
>
> 杨雨阳
>
> Site Operations Center
>
> Ext.15554 Seat.1#3F049
>
> We are the F.I.R.S.T.
>
> Fast, Innovation, Reliability , Secure, Team
>
>
>
>
>
>
>
> -----邮件原件-----
>
> 发件人: Atin Mukherjee [mailto:amukherj at redhat.com<mailto:amukherj at redhat.com>]
>
> 发送时间: Wednesday, June 03, 2015 2:36 PM
>
> 收件人: vyyy杨雨阳
>
> 主题: Re: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start
>
>
>
>
>
> Is it possible to share your ip/user/passwd for few of these nodes?
>
>
>
> On 06/03/2015 11:50 AM, vyyy杨雨阳 wrote:
>
>> Can you remote control my desktop (like webex)?
>
>> The nodes host bricks. And on the new nodes, I did "clean /var/lib/glusterd and restart" several times, but have no effect.
>
>> Is there any other work around?
>
>>
>
>>
>
>> Best Regards
>
>> Yang
>
>>
>
>> -----邮件原件-----
>
>> 发件人: Atin Mukherjee [mailto:amukherj at redhat.com<mailto:amukherj at redhat.com>]
>
>> 发送时间: Wednesday, June 03, 2015 12:33 PM
>
>> 收件人: vyyy杨雨阳; Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
>
>> 主题: Re: 答复: [Gluster-users] Gluster peer rejected and failed to start
>
>>
>
>> I can't comment on what went wrong till I can have visibility on the
>
>> entire cluster. Does the node which fails to start host any brick? If
>
>> not then work around could be to clean /var/lib/glusterd and restart
>
>> glusterd, that should sync up data and bring this node in consistency.
>
>>
>
>> HTH,
>
>> Atin
>
>>
>
>> On 06/03/2015 08:34 AM, vyyy杨雨阳 wrote:
>
>>> I found that /var/lib/glusterd/peers can't include node's own peer, I delete this file. Glusterd started finally.
>
>>> But the peer status still not normal.
>
>>> Seems that glusterfs10 leave the cluster and join the new nodes alone.
>
>>>
>
>>> Peer status on glusterfs10
>
>>> [root at SH02SVR5951 ~]# gluster peer status
>
>>> Number of Peers: 7
>
>>>
>
>>> Hostname: glusterfs07.sh2.ctripcorp.com
>
>>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>
>>> State: Peer Rejected (Connected)
>
>>>
>
>>> Hostname: glusterfs11.sh2.ctripcorp.com
>
>>> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
>
>>> State: Accepted peer request (Connected)
>
>>>
>
>>> Hostname: glusterfs09.sh2.ctripcorp.com
>
>>> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>
>>> State: Peer Rejected (Connected)
>
>>>
>
>>> Hostname: glusterfs06.sh2.ctripcorp.com
>
>>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>
>>> State: Peer Rejected (Connected)
>
>>>
>
>>> Hostname: 10.8.230.209
>
>>> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>
>>> State: Peer Rejected (Connected)
>
>>>
>
>>> Hostname: glusterfs08.sh2.ctripcorp.com
>
>>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>
>>> State: Peer Rejected (Connected)
>
>>>
>
>>> Hostname: glusterfs12.sh2.ctripcorp.com
>
>>> Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15
>
>>> State: Peer in Cluster (Connected)
>
>>>
>
>>>
>
>>>
>
>>> Peer status on glusterfs10
>
>>>
>
>>> [root at SVR6993HW2285 ~]# gluster peer status
>
>>> Number of Peers: 6
>
>>>
>
>>> Hostname: glusterfs05.sh2.ctripcorp.com
>
>>> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>
>>> State: Peer Rejected (Connected)
>
>>>
>
>>> Hostname: glusterfs09.sh2.ctripcorp.com
>
>>> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>
>>> State: Peer Rejected (Connected)
>
>>>
>
>>> Hostname: glusterfs07.sh2.ctripcorp.com
>
>>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>
>>> State: Peer Rejected (Connected)
>
>>>
>
>>> Hostname: glusterfs08.sh2.ctripcorp.com
>
>>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>
>>> State: Peer Rejected (Connected)
>
>>>
>
>>> Hostname: glusterfs06.sh2.ctripcorp.com
>
>>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>
>>> State: Peer Rejected (Connected)
>
>>>
>
>>> Hostname: glusterfs10.sh2.ctripcorp.com
>
>>> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>
>>> State: Peer in Cluster (Connected)
>
>>>
>
>>>
>
>>> Peer status on glusterfs05, this is the node that I tried to peer probe new nodes (glusterfs11 to glusterfs14)
>
>>>
>
>>> [root at sh02svr5956 ~]# gluster peer status
>
>>> Number of Peers: 9
>
>>>
>
>>> Hostname: glusterfs06.sh2.ctripcorp.com
>
>>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>
>>> State: Peer in Cluster (Connected)
>
>>>
>
>>> Hostname: glusterfs07.sh2.ctripcorp.com
>
>>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>
>>> State: Peer in Cluster (Connected)
>
>>>
>
>>> Hostname: glusterfs08.sh2.ctripcorp.com
>
>>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>
>>> State: Peer in Cluster (Connected)
>
>>>
>
>>> Hostname: glusterfs10.sh2.ctripcorp.com
>
>>> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>
>>> State: Peer Rejected (Connected)
>
>>>
>
>>> Hostname: glusterfs09.sh2.ctripcorp.com
>
>>> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>
>>> State: Peer in Cluster (Connected)
>
>>>
>
>>> Hostname: glusterfs11.sh2.ctripcorp.com
>
>>> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
>
>>> State: Peer Rejected (Connected)
>
>>>
>
>>> Hostname: glusterfs12.sh2.ctripcorp.com
>
>>> Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15
>
>>> State: Peer in Cluster (Connected)
>
>>>
>
>>> Hostname: glusterfs13.sh2.ctripcorp.com
>
>>> Uuid: 43e96d31-6c6d-4096-832a-440e544554ca
>
>>> State: Peer Rejected (Connected)
>
>>>
>
>>> Hostname: glusterfs14.sh2.ctripcorp.com
>
>>> Uuid: 45c11af9-7850-4598-ad62-4bb343b62524
>
>>> State: Peer Rejected (Connected)
>
>>> [root at sh02svr5956 ~]#
>
>>>
>
>>>
>
>>>
>
>>>
>
>>> Best Regards
>
>>> Yang
>
>>>
>
>>>
>
>>> -----邮件原件-----
>
>>> 发件人: vyyy杨雨阳
>
>>> 发送时间: Wednesday, June 03, 2015 10:04 AM
>
>>> 收件人: 'Atin Mukherjee'; 'Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>'
>
>>> 主题: [Gluster-users] Gluster peer rejected and failed to start
>
>>>
>
>>> I am still messed, can you help me figure out problem 2 (A node of cluster failed to start glusterd) priority
>
>>>
>
>>> The glusterd log on the node gluster10 shown ' resolve brick failed in restore'
>
>>> I checked /var/lib/glusterd/vols, seems ok
>
>>>
>
>>> Glusterd log file Error message:
>
>>>
>
>>> [2015-06-03 01:31:32.564010] E [glusterd-store.c:4244:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore
>
>>> [2015-06-03 01:31:32.564036] E [xlator.c:425:xlator_init] 0-management: Initialization of volume 'management' failed, review
>
>>> your volfile again
>
>>>
>
>>>
>
>>>
>
>>>
>
>>> Best Regards
>
>>> Yang
>
>>>
>
>>> -----邮件原件-----
>
>>> 发件人: vyyy杨雨阳
>
>>> 发送时间: Tuesday, June 02, 2015 6:27 PM
>
>>> 收件人: 'Atin Mukherjee'; Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
>
>>> 主题: 答复: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start
>
>>>
>
>>> Gluster11 to gluster14 are new nodes I tried to join exit cluster but failed
>
>>>
>
>>>
>
>>> On gluster12:
>
>>> [root at SVR6994HW2285 ~]# gluster peer status
>
>>> Number of Peers: 1
>
>>>
>
>>> Hostname: glusterfs05.sh2.ctripcorp.com
>
>>> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>
>>> State: Accepted peer request (Connected)
>
>>> [root at SVR6994HW2285 ~]#
>
>>>
>
>>>
>
>>>
>
>>> On gluster05: (peer status seems strange on all those nodes):
>
>>>
>
>>> [root at sh02svr5956 ~]# gluster peer status
>
>>> Number of Peers: 9
>
>>>
>
>>> Hostname: glusterfs06.sh2.ctripcorp.com
>
>>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>
>>> State: Peer in Cluster (Connected)
>
>>>
>
>>> Hostname: glusterfs07.sh2.ctripcorp.com
>
>>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>
>>> State: Peer in Cluster (Connected)
>
>>>
>
>>> Hostname: glusterfs08.sh2.ctripcorp.com
>
>>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>
>>> State: Peer in Cluster (Connected)
>
>>>
>
>>> Hostname: glusterfs10.sh2.ctripcorp.com
>
>>> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>
>>> State: Peer in Cluster (Disconnected)
>
>>>
>
>>> Hostname: glusterfs09.sh2.ctripcorp.com
>
>>> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>
>>> State: Peer in Cluster (Connected)
>
>>>
>
>>> Hostname: glusterfs11.sh2.ctripcorp.com
>
>>> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
>
>>> State: Peer Rejected (Connected)
>
>>>
>
>>> Hostname: glusterfs12.sh2.ctripcorp.com
>
>>> Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15
>
>>> State: Peer in Cluster (Connected)
>
>>>
>
>>> Hostname: glusterfs13.sh2.ctripcorp.com
>
>>> Uuid: 43e96d31-6c6d-4096-832a-440e544554ca
>
>>> State: Peer Rejected (Connected)
>
>>>
>
>>> Hostname: glusterfs14.sh2.ctripcorp.com
>
>>> Uuid: 45c11af9-7850-4598-ad62-4bb343b62524
>
>>> State: Peer Rejected (Connected)
>
>>> [root at sh02svr5956 ~]#
>
>>>
>
>>>
>
>>>
>
>>> The glusterd log file
>
>>>
>
>>> [root at SVR6994HW2285 glusterfs]# more etc-glusterfs-glusterd.vol.log
>
>>> [2015-06-02 10:23:48.368697] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glust
>
>>> erd version 3.6.3 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
>
>>> [2015-06-02 10:23:48.372587] I [glusterd.c:1214:init] 0-management: Maximum allowed open file descriptors set to 65536
>
>>> [2015-06-02 10:23:48.372625] I [glusterd.c:1259:init] 0-management: Using /var/lib/glusterd as working directory
>
>>> [2015-06-02 10:23:48.374890] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-trans
>
>>> port/rdma.so: cannot open shared object file: No such file or directory
>
>>> [2015-06-02 10:23:48.374908] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-
>
>>> type 'rdma' is not valid or not found on this machine
>
>>> [2015-06-02 10:23:48.374920] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the tra
>
>>> nsport failed
>
>>> [2015-06-02 10:23:48.376539] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed
>
>>> in the system
>
>>> [2015-06-02 10:23:48.377464] I [glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2
>
>>> [2015-06-02 10:23:48.378122] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
>
>>> [2015-06-02 10:23:48.378191] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>
>>> [2015-06-02 10:23:48.383072] I [glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snap
>
>>> s list.
>
>>> Final graph:
>
>>> +------------------------------------------------------------------------------+
>
>>> 1: volume management
>
>>> 2: type mgmt/glusterd
>
>>> 3: option rpc-auth.auth-glusterfs on
>
>>> 4: option rpc-auth.auth-unix on
>
>>> 5: option rpc-auth.auth-null on
>
>>> 6: option transport.socket.listen-backlog 128
>
>>> 7: option ping-timeout 30
>
>>> 8: option transport.socket.read-fail-log off
>
>>> 9: option transport.socket.keepalive-interval 2
>
>>> 10: option transport.socket.keepalive-time 10
>
>>> 11: option transport-type rdma
>
>>> 12: option working-directory /var/lib/glusterd
>
>>> 13: end-volume
>
>>> 14:
>
>>> +------------------------------------------------------------------------------+
>
>>> [2015-06-02 10:23:48.413755] I [glusterd.c:146:glusterd_uuid_init] 0-management: retrieved UUID: 0475e11a-3c43-4b76-99dd-1d2e
>
>>> 814c5f15
>
>>> [2015-06-02 10:23:48.414345] I [glusterd-rpc-ops.c:245:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 04f
>
>>> 22ee8-8e00-4c32-a924-b40a0e413aa6, host: glusterfs05.sh2.ctripcorp.com
>
>>> [2015-06-02 10:23:48.414883] I [glusterd-rpc-ops.c:387:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req
>
>>> [2015-06-02 10:23:50.088645] I [glusterd-rpc-ops.c:437:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 04f22ee
>
>>> 8-8e00-4c32-a924-b40a0e413aa6, host: glusterfs05.sh2.ctripcorp.com, port: 0
>
>>> [2015-06-02 10:23:50.090245] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 04
>
>>> f22ee8-8e00-4c32-a924-b40a0e413aa6
>
>>> [2015-06-02 10:23:50.107013] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.212:1016
>
>>> [2015-06-02 10:23:50.107805] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.211:1018
>
>>> [2015-06-02 10:23:50.108358] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.210:1020
>
>>> [2015-06-02 10:23:50.139963] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.213:1008
>
>>> [2015-06-02 10:23:53.286070] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.213:1006
>
>>> [2015-06-02 10:23:53.893225] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.210:1019
>
>>> [2015-06-02 10:23:53.928314] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.211:1017
>
>>> [2015-06-02 10:23:54.013229] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.212:1014
>
>>> [2015-06-02 10:23:54.273022] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
>>> [2015-06-02 10:23:55.531073] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
>>> [2015-06-02 10:23:56.293171] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.213:1004
>
>>> [2015-06-02 10:23:56.900676] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.210:1018
>
>>> [2015-06-02 10:23:56.935113] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>
>>> [2015-06-02 10:23:56.936892] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.211:1016
>
>>> [2015-06-02 10:23:57.020995] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.212:1012
>
>>> [2015-06-02 10:23:57.659165] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
>>> [2015-06-02 10:23:58.332111] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
>>> [2015-06-02 10:23:58.929018] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
>>> [2015-06-02 10:23:59.100566] I [glusterd-handshake.c:1119:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-vers
>
>>> ion 2
>
>>> [2015-06-02 10:23:59.103665] I [glusterd-handler.c:2216:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe fro
>
>>> m uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>
>>> [2015-06-02 10:23:59.110710] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick
>
>>> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick resides
>
>>> [2015-06-02 10:23:59.110735] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct
>
>>> ripcorp.com:/export/sdb/brick for JQStore2
>
>>> [2015-06-02 10:23:59.131973] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick
>
>>> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick3 resides
>
>>> [2015-06-02 10:23:59.131994] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct
>
>>> ripcorp.com:/export/sdb/brick3 for Webresource
>
>>> [2015-06-02 10:23:59.152949] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick
>
>>> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick2 resides
>
>>> [2015-06-02 10:23:59.152969] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct
>
>>> ripcorp.com:/export/sdb/brick2 for ccim
>
>>> [2015-06-02 10:23:59.172948] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick
>
>>> glusterfs07.sh2.ctripcorp.com:/export/sdb/brick4 resides
>
>>> [2015-06-02 10:23:59.172968] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs07.sh2.ct
>
>>> ripcorp.com:/export/sdb/brick4 for cloudimage
>
>>> [2015-06-02 10:23:59.189033] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick
>
>>> glusterfs07.sh2.ctripcorp.com:/export/sdb/brick1 resides
>
>>> [2015-06-02 10:23:59.189053] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs07.sh2.ct
>
>>> ripcorp.com:/export/sdb/brick1 for filews
>
>>> [2015-06-02 10:23:59.203985] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick
>
>>> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick1 resides
>
>>> [2015-06-02 10:23:59.204005] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct
>
>>> ripcorp.com:/export/sdb/brick1 for filex
>
>>> [2015-06-02 10:23:59.212727] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick
>
>>> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick4 resides
>
>>> [2015-06-02 10:23:59.212747] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct
>
>>> ripcorp.com:/export/sdb/brick4 for pubmail
>
>>> [2015-06-02 10:23:59.237408] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>
>>> [2015-06-02 10:23:59.243681] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>
>>> [2015-06-02 10:23:59.243947] I [glusterd-handler.c:3334:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glusterfs05.s
>
>>> h2.ctripcorp.com (0), ret: 0
>
>>> [2015-06-02 10:23:59.245638] W [socket.c:620:__socket_rwv] 0-management: readv on /var/run/5dd5697669073fb549309773b211e62a.s
>
>>> ocket failed (Invalid argument)
>
>>> [2015-06-02 10:23:59.245679] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: nfs has
>
>>> disconnected from glusterd.
>
>>> [2015-06-02 10:23:59.245703] W [socket.c:620:__socket_rwv] 0-management: readv on /var/run/1578ba566e5a93f452fe8ad2e91b8682.s
>
>>> ocket failed (Invalid argument)
>
>>> [2015-06-02 10:23:59.245719] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: glusters
>
>>> hd has disconnected from glusterd.
>
>>> [2015-06-02 10:23:59.245748] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 04
>
>>> f22ee8-8e00-4c32-a924-b40a0e413aa6
>
>>> [2015-06-02 10:23:59.300666] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.213:1002
>
>>> [2015-06-02 10:23:59.569403] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
>>> [2015-06-02 10:23:59.570725] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
>>> [2015-06-02 10:23:59.571481] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
>>> [2015-06-02 10:23:59.572218] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
>>> [2015-06-02 10:23:59.572858] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
>>> [2015-06-02 10:23:59.573422] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
>>> [2015-06-02 10:23:59.574062] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
>>> [2015-06-02 10:23:59.574655] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
>>> [2015-06-02 10:23:59.908305] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.210:1017
>
>>> [2015-06-02 10:23:59.944630] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.211:1014
>
>>> [2015-06-02 10:24:00.028556] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.212:1011
>
>>> [2015-06-02 10:24:02.309238] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.213:1000
>
>>> [2015-06-02 10:24:02.915960] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.210:1016
>
>>> [2015-06-02 10:24:02.952056] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.211:1013
>
>>> [2015-06-02 10:24:03.040358] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.212:1010
>
>>> [2015-06-02 10:24:05.317047] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.213:996
>
>>> [2015-06-02 10:24:05.923400] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.210:1015
>
>>> [2015-06-02 10:24:05.959144] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.211:1012
>
>>> [2015-06-02 10:24:06.044976] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.212:1009
>
>>> [2015-06-02 10:24:08.324821] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.213:994
>
>>> [2015-06-02 10:24:08.932287] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.210:1014
>
>>> [2015-06-02 10:24:08.967433] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.211:1011
>
>>> [2015-06-02 10:24:09.052451] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.212:1006
>
>>> [2015-06-02 10:24:11.332558] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.213:993
>
>>> [2015-06-02 10:24:11.939689] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.210:1013
>
>>> [2015-06-02 10:24:11.975001] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.211:1010
>
>>> [2015-06-02 10:24:12.060625] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.212:1001
>
>>> [2015-06-02 10:24:14.339808] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.213:992
>
>>> [2015-06-02 10:24:14.947317] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.210:1012
>
>>> [2015-06-02 10:24:14.983732] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.211:1007
>
>>> [2015-06-02 10:24:15.068444] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>
>>> hake request from unknown peer 10.8.230.212:1000
>
>>> [root at SVR6994HW2285 glusterfs]#
>
>>>
>
>>>
>
>>>
>
>>>
>
>>> Best Regards
>
>>> Yuyang Yang
>
>>>
>
>>>
>
>>> -----邮件原件-----
>
>>> 发件人: Atin Mukherjee [mailto:amukherj at redhat.com<mailto:amukherj at redhat.com>]
>
>>> 发送时间: Tuesday, June 02, 2015 5:49 PM
>
>>> 收件人: vyyy杨雨阳; Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
>
>>> 主题: Re: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start
>
>>>
>
>>>
>
>>>
>
>>> On 06/02/2015 12:23 PM, vyyy杨雨阳 wrote:
>
>>>> Attached glusterd log files for 10 & 11
>
>>> [2015-06-02 06:33:42.668268] E
>
>>> [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management:
>
>>> Rejecting management handshake request from unknown peer 10.8.230.212:1002
>
>>>
>
>>> From the above log it looks like node 11 tried to handshake with 12, but
>
>>> how come 12 is part of the cluster? Could you run gluster peer status in
>
>>> 12 and share glusterd log? I am still under the same opinion which I had
>
>>> earlier - you are trying to add node which is already a member of
>
>>> another cluster.
>
>>>>
>
>>>> Best Regards
>
>>>> Yuyang Yang
>
>>>>
>
>>>> -----邮件原件-----
>
>>>> 发件人: Atin Mukherjee [mailto:amukherj at redhat.com<mailto:amukherj at redhat.com>]
>
>>>> 发送时间: Tuesday, June 02, 2015 2:42 PM
>
>>>> 收件人: vyyy杨雨阳; Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
>
>>>> 主题: Re: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start
>
>>>>
>
>>>>
>
>>>>
>
>>>> On 06/02/2015 12:04 PM, vyyy杨雨阳 wrote:
>
>>>>> Glusterfs05~glusterfs10 are clustered for 2 years, recently upgrade to 3.6.3
>
>>>>> Glusterfs11~glusterfs14 are new nodes need to join the cluster
>
>>>>>
>
>>>>> On glusterfs09:
>
>>>>>
>
>>>>> [root at SH02SVR5952 ~]# gluster peer status
>
>>>>> Number of Peers: 6
>
>>>>>
>
>>>>> Hostname: glusterfs06.sh2.ctripcorp.com
>
>>>>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>
>>>>> State: Peer in Cluster (Connected)
>
>>>>>
>
>>>>> Hostname: glusterfs07.sh2.ctripcorp.com
>
>>>>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>
>>>>> State: Peer in Cluster (Connected)
>
>>>>>
>
>>>>> Hostname: glusterfs08.sh2.ctripcorp.com
>
>>>>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>
>>>>> State: Peer in Cluster (Connected)
>
>>>>>
>
>>>>> Hostname: 10.8.230.209
>
>>>>> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>
>>>>> State: Peer in Cluster (Connected)
>
>>>>>
>
>>>>> Hostname: glusterfs10.sh2.ctripcorp.com
>
>>>>> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>
>>>>> State: Peer in Cluster (Disconnected)
>
>>>>>
>
>>>>> Hostname: glusterfs11.sh2.ctripcorp.com
>
>>>>> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
>
>>>>> State: Peer Rejected (Connected)
>
>>>>> [root at SH02SVR5952 ~]#
>
>>>> Can you attach glusterd log files for 10 & 11?
>
>>>>>
>
>>>>> [root at SH02SVR5952 ~]# gluster volume status
>
>>>>> Status of volume: JQStore2
>
>>>>> Gluster process Port Online Pid
>
>>>>> ------------------------------------------------------------------------------
>
>>>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 2782
>
>>>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 2744
>
>>>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 5307
>
>>>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 3986
>
>>>>> NFS Server on localhost 2049 Y 51697
>
>>>>> Self-heal Daemon on localhost N/A Y 51710
>
>>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894
>
>>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905
>
>>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185
>
>>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192
>
>>>>> NFS Server on 10.8.230.209 2049 Y 4091
>
>>>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104
>
>>>>>
>
>>>>> Task Status of Volume JQStore2
>
>>>>> ------------------------------------------------------------------------------
>
>>>>> There are no active volume tasks
>
>>>>>
>
>>>>> Status of volume: Webresource
>
>>>>> Gluster process Port Online Pid
>
>>>>> ------------------------------------------------------------------------------
>
>>>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 2787
>
>>>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 2753
>
>>>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 5313
>
>>>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 3992
>
>>>>> NFS Server on localhost 2049 Y 51697
>
>>>>> Self-heal Daemon on localhost N/A Y 51710
>
>>>>> NFS Server on 10.8.230.209 2049 Y 4091
>
>>>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104
>
>>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185
>
>>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192
>
>>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894
>
>>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905
>
>>>>>
>
>>>>> Task Status of Volume Webresource
>
>>>>> ------------------------------------------------------------------------------
>
>>>>> There are no active volume tasks
>
>>>>>
>
>>>>> Status of volume: ccim
>
>>>>> Gluster process Port Online Pid
>
>>>>> ------------------------------------------------------------------------------
>
>>>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 2793
>
>>>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 2745
>
>>>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 5320
>
>>>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 3999
>
>>>>> NFS Server on localhost 2049 Y 51697
>
>>>>> Self-heal Daemon on localhost N/A Y 51710
>
>>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185
>
>>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192
>
>>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894
>
>>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905
>
>>>>> NFS Server on 10.8.230.209 2049 Y 4091
>
>>>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104
>
>>>>>
>
>>>>> Task Status of Volume ccim
>
>>>>> ------------------------------------------------------------------------------
>
>>>>> There are no active volume tasks
>
>>>>>
>
>>>>> Status of volume: cloudimage
>
>>>>> Gluster process Port Online Pid
>
>>>>> ------------------------------------------------------------------------------
>
>>>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 5325
>
>>>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 4005
>
>>>>> NFS Server on localhost 2049 Y 51697
>
>>>>> Self-heal Daemon on localhost N/A Y 51710
>
>>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185
>
>>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192
>
>>>>> NFS Server on 10.8.230.209 2049 Y 4091
>
>>>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104
>
>>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894
>
>>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905
>
>>>>>
>
>>>>> Task Status of Volume cloudimage
>
>>>>> ------------------------------------------------------------------------------
>
>>>>> There are no active volume tasks
>
>>>>>
>
>>>>> Status of volume: filews
>
>>>>> Gluster process Port Online Pid
>
>>>>> ------------------------------------------------------------------------------
>
>>>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 5330
>
>>>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 4010
>
>>>>> NFS Server on localhost 2049 Y 51697
>
>>>>> Self-heal Daemon on localhost N/A Y 51710
>
>>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185
>
>>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192
>
>>>>> NFS Server on 10.8.230.209 2049 Y 4091
>
>>>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104
>
>>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894
>
>>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905
>
>>>>>
>
>>>>> Task Status of Volume filews
>
>>>>> ------------------------------------------------------------------------------
>
>>>>> There are no active volume tasks
>
>>>>>
>
>>>>> Status of volume: filex
>
>>>>> Gluster process Port Online Pid
>
>>>>> ------------------------------------------------------------------------------
>
>>>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 2794
>
>>>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 2755
>
>>>>> NFS Server on localhost 2049 Y 51697
>
>>>>> Self-heal Daemon on localhost N/A Y 51710
>
>>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894
>
>>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905
>
>>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185
>
>>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192
>
>>>>> NFS Server on 10.8.230.209 2049 Y 4091
>
>>>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104
>
>>>>>
>
>>>>> Task Status of Volume filex
>
>>>>> ------------------------------------------------------------------------------
>
>>>>> There are no active volume tasks
>
>>>>>
>
>>>>> Status of volume: pubmail
>
>>>>> Gluster process Port Online Pid
>
>>>>> ------------------------------------------------------------------------------
>
>>>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 2792
>
>>>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 2754
>
>>>>> NFS Server on localhost 2049 Y 51697
>
>>>>> Self-heal Daemon on localhost N/A Y 51710
>
>>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894
>
>>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905
>
>>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185
>
>>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192
>
>>>>> NFS Server on 10.8.230.209 2049 Y 4091
>
>>>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104
>
>>>>>
>
>>>>> Task Status of Volume pubmail
>
>>>>> ------------------------------------------------------------------------------
>
>>>>> There are no active volume tasks
>
>>>>>
>
>>>>> [root at SH02SVR5952 ~]#
>
>>>>>
>
>>>>>
>
>>>>>
>
>>>>> Best Regards
>
>>>>> 杨雨阳
>
>>>>> Site Operations Center
>
>>>>> Ext.15554 Seat.1#3F049
>
>>>>> We are the F.I.R.S.T.
>
>>>>> Fast, Innovation, Reliability , Secure, Team
>
>>>>>
>
>>>>>
>
>>>>>
>
>>>>> -----邮件原件-----
>
>>>>> 发件人: Atin Mukherjee [mailto:amukherj at redhat.com<mailto:amukherj at redhat.com>]
>
>>>>> 发送时间: Tuesday, June 02, 2015 2:20 PM
>
>>>>> 收件人: vyyy杨雨阳; Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
>
>>>>> 主题: Re: 答复: [Gluster-users] Gluster peer rejected and failed to start
>
>>>>>
>
>>>>>
>
>>>>>
>
>>>>> On 06/02/2015 11:33 AM, vyyy杨雨阳 wrote:
>
>>>>>> Actually I have 2 problems
>
>>>>>> 1、New nodes can't add to the clusters
>
>>>>>> I cleaned /var/lib/glusterd, now status is
>
>>>>>> State: Accepted peer request (Connected)
>
>>>>>>
>
>>>>>> 2、One of clusterd nodes shown 'Peer rejected' and glusterd failed to start.
>
>>>>>> The log is attached in pre-mail
>
>>>>>> This is a product cluster, this problem is more egent
>
>>>>> From the log I can clearly see the problematic node is
>
>>>>> glusterfs09.sh2.ctripcorp.com. Your existing cluster configuration has
>
>>>>> bricks hosted in glusterfs09.sh2.ctripcorp.com however the same is not
>
>>>>> part of the cluster. Could you paste the output of gluster peer status
>
>>>>> and gluster volume status ?
>
>>>>>>
>
>>>>>>
>
>>>>>>
>
>>>>>> Best Regards
>
>>>>>> Yuyang Yang
>
>>>>>>
>
>>>>>> -----邮件原件-----
>
>>>>>> 发件人: Atin Mukherjee [mailto:amukherj at redhat.com<mailto:amukherj at redhat.com>]
>
>>>>>> 发送时间: Tuesday, June 02, 2015 12:52 PM
>
>>>>>> 收件人: vyyy杨雨阳; Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
>
>>>>>> 主题: Re: [Gluster-users] Gluster peer rejected and failed to start
>
>>>>>>
>
>>>>>>
>
>>>>>>
>
>>>>>> On 06/02/2015 10:00 AM, vyyy杨雨阳 wrote:
>
>>>>>>> Hi
>
>>>>>>>
>
>>>>>>> We have a gluster (Version 3.6.3) cluster with 6 nodes, I tried to add 4 more nodes, but ‘Peer Rejected’, then I tried to resolve it by dump /var/lib/glusterd and probe again, not success, this is a question, But strange thing is:
>
>>>>>>>
>
>>>>>>> A node already in cluster also shown “Peer Reject”
>
>>>>>>>
>
>>>>>>> I tried to restart glusterd, It failed
>
>>>>>>>
>
>>>>>>> I found that /var/lib/glusterd/peers is empty, I copied the files from other nodes, still can’t start glusterd
>
>>>>>> It seems like you are trying to peer probe nodes which are either either
>
>>>>>> part of some other clusters (uncleaned nodes). Could you check whether
>
>>>>>> the nodes which you are adding have empty /var/lib/glusterd? If not
>
>>>>>> clean them and retry.
>
>>>>>>
>
>>>>>> ~Atin
>
>>>>>>>
>
>>>>>>>
>
>>>>>>>
>
>>>>>>> etc-glusterfs-glusterd.vol.log shown that cluster member as “unknown peer ”
>
>>>>>>>
>
>>>>>>> [2015-06-02 01:52:14.650635] C [glusterd-handler.c:2369:__glusterd_handle_friend_update] 0-: Received friend update request from unknown peer 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>
>>>>>>> [2015-06-02 01:52:14.650786] C [glusterd-handler.c:2369:__glusterd_handle_friend_update] 0-: Received friend update request from unknown peer 674a78b5-0590-48d4-8752-d4608832ed1d
>
>>>>>>> [2015-06-02 01:52:14.657881] C [glusterd-handler.c:2369:__glusterd_handle_friend_update] 0-: Received friend update request from unknown peer 83e1a9db-3134-45e4-acd2-387b12b5b207
>
>>>>>>> [2015-06-02 01:52:17.747865] W [glusterd-handler.c:697:__glusterd_handle_cluster_lock] 0-management: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 doesn't belong to the cluster. Ignoring request.
>
>>>>>>> [2015-06-02 01:52:17.747908] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
>
>>>>>>> [2015-06-02 01:52:40.338885] W [glusterd-handler.c:697:__glusterd_handle_cluster_lock] 0-management: 674a78b5-0590-48d4-8752-d4608832ed1d doesn't belong to the cluster. Ignoring request.
>
>>>>>>> [2015-06-02 01:52:40.338929] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
>
>>>>>>> [2015-06-02 01:52:41.310451] W [glusterd-handler.c:697:__glusterd_handle_cluster_lock] 0-management: 674a78b5-0590-48d4-8752-d4608832ed1d doesn't belong to the cluster. Ignoring request.
>
>>>>>>> [2015-06-02 01:52:41.310486] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
>
>>>>>>>
>
>>>>>>>
>
>>>>>>>
>
>>>>>>> Debug info is as following,
>
>>>>>>>
>
>>>>>>>
>
>>>>>>>
>
>>>>>>> /usr/sbin/glusterd
>
>>>>>>> [root at SH02SVR5951 peers]# /usr/sbin/glusterd --debug
>
>>>>>>> [2015-06-02 04:09:24.626690] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.6.3 (args: /usr/sbin/glusterd --debug)
>
>>>>>>> [2015-06-02 04:09:24.626739] D [logging.c:1763:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5
>
>>>>>>> [2015-06-02 04:09:24.627052] D [MSGID: 0] [glusterfsd.c:613:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol
>
>>>>>>> [2015-06-02 04:09:24.629683] I [glusterd.c:1214:init] 0-management: Maximum allowed open file descriptors set to 65536
>
>>>>>>> [2015-06-02 04:09:24.629706] I [glusterd.c:1259:init] 0-management: Using /var/lib/glusterd as working directory
>
>>>>>>> [2015-06-02 04:09:24.629764] D [glusterd.c:391:glusterd_rpcsvc_options_build] 0-: listen-backlog value: 128
>
>>>>>>> [2015-06-02 04:09:24.629895] D [rpcsvc.c:2198:rpcsvc_init] 0-rpc-service: RPC service inited.
>
>>>>>>> [2015-06-02 04:09:24.629904] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0
>
>>>>>>> [2015-06-02 04:09:24.629930] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>
>>>>>>> [2015-06-02 04:09:24.631989] D [socket.c:3807:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled
>
>>>>>>> [2015-06-02 04:09:24.632005] D [socket.c:3810:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled
>
>>>>>>> [2015-06-02 04:09:24.632013] D [socket.c:3827:socket_init] 0-socket.management: using system polling thread
>
>>>>>>> [2015-06-02 04:09:24.632024] D [name.c:550:server_fill_address_family] 0-socket.management: option address-family not specified, defaulting to inet
>
>>>>>>> [2015-06-02 04:09:24.632072] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/rdma.so
>
>>>>>>> [2015-06-02 04:09:24.632102] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
>
>>>>>>> [2015-06-02 04:09:24.632112] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
>
>>>>>>> [2015-06-02 04:09:24.632122] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
>
>>>>>>> [2015-06-02 04:09:24.632132] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, Ver: 2, Port: 0
>
>>>>>>> [2015-06-02 04:09:24.632140] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: 1238463, Ver: 2, Port: 0
>
>>>>>>> [2015-06-02 04:09:24.632148] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, Ver: 2, Port: 0
>
>>>>>>> [2015-06-02 04:09:24.632155] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433, Ver: 3, Port: 0
>
>>>>>>> [2015-06-02 04:09:24.632163] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: 1, Port: 0
>
>>>>>>> [2015-06-02 04:09:24.632170] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0
>
>>>>>>> [2015-06-02 04:09:24.632177] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 1239873, Ver: 1, Port: 0
>
>>>>>>> [2015-06-02 04:09:24.632212] D [rpcsvc.c:2198:rpcsvc_init] 0-rpc-service: RPC service inited.
>
>>>>>>> [2015-06-02 04:09:24.632220] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0
>
>>>>>>> [2015-06-02 04:09:24.632236] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>
>>>>>>> [2015-06-02 04:09:24.632302] D [socket.c:3692:socket_init] 0-socket.management: disabling nodelay
>
>>>>>>> [2015-06-02 04:09:24.632314] D [socket.c:3807:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled
>
>>>>>>> [2015-06-02 04:09:24.632321] D [socket.c:3810:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled
>
>>>>>>> [2015-06-02 04:09:24.632329] D [socket.c:3827:socket_init] 0-socket.management: using system polling thread
>
>>>>>>> [2015-06-02 04:09:24.632402] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0
>
>>>>>>> [2015-06-02 04:09:24.632414] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), Num: 14398633, Ver: 2, Port: 0
>
>>>>>>> [2015-06-02 04:09:24.632443] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>
>>>>>>> [2015-06-02 04:09:24.632455] D [glusterd.c:1484:init] 0-management: cannot get run-with-valgrind value
>
>>>>>>> [2015-06-02 04:09:24.633086] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system
>
>>>>>>> [2015-06-02 04:09:24.633518] D [glusterd.c:446:glusterd_check_gsync_present] 0-glusterd: Returning -1
>
>>>>>>> [2015-06-02 04:09:24.634031] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.634057] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.634115] D [store.c:304:gf_store_retrieve_value] 0-: key operating-version found
>
>>>>>>> [2015-06-02 04:09:24.634141] I [glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2
>
>>>>>>> [2015-06-02 04:09:24.634196] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.634436] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.634446] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.634469] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634488] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634496] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2
>
>>>>>>> [2015-06-02 04:09:24.634510] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634517] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 6
>
>>>>>>> [2015-06-02 04:09:24.634528] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634534] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1
>
>>>>>>> [2015-06-02 04:09:24.634544] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634551] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2
>
>>>>>>> [2015-06-02 04:09:24.634562] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634569] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1
>
>>>>>>> [2015-06-02 04:09:24.634580] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634587] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2
>
>>>>>>> [2015-06-02 04:09:24.634597] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634603] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 24
>
>>>>>>> [2015-06-02 04:09:24.634613] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634620] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0
>
>>>>>>> [2015-06-02 04:09:24.634631] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634638] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = 7bf7d81a-f835-4c3b-9c58-4d4ad7ec6e07
>
>>>>>>> [2015-06-02 04:09:24.634652] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634659] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = c7374917-2344-4c42-aeae-1d0bdb7ece0a
>
>>>>>>> [2015-06-02 04:09:24.634678] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634689] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = 703c3b9d-4343-4546-9ca3-350aecdd52fe
>
>>>>>>> [2015-06-02 04:09:24.634700] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634707] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2
>
>>>>>>> [2015-06-02 04:09:24.634719] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634726] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2
>
>>>>>>> [2015-06-02 04:09:24.634736] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634742] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.latency-measurement value = on
>
>>>>>>> [2015-06-02 04:09:24.634773] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.latency-measurement,value:on
>
>>>>>>> [2015-06-02 04:09:24.634785] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634791] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.count-fop-hits value = on
>
>>>>>>> [2015-06-02 04:09:24.634803] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.count-fop-hits,value:on
>
>>>>>>> [2015-06-02 04:09:24.634814] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634821] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.*
>
>>>>>>> [2015-06-02 04:09:24.634838] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.*
>
>>>>>>> [2015-06-02 04:09:24.634849] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634856] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs05.sh2.ctripcorp.com:-export-sdb-brick3
>
>>>>>>> [2015-06-02 04:09:24.634868] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634874] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs06.sh2.ctripcorp.com:-export-sdb-brick3
>
>>>>>>> [2015-06-02 04:09:24.634885] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634891] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick3
>
>>>>>>> [2015-06-02 04:09:24.634902] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634908] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick3
>
>>>>>>> [2015-06-02 04:09:24.634919] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634925] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-4 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick3
>
>>>>>>> [2015-06-02 04:09:24.634935] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.634942] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-5 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick3
>
>>>>>>> [2015-06-02 04:09:24.634954] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.634984] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635002] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.635019] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635030] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635039] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635048] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635056] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635066] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635074] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635083] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635092] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635101] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635110] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635118] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635127] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635136] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635144] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635153] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635161] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635368] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.635380] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.635396] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635416] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635426] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635436] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635447] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635456] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635465] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.635488] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.635501] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635748] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.635766] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.635786] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635807] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635818] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635828] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635838] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635847] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.635856] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.635909] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.635922] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636158] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.636167] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.636180] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636202] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636213] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636222] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636232] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636241] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636250] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.636274] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.636286] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636471] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.636479] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.636493] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636514] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636524] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636533] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636542] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636552] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636561] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.636586] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.636598] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636856] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.636889] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.636922] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636943] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636954] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636964] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636975] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636984] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.636994] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.637020] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.637036] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.637307] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.637325] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.637346] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.637367] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.637378] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.637388] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.819347] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.819374] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.819385] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.819413] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.819707] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.820187] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.820222] D [store.c:304:gf_store_retrieve_value] 0-: key version found
>
>>>>>>> [2015-06-02 04:09:24.820239] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.820532] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.820585] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.820959] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.821064] D [store.c:372:gf_store_save_value] 0-: returning: 0
>
>>>>>>> [2015-06-02 04:09:24.821090] D [store.c:372:gf_store_save_value] 0-: returning: 0
>
>>>>>>> [2015-06-02 04:09:24.837290] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.837540] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.837551] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.837568] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.837589] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.837600] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.837862] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.837880] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.837896] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.837918] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.837929] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.837939] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.837964] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.838216] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.838257] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.838273] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838295] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838303] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2
>
>>>>>>> [2015-06-02 04:09:24.838314] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838320] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 6
>
>>>>>>> [2015-06-02 04:09:24.838330] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838336] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1
>
>>>>>>> [2015-06-02 04:09:24.838346] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838352] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2
>
>>>>>>> [2015-06-02 04:09:24.838361] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838367] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1
>
>>>>>>> [2015-06-02 04:09:24.838377] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838383] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2
>
>>>>>>> [2015-06-02 04:09:24.838393] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838399] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 6
>
>>>>>>> [2015-06-02 04:09:24.838408] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838414] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0
>
>>>>>>> [2015-06-02 04:09:24.838423] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838430] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = cdc279f4-6cd0-4087-8d2a-440bccb51e33
>
>>>>>>> [2015-06-02 04:09:24.838443] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838449] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = 0b6d3496-0865-45e4-a45b-c221b22d3d92
>
>>>>>>> [2015-06-02 04:09:24.838459] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838465] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = ecedfb4d-3308-4f6c-b188-4d5b868958e1
>
>>>>>>> [2015-06-02 04:09:24.838475] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838482] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2
>
>>>>>>> [2015-06-02 04:09:24.838493] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838499] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2
>
>>>>>>> [2015-06-02 04:09:24.838508] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838515] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.68.*
>
>>>>>>> [2015-06-02 04:09:24.838533] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.68.*
>
>>>>>>> [2015-06-02 04:09:24.838545] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838552] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs05.sh2.ctripcorp.com:-export-sdb-brick2
>
>>>>>>> [2015-06-02 04:09:24.838563] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838570] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs06.sh2.ctripcorp.com:-export-sdb-brick2
>
>>>>>>> [2015-06-02 04:09:24.838580] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838587] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick2
>
>>>>>>> [2015-06-02 04:09:24.838597] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838604] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick2
>
>>>>>>> [2015-06-02 04:09:24.838614] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838621] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-4 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick2
>
>>>>>>> [2015-06-02 04:09:24.838631] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838637] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-5 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick2
>
>>>>>>> [2015-06-02 04:09:24.838649] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.838697] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838719] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.838738] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838747] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838756] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838764] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838772] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838781] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838797] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838806] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838815] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838823] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838832] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838840] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838848] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.838857] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839051] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.839063] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.839084] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839102] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839113] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839123] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839133] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839142] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839151] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.839177] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.839189] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839382] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.839390] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.839403] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839421] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839431] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839440] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839450] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839459] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839468] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.839491] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.839502] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839700] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.839712] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.839732] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839750] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839760] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839770] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839780] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.839799] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.839822] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.839834] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840057] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.840075] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.840096] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840117] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840128] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840137] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840147] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840156] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840166] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.840192] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.840206] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840402] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.840410] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.840424] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840440] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840450] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840459] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840469] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840478] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840487] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.840513] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.840525] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840799] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.840818] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.840839] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840859] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840871] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840880] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840891] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840900] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.840910] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.840931] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.841118] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.841359] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.841396] D [store.c:304:gf_store_retrieve_value] 0-: key version found
>
>>>>>>> [2015-06-02 04:09:24.841413] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.841557] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.841611] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.841799] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.841875] D [store.c:372:gf_store_save_value] 0-: returning: 0
>
>>>>>>> [2015-06-02 04:09:24.841901] D [store.c:372:gf_store_save_value] 0-: returning: 0
>
>>>>>>> [2015-06-02 04:09:24.842350] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.842469] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.842477] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.842489] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.842514] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.842525] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.842636] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.842645] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.842656] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.842696] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.842708] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.842718] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.842738] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.842848] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.842859] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.842872] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.842891] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.842898] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2
>
>>>>>>> [2015-06-02 04:09:24.842908] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.842914] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 4
>
>>>>>>> [2015-06-02 04:09:24.842924] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.842930] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1
>
>>>>>>> [2015-06-02 04:09:24.842939] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.842945] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2
>
>>>>>>> [2015-06-02 04:09:24.842954] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.842960] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1
>
>>>>>>> [2015-06-02 04:09:24.842969] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.842975] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2
>
>>>>>>> [2015-06-02 04:09:24.842984] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.842990] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 3
>
>>>>>>> [2015-06-02 04:09:24.842999] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843005] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0
>
>>>>>>> [2015-06-02 04:09:24.843015] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843021] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = a3369f63-9fce-4a56-a921-f7206cb8d3ae
>
>>>>>>> [2015-06-02 04:09:24.843033] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843039] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = c6012954-928a-43c1-b7ea-336d6ee0bea9
>
>>>>>>> [2015-06-02 04:09:24.843050] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843056] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = bfbd7007-0e0c-45f9-8d59-05e1aee66728
>
>>>>>>> [2015-06-02 04:09:24.843066] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843072] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2
>
>>>>>>> [2015-06-02 04:09:24.843082] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843089] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2
>
>>>>>>> [2015-06-02 04:09:24.843098] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843104] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.83.*
>
>>>>>>> [2015-06-02 04:09:24.843119] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.83.*
>
>>>>>>> [2015-06-02 04:09:24.843130] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843136] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick4
>
>>>>>>> [2015-06-02 04:09:24.843147] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843154] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick4
>
>>>>>>> [2015-06-02 04:09:24.843164] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843171] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick4
>
>>>>>>> [2015-06-02 04:09:24.843181] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843188] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick4
>
>>>>>>> [2015-06-02 04:09:24.843198] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.843221] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843238] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.843255] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843264] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843273] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843281] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843290] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843298] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843306] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843314] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843323] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843332] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843340] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843348] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843357] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843365] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843374] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843476] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.843484] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.843499] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843519] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843720] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843739] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843751] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843761] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843771] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.843802] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.843817] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.843984] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.843996] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.844013] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844030] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844041] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844050] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844060] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844069] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844078] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.844104] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.844116] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844222] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.844230] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.844243] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844266] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844276] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844285] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844294] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844303] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844312] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.844334] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.844346] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844443] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.844451] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.844468] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844487] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844497] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844506] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844516] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844525] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844534] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.844550] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.844753] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.845000] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.845033] D [store.c:304:gf_store_retrieve_value] 0-: key version found
>
>>>>>>> [2015-06-02 04:09:24.845049] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.845199] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.845244] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.845396] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.845448] D [store.c:372:gf_store_save_value] 0-: returning: 0
>
>>>>>>> [2015-06-02 04:09:24.845471] D [store.c:372:gf_store_save_value] 0-: returning: 0
>
>>>>>>> [2015-06-02 04:09:24.846053] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.846195] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.846206] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.846220] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846241] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846252] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.846362] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.846371] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.846382] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846404] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846414] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846424] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.846442] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.846543] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.846552] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.846562] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846585] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846592] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2
>
>>>>>>> [2015-06-02 04:09:24.846603] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846609] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 6
>
>>>>>>> [2015-06-02 04:09:24.846618] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846624] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1
>
>>>>>>> [2015-06-02 04:09:24.846633] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846639] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2
>
>>>>>>> [2015-06-02 04:09:24.846648] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846654] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1
>
>>>>>>> [2015-06-02 04:09:24.846663] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846690] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2
>
>>>>>>> [2015-06-02 04:09:24.846702] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846708] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 3
>
>>>>>>> [2015-06-02 04:09:24.846717] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846723] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0
>
>>>>>>> [2015-06-02 04:09:24.846733] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846739] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = 8869b21b-1bf1-4eba-a52a-1158abde7d23
>
>>>>>>> [2015-06-02 04:09:24.846752] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846758] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = 27298b95-919d-43f7-9507-8cbfbe5292e3
>
>>>>>>> [2015-06-02 04:09:24.846769] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846775] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = 47f52166-020b-4bef-ada1-7e65fcd2834f
>
>>>>>>> [2015-06-02 04:09:24.846785] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846792] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2
>
>>>>>>> [2015-06-02 04:09:24.846802] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846808] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2
>
>>>>>>> [2015-06-02 04:09:24.846818] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846824] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.count-fop-hits value = on
>
>>>>>>> [2015-06-02 04:09:24.846837] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.count-fop-hits,value:on
>
>>>>>>> [2015-06-02 04:09:24.846848] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846854] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.latency-measurement value = on
>
>>>>>>> [2015-06-02 04:09:24.846864] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.latency-measurement,value:on
>
>>>>>>> [2015-06-02 04:09:24.846875] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846881] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs05.sh2.ctripcorp.com:-export-sdb-brick
>
>>>>>>> [2015-06-02 04:09:24.846892] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846899] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs06.sh2.ctripcorp.com:-export-sdb-brick
>
>>>>>>> [2015-06-02 04:09:24.846910] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846916] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick
>
>>>>>>> [2015-06-02 04:09:24.846927] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846934] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick
>
>>>>>>> [2015-06-02 04:09:24.846944] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846952] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-4 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick
>
>>>>>>> [2015-06-02 04:09:24.846962] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.846969] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-5 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick
>
>>>>>>> [2015-06-02 04:09:24.846981] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.847003] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847027] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.847045] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847055] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847064] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847072] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847081] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847089] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847098] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847106] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847115] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847124] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847133] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847141] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847150] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847159] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847167] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847176] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847286] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.847294] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.847308] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847328] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847339] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847348] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847358] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847367] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847376] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.847402] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.847414] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847516] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.847523] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.847537] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847559] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847569] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847579] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847588] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847597] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847607] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.847628] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.847640] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847747] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.847756] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.847771] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847794] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847804] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847814] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847823] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847832] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847841] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.847862] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.847873] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.847969] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.847977] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.847990] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848012] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848022] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848031] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848040] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848049] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848058] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.848080] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.848091] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848187] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.848195] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.848208] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848230] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848240] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848249] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848259] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848267] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848277] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.848299] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.848310] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848411] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.848419] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.848432] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848455] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848465] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848474] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848484] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848493] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848503] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.848519] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848727] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.848975] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.849002] D [store.c:304:gf_store_retrieve_value] 0-: key version found
>
>>>>>>> [2015-06-02 04:09:24.849019] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.849167] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.849212] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.849367] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.849419] D [store.c:372:gf_store_save_value] 0-: returning: 0
>
>>>>>>> [2015-06-02 04:09:24.849442] D [store.c:372:gf_store_save_value] 0-: returning: 0
>
>>>>>>> [2015-06-02 04:09:24.849880] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.850003] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.850012] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.850024] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850043] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850054] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.850157] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.850166] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.850177] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850192] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850202] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850211] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.850228] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.850402] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.850414] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.850426] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850444] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850451] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2
>
>>>>>>> [2015-06-02 04:09:24.850461] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850467] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 4
>
>>>>>>> [2015-06-02 04:09:24.850476] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850482] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1
>
>>>>>>> [2015-06-02 04:09:24.850491] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850497] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2
>
>>>>>>> [2015-06-02 04:09:24.850506] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850512] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1
>
>>>>>>> [2015-06-02 04:09:24.850522] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850528] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2
>
>>>>>>> [2015-06-02 04:09:24.850537] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850543] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = disperse_count value = 0
>
>>>>>>> [2015-06-02 04:09:24.850552] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850558] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = redundancy_count value = 0
>
>>>>>>> [2015-06-02 04:09:24.850567] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850573] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 11
>
>>>>>>> [2015-06-02 04:09:24.850582] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850588] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0
>
>>>>>>> [2015-06-02 04:09:24.850598] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850604] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = ea46609c-be9a-4283-8357-8fdc2e5a190e
>
>>>>>>> [2015-06-02 04:09:24.850616] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850622] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = 21b247ce-58ce-4087-8c32-3d578b4cc22d
>
>>>>>>> [2015-06-02 04:09:24.850632] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850638] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = 0724171a-c7ce-4528-b54b-2d983f969537
>
>>>>>>> [2015-06-02 04:09:24.850648] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850654] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2
>
>>>>>>> [2015-06-02 04:09:24.850664] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850692] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2
>
>>>>>>> [2015-06-02 04:09:24.850703] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850709] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.92.*,10.8.230.*,192.168.98.*
>
>>>>>>> [2015-06-02 04:09:24.850724] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.92.*,10.8.230.*,192.168.98.*
>
>>>>>>> [2015-06-02 04:09:24.850736] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850743] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick1
>
>>>>>>> [2015-06-02 04:09:24.850754] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850760] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick1
>
>>>>>>> [2015-06-02 04:09:24.850772] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850778] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick1
>
>>>>>>> [2015-06-02 04:09:24.850789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850795] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick1
>
>>>>>>> [2015-06-02 04:09:24.850807] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.850830] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850850] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.850867] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850877] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850885] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850894] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850902] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850911] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850919] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850928] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850936] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850945] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850953] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850962] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850971] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850980] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850988] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.850996] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851005] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851164] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.851173] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.851187] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851203] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851213] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851223] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851232] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851242] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851251] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851261] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.851283] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.851295] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851457] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.851465] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.851478] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851493] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851504] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851513] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851522] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851531] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851540] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851549] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.851571] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.851583] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851732] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.851741] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.851755] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851770] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851780] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851799] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851807] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851816] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851826] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.851850] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.851863] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.851959] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.851974] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.852005] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.852026] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.852037] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.852047] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.852061] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.852070] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.852080] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.852090] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.852111] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.852298] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.852524] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.852556] D [store.c:304:gf_store_retrieve_value] 0-: key version found
>
>>>>>>> [2015-06-02 04:09:24.852576] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.852723] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.852776] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.852930] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.852982] D [store.c:372:gf_store_save_value] 0-: returning: 0
>
>>>>>>> [2015-06-02 04:09:24.853006] D [store.c:372:gf_store_save_value] 0-: returning: 0
>
>>>>>>> [2015-06-02 04:09:24.853412] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.853524] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.853539] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.853553] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.853573] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.853583] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.853699] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.853718] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.853735] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.853757] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.853768] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.853778] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.853787] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.853798] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.853820] D [glusterd-store.c:3094:glusterd_store_retrieve_volumes] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.853962] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.853972] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.853983] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854000] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854011] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>
>>>>>>> [2015-06-02 04:09:24.854023] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854032] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854043] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.854054] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.854070] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
>
>>>>>>> [2015-06-02 04:09:24.854162] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.854171] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.854181] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854197] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854205] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>
>>>>>>> [2015-06-02 04:09:24.854216] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854225] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854232] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.854241] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.854255] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
>
>>>>>>> [2015-06-02 04:09:24.854345] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.854354] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.854364] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854380] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854390] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>
>>>>>>> [2015-06-02 04:09:24.854402] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854411] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854417] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.854427] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.854439] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
>
>>>>>>> [2015-06-02 04:09:24.854527] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.854536] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.854546] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854561] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854569] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>
>>>>>>> [2015-06-02 04:09:24.854580] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854589] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854596] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.854605] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.854617] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
>
>>>>>>> [2015-06-02 04:09:24.854704] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.854713] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.854725] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854740] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854749] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>
>>>>>>> [2015-06-02 04:09:24.854760] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854769] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.854776] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.854789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.854803] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
>
>>>>>>> [2015-06-02 04:09:24.855064] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.855072] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.855083] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.855098] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.855106] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>
>>>>>>> [2015-06-02 04:09:24.855116] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.855125] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.855132] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.855141] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>
>>>>>>> [2015-06-02 04:09:24.855154] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
>
>>>>>>> [2015-06-02 04:09:24.855186] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.855210] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>
>>>>>>> [2015-06-02 04:09:24.855218] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30
>
>>>>>>> [2015-06-02 04:09:24.855231] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>
>>>>>>> [2015-06-02 04:09:24.855324] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled
>
>>>>>>> [2015-06-02 04:09:24.855333] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled
>
>>>>>>> [2015-06-02 04:09:24.855340] D [socket.c:3827:socket_init] 0-management: using system polling thread
>
>>>>>>> [2015-06-02 04:09:24.859610] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.211 (port-24007) for hostname: glusterfs07.sh2.ctripcorp.com and port: 24007
>
>>>>>>> [2015-06-02 04:09:24.859628] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection
>
>>>>>>> [2015-06-02 04:09:24.859707] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0
>
>>>>>>> [2015-06-02 04:09:24.859733] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.859772] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>
>>>>>>> [2015-06-02 04:09:24.859781] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30
>
>>>>>>> [2015-06-02 04:09:24.859794] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>
>>>>>>> [2015-06-02 04:09:24.859905] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled
>
>>>>>>> [2015-06-02 04:09:24.859916] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled
>
>>>>>>> [2015-06-02 04:09:24.859923] D [socket.c:3827:socket_init] 0-management: using system polling thread
>
>>>>>>> [2015-06-02 04:09:24.863505] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.231.160 (port-24007) for hostname: glusterfs11.sh2.ctripcorp.com and port: 24007
>
>>>>>>> [2015-06-02 04:09:24.863521] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection
>
>>>>>>> [2015-06-02 04:09:24.863562] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0
>
>>>>>>> [2015-06-02 04:09:24.863585] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.863607] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>
>>>>>>> [2015-06-02 04:09:24.863615] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30
>
>>>>>>> [2015-06-02 04:09:24.863625] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>
>>>>>>> [2015-06-02 04:09:24.863754] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled
>
>>>>>>> [2015-06-02 04:09:24.863765] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled
>
>>>>>>> [2015-06-02 04:09:24.863772] D [socket.c:3827:socket_init] 0-management: using system polling thread
>
>>>>>>> [2015-06-02 04:09:24.867255] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.210 (port-24007) for hostname: glusterfs06.sh2.ctripcorp.com and port: 24007
>
>>>>>>> [2015-06-02 04:09:24.867271] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection
>
>>>>>>> [2015-06-02 04:09:24.867314] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0
>
>>>>>>> [2015-06-02 04:09:24.867337] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.867358] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>
>>>>>>> [2015-06-02 04:09:24.867366] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30
>
>>>>>>> [2015-06-02 04:09:24.867377] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>
>>>>>>> [2015-06-02 04:09:24.867463] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled
>
>>>>>>> [2015-06-02 04:09:24.867476] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled
>
>>>>>>> [2015-06-02 04:09:24.867483] D [socket.c:3827:socket_init] 0-management: using system polling thread
>
>>>>>>> [2015-06-02 04:09:24.870920] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.214 (port-24007) for hostname: glusterfs10.sh2.ctripcorp.com and port: 24007
>
>>>>>>> [2015-06-02 04:09:24.870935] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection
>
>>>>>>> [2015-06-02 04:09:24.870997] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0
>
>>>>>>> [2015-06-02 04:09:24.871021] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.871042] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>
>>>>>>> [2015-06-02 04:09:24.871050] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30
>
>>>>>>> [2015-06-02 04:09:24.871061] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>
>>>>>>> [2015-06-02 04:09:24.871129] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled
>
>>>>>>> [2015-06-02 04:09:24.871138] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled
>
>>>>>>> [2015-06-02 04:09:24.871144] D [socket.c:3827:socket_init] 0-management: using system polling thread
>
>>>>>>> [2015-06-02 04:09:24.874102] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.209 (port-24007) for hostname: 10.8.230.209 and port: 24007
>
>>>>>>> [2015-06-02 04:09:24.874116] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection
>
>>>>>>> [2015-06-02 04:09:24.874159] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0
>
>>>>>>> [2015-06-02 04:09:24.874181] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.874202] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>
>>>>>>> [2015-06-02 04:09:24.874210] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30
>
>>>>>>> [2015-06-02 04:09:24.874219] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>
>>>>>>> [2015-06-02 04:09:24.874292] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled
>
>>>>>>> [2015-06-02 04:09:24.874301] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled
>
>>>>>>> [2015-06-02 04:09:24.874326] D [socket.c:3827:socket_init] 0-management: using system polling thread
>
>>>>>>> [2015-06-02 04:09:24.877759] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.212 (port-24007) for hostname: glusterfs08.sh2.ctripcorp.com and port: 24007
>
>>>>>>> [2015-06-02 04:09:24.877774] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection
>
>>>>>>> [2015-06-02 04:09:24.877818] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0
>
>>>>>>> [2015-06-02 04:09:24.877831] D [glusterd-store.c:4106:glusterd_store_retrieve_peers] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.877863] I [glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list.
>
>>>>>>> [2015-06-02 04:09:24.877873] D [glusterd-store.c:3612:glusterd_store_retrieve_snaps] 0-: Returning with 0
>
>>>>>>> [2015-06-02 04:09:24.881125] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0
>
>>>>>>> [2015-06-02 04:09:24.881140] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.881148] D [glusterd-peer-utils.c:600:gd_peerinfo_find_from_hostname] 0-management: Friend glusterfs06.sh2.ctripcorp.com found.. state: 3
>
>>>>>>> [2015-06-02 04:09:24.881156] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0
>
>>>>>>> [2015-06-02 04:09:24.881162] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.881169] D [glusterd-peer-utils.c:600:gd_peerinfo_find_from_hostname] 0-management: Friend glusterfs07.sh2.ctripcorp.com found.. state: 3
>
>>>>>>> [2015-06-02 04:09:24.881176] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0
>
>>>>>>> [2015-06-02 04:09:24.881182] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.881188] D [glusterd-peer-utils.c:600:gd_peerinfo_find_from_hostname] 0-management: Friend glusterfs08.sh2.ctripcorp.com found.. state: 3
>
>>>>>>> [2015-06-02 04:09:24.881195] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0
>
>>>>>>> [2015-06-02 04:09:24.881201] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0
>
>>>>>>> [2015-06-02 04:09:24.890739] D [glusterd-peer-utils.c:121:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: glusterfs09.sh2.ctripcorp.com
>
>>>>>>> [2015-06-02 04:09:24.891607] D [common-utils.c:2897:gf_is_local_addr] 0-management: 10.8.230.213
>
>>>>>>> [2015-06-02 04:09:24.891750] D [common-utils.c:2897:gf_is_local_addr] 0-management: 10.8.230.213
>
>>>>>>> [2015-06-02 04:09:24.891820] D [common-utils.c:2897:gf_is_local_addr] 0-management: 10.8.230.213
>
>>>>>>> [2015-06-02 04:09:24.891881] D [common-utils.c:2913:gf_is_local_addr] 0-management: glusterfs09.sh2.ctripcorp.com is not local
>
>>>>>>> [2015-06-02 04:09:24.891890] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning -1
>
>>>>>>> [2015-06-02 04:09:24.891898] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning -1
>
>>>>>>> [2015-06-02 04:09:24.891904] E [glusterd-store.c:4244:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore
>
>>>>>>> [2015-06-02 04:09:24.891911] D [glusterd-store.c:4321:glusterd_restore] 0-: Returning -1
>
>>>>>>> [2015-06-02 04:09:24.891923] E [xlator.c:425:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again
>
>>>>>>> [2015-06-02 04:09:24.891933] E [graph.c:322:glusterfs_graph_init] 0-management: initializing translator failed
>
>>>>>>> [2015-06-02 04:09:24.891941] E [graph.c:525:glusterfs_graph_activate] 0-graph: init failed
>
>>>>>>> [2015-06-02 04:09:24.892164] D [logging.c:1740:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size reduced. About to flush 2 extra log messages
>
>>>>>>> [2015-06-02 04:09:24.892178] D [logging.c:1743:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 2 extra log messages
>
>>>>>>> [2015-06-02 04:09:24.892325] W [glusterfsd.c:1194:cleanup_and_exit] (--> 0-: received signum (0), shutting down
>
>>>>>>> [2015-06-02 04:09:24.892345] D [glusterfsd-mgmt.c:2244:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given
>
>>>>>>> [root at SH02SVR5951 peers]#
>
>>>>>>>
>
>>>>>>>
>
>>>>>>>
>
>>>>>>> Best Regards
>
>>>>>>> Yuyang Yang
>
>>>>>>>
>
>>>>>>>
>
>>>>>>>
>
>>>>>>> _______________________________________________
>
>>>>>>> Gluster-users mailing list
>
>>>>>>> Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
>
>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>
>>>>>>>
>
>>>>>>
>
>>>>>
>
>>>>
>
>>>
>
>>
>
>
>
> --
>
> ~Atin
>
> _______________________________________________
>
> Gluster-users mailing list
>
> Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
>
> http://www.gluster.org/mailman/listinfo/gluster-users
>
--
~Atin
More information about the Gluster-users
mailing list