[Gluster-users] 回复： 答复: 答复: 答复: Gluster peer rejected and failed to start
huangql
huangql at ihep.ac.cn
Fri Jun 5 02:57:23 UTC 2015
Maybe you can check selinux on the server side. Make sure selinux is disabled.
原始邮件
发件人:vyyy杨雨阳yuyangyang at Ctrip.com
收件人:Atin Mukherjeeamukherj at redhat.com; Gluster-users at gluster.orgGluster-users@gluster.org
发送时间:2015年6月4日(周四) 21:24
主题:[Gluster-users] 答复: 答复: 答复: Gluster peer rejected and failed to start
Can anyone help me? New nodes still can't join the cluster. The new nodes can peer each other, just not exist cluster nodes. I can not find any network problem, there were no firewall, port 24007 is connected, there were all at the same subnet... Thanks [root at SVR6993HW2285 glusterfs]# more etc-glusterfs-glusterd.vol.log [2015-06-05 01:59:02.276107] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glust erd version 3.6.3 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid) [2015-06-05 01:59:02.318477] I [glusterd.c:1214:init] 0-management: Maximum allowed open file descriptors set to 65536 [2015-06-05 01:59:02.318524] I [glusterd.c:1259:init] 0-management: Using /var/lib/glusterd as working directory [2015-06-05 01:59:02.329603] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-trans port/rdma.so: cannot open shared object file: No such file or directory [2015-06-05 01:59:02.329625] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport- type 'rdma' is not valid or not found on this machine [2015-06-05 01:59:02.329637] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the tra nsport failed [2015-06-05 01:59:02.343346] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [2015-06-05 01:59:02.351861] I [glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2 [2015-06-05 01:59:02.363138] I [glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snap s list. Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option transport.socket.listen-backlog 128 7: option ping-timeout 30 8: option transport.socket.read-fail-log off 9: option transport.socket.keepalive-interval 2 10: option transport.socket.keepalive-time 10 11: option transport-type rdma 12: option working-directory /var/lib/glusterd 13: end-volume 14: +------------------------------------------------------------------------------+ [2015-06-05 02:00:43.025378] I [glusterd.c:146:glusterd_uuid_init] 0-management: retrieved UUID: 1b98af34-08f9-492f-b54e-7608 2588d63b [2015-06-05 02:00:43.025462] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:00:44.025069] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:00:44.599862] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:00:46.029823] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:00:46.701845] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:01:43.119612] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:01:44.165527] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:01:44.851903] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:01:46.190401] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:01:46.722394] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:01:58.719968] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:02:32.584999] I [glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glust erfs05.sh2.ctripcorp.com 24007 [2015-06-05 02:02:32.586281] I [glusterd-handler.c:3165:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: g lusterfs05.sh2.ctripcorp.com (24007) [2015-06-05 02:02:32.602192] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-05 02:02:32.606053] I [glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect returned 0 [2015-06-05 02:02:32.620958] E [glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk] 0-management: failed to get the 'versions' from peer (10.8.230.209:24007) [2015-06-05 02:02:32.621020] I [MSGID: 106004] [glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management: Peer 000000 00-0000-0000-0000-000000000000, in Establishing Connection state, has disconnected from glusterd. [2015-06-05 02:02:43.264141] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:02:44.312918] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:02:44.781434] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:02:46.176888] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:02:46.799916] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:03:42.720994] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:03:43.771987] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:03:44.659775] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:03:45.956807] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:03:46.688787] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:04:42.891131] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:04:43.957423] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:04:44.995638] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:04:46.062171] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:04:46.957327] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:05:42.966651] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:05:44.008498] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:05:45.049936] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:05:46.087007] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:05:47.029325] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:06:42.874089] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:06:43.910998] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:06:44.932802] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:06:46.053731] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:06:47.055859] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:07:43.099647] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:07:44.146454] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:07:45.218982] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:07:46.246303] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:07:47.106954] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:08:43.226090] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:08:44.292041] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-05 02:08:45.351830] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:08:45.688349] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-05 02:08:46.651768] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-05 02:09:42.748628] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed Best Regards -----邮件原件----- 发件人: vyyy杨雨阳 发送时间: Thursday, June 04, 2015 11:00 AM 收件人: 'Atin Mukherjee'; Gluster-users at gluster.org 主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start The problem 2 has resolved. From glusterd log file, I find Cksums error as following: [2015-06-04 01:25:03.137798] E [MSGID: 106010] [glusterd-utils.c:3297:glusterd_compare_friend_volume] 0-management: Version of Cksums JQStore2 differ. local cksum = 2617028277, remote cksum = 2210669223 on peer glusterfs09.sh2.ctripcorp.com The solution for this problem was to do the following, on the BAD node: /etc/init.d/glusterd stop rsync -havP --delete 172.26.177.115:/var/lib/glusterd/vols/ /var/lib/glusterd/vols/ /etc/init.d/glusterd start The problem 1 is still under inverstigate: New nodes can not join the cluster, log file shown: failed to get the 'versions' from peer I have checked the file /var/lib/glusterd/ glusterd.info, all nodes are " operating-version=2" [2015-06-04 02:13:49.199416] E [glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk] 0-management: failed to get the 'versions' from peer (10.8.230.211:24007) [2015-06-04 02:14:07.112577] I [glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glust erfs05.sh2.ctripcorp.com 24007 [2015-06-04 02:14:07.113778] I [glusterd-handler.c:3165:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: g lusterfs05.sh2.ctripcorp.com (24007) [2015-06-04 02:14:07.131023] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-04 02:14:07.134829] I [glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect returned 0 [ Best Regards 杨雨阳 Site Operations Center Ext.15554 Seat.1#3F049 We are the F.I.R.S.T. Fast, Innovation, Reliability , Secure, Team -----邮件原件----- 发件人: vyyy杨雨阳 发送时间: Wednesday, June 03, 2015 5:37 PM 收件人: 'Atin Mukherjee'; Gluster-users at gluster.org 主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start Hi Atin and everybody, I have forced peer detach the news nodes ,The peer status seems even strange, glusterfs05 and glusterfs10 are peer in cluster, Can you help me? Any advice is appreciated. Glusterfs05 [root at sh02svr5956 ~]# gluster peer status Number of Peers: 5 Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer Rejected (Connected) Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer Rejected (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer Rejected (Connected) Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer in Cluster (Connected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer Rejected (Connected) [root at sh02svr5956 ~]# Glusterfs06 [root at SH02SVR5955 ~]# gluster peer status Number of Peers: 5 Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer Rejected (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer in Cluster (Connected) Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer in Cluster (Connected) Hostname: 10.8.230.209 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer Rejected (Connected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer in Cluster (Connected) [root at SH02SVR5955 ~]# Glusterfs07 [root at SH02SVR5954 ~]# gluster peer status Number of Peers: 5 Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer Rejected (Connected) Hostname: 10.8.230.209 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer Rejected (Connected) Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer in Cluster (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer in Cluster (Connected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer in Cluster (Connected) [root at SH02SVR5954 ~]# Glusterfs08 [root at SH02SVR5953 ~]# gluster peer status Number of Peers: 5 Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer in Cluster (Connected) Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer Rejected (Connected) Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer in Cluster (Connected) Hostname: 10.8.230.209 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer Rejected (Connected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer in Cluster (Connected) [root at SH02SVR5953 ~]# [root at SH02SVR5952 ~]# gluster peer status Number of Peers: 5 Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer in Cluster (Connected) Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer in Cluster (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer in Cluster (Connected) Hostname: 10.8.230.209 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer Rejected (Connected) Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer Rejected (Connected) [root at SH02SVR5952 ~]# Glusterfs10 [root at SH02SVR5951 ~]# gluster peer status Number of Peers: 6 Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer Rejected (Connected) Hostname: glusterfs11.sh2.ctripcorp.com Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0 State: Peer Rejected (Connected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer Rejected (Connected) Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer Rejected (Connected) Hostname: 10.8.230.209 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer in Cluster (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer Rejected (Connected) [root at SH02SVR5951 ~]# Best Regards 杨雨阳 Site Operations Center Ext.15554 Seat.1#3F049 We are the F.I.R.S.T. Fast, Innovation, Reliability , Secure, Team -----邮件原件----- 发件人: vyyy杨雨阳 发送时间: Wednesday, June 03, 2015 2:55 PM 收件人: 'Atin Mukherjee' 主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start Can I delete the files in /var/lib/glusterd/peers which state is not =3 on all nodes, then restart glusterd? [root at sh02svr5956 peers]# more 0475e11a-3c43-4b76-99dd-1d2e814c5f15 uuid=0475e11a-3c43-4b76-99dd-1d2e814c5f15 state=6 hostname1=glusterfs12.sh2.ctripcorp.com Best Regards Yang -----邮件原件----- 发件人: vyyy杨雨阳 发送时间: Wednesday, June 03, 2015 2:39 PM 收件人: 'Atin Mukherjee' 主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start Since it's intranet, You can only control my desktop, and then access these node. Should I apply a webex room? Best Regards 杨雨阳 Site Operations Center Ext.15554 Seat.1#3F049 We are the F.I.R.S.T. Fast, Innovation, Reliability , Secure, Team -----邮件原件----- 发件人: Atin Mukherjee [mailto:amukherj at redhat.com] 发送时间: Wednesday, June 03, 2015 2:36 PM 收件人: vyyy杨雨阳 主题: Re: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start Is it possible to share your ip/user/passwd for few of these nodes? On 06/03/2015 11:50 AM, vyyy杨雨阳 wrote: Can you remote control my desktop (like webex)? The nodes host bricks. And on the new nodes, I did "clean /var/lib/glusterd and restart" several times, but have no effect. Is there any other work around? Best Regards Yang -----邮件原件----- 发件人: Atin Mukherjee [mailto:amukherj at redhat.com] 发送时间: Wednesday, June 03, 2015 12:33 PM 收件人: vyyy杨雨阳; Gluster-users at gluster.org 主题: Re: 答复: [Gluster-users] Gluster peer rejected and failed to start I can't comment on what went wrong till I can have visibility on the entire cluster. Does the node which fails to start host any brick? If not then work around could be to clean /var/lib/glusterd and restart glusterd, that should sync up data and bring this node in consistency. HTH, Atin On 06/03/2015 08:34 AM, vyyy杨雨阳 wrote: I found that /var/lib/glusterd/peers can't include node's own peer, I delete this file. Glusterd started finally. But the peer status still not normal. Seems that glusterfs10 leave the cluster and join the new nodes alone. Peer status on glusterfs10 [root at SH02SVR5951 ~]# gluster peer status Number of Peers: 7 Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer Rejected (Connected) Hostname: glusterfs11.sh2.ctripcorp.com Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0 State: Accepted peer request (Connected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer Rejected (Connected) Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer Rejected (Connected) Hostname: 10.8.230.209 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer Rejected (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer Rejected (Connected) Hostname: glusterfs12.sh2.ctripcorp.com Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15 State: Peer in Cluster (Connected) Peer status on glusterfs10 [root at SVR6993HW2285 ~]# gluster peer status Number of Peers: 6 Hostname: glusterfs05.sh2.ctripcorp.com Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer Rejected (Connected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer Rejected (Connected) Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer Rejected (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer Rejected (Connected) Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer Rejected (Connected) Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer in Cluster (Connected) Peer status on glusterfs05, this is the node that I tried to peer probe new nodes (glusterfs11 to glusterfs14) [root at sh02svr5956 ~]# gluster peer status Number of Peers: 9 Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer in Cluster (Connected) Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer in Cluster (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer in Cluster (Connected) Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer Rejected (Connected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer in Cluster (Connected) Hostname: glusterfs11.sh2.ctripcorp.com Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0 State: Peer Rejected (Connected) Hostname: glusterfs12.sh2.ctripcorp.com Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15 State: Peer in Cluster (Connected) Hostname: glusterfs13.sh2.ctripcorp.com Uuid: 43e96d31-6c6d-4096-832a-440e544554ca State: Peer Rejected (Connected) Hostname: glusterfs14.sh2.ctripcorp.com Uuid: 45c11af9-7850-4598-ad62-4bb343b62524 State: Peer Rejected (Connected) [root at sh02svr5956 ~]# Best Regards Yang -----邮件原件----- 发件人: vyyy杨雨阳 发送时间: Wednesday, June 03, 2015 10:04 AM 收件人: 'Atin Mukherjee'; 'Gluster-users at gluster.org' 主题: [Gluster-users] Gluster peer rejected and failed to start I am still messed, can you help me figure out problem 2 (A node of cluster failed to start glusterd) priority The glusterd log on the node gluster10 shown ' resolve brick failed in restore' I checked /var/lib/glusterd/vols, seems ok Glusterd log file Error message: [2015-06-03 01:31:32.564010] E [glusterd-store.c:4244:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore [2015-06-03 01:31:32.564036] E [xlator.c:425:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again Best Regards Yang -----邮件原件----- 发件人: vyyy杨雨阳 发送时间: Tuesday, June 02, 2015 6:27 PM 收件人: 'Atin Mukherjee'; Gluster-users at gluster.org 主题: 答复: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start Gluster11 to gluster14 are new nodes I tried to join exit cluster but failed On gluster12: [root at SVR6994HW2285 ~]# gluster peer status Number of Peers: 1 Hostname: glusterfs05.sh2.ctripcorp.com Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Accepted peer request (Connected) [root at SVR6994HW2285 ~]# On gluster05: (peer status seems strange on all those nodes): [root at sh02svr5956 ~]# gluster peer status Number of Peers: 9 Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer in Cluster (Connected) Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer in Cluster (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer in Cluster (Connected) Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer in Cluster (Disconnected) Hostname: glusterfs09.sh2.ctripcorp.com Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d State: Peer in Cluster (Connected) Hostname: glusterfs11.sh2.ctripcorp.com Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0 State: Peer Rejected (Connected) Hostname: glusterfs12.sh2.ctripcorp.com Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15 State: Peer in Cluster (Connected) Hostname: glusterfs13.sh2.ctripcorp.com Uuid: 43e96d31-6c6d-4096-832a-440e544554ca State: Peer Rejected (Connected) Hostname: glusterfs14.sh2.ctripcorp.com Uuid: 45c11af9-7850-4598-ad62-4bb343b62524 State: Peer Rejected (Connected) [root at sh02svr5956 ~]# The glusterd log file [root at SVR6994HW2285 glusterfs]# more etc-glusterfs-glusterd.vol.log [2015-06-02 10:23:48.368697] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glust erd version 3.6.3 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid) [2015-06-02 10:23:48.372587] I [glusterd.c:1214:init] 0-management: Maximum allowed open file descriptors set to 65536 [2015-06-02 10:23:48.372625] I [glusterd.c:1259:init] 0-management: Using /var/lib/glusterd as working directory [2015-06-02 10:23:48.374890] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-trans port/rdma.so: cannot open shared object file: No such file or directory [2015-06-02 10:23:48.374908] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport- type 'rdma' is not valid or not found on this machine [2015-06-02 10:23:48.374920] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the tra nsport failed [2015-06-02 10:23:48.376539] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [2015-06-02 10:23:48.377464] I [glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2 [2015-06-02 10:23:48.378122] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2015-06-02 10:23:48.378191] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-02 10:23:48.383072] I [glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snap s list. Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option transport.socket.listen-backlog 128 7: option ping-timeout 30 8: option transport.socket.read-fail-log off 9: option transport.socket.keepalive-interval 2 10: option transport.socket.keepalive-time 10 11: option transport-type rdma 12: option working-directory /var/lib/glusterd 13: end-volume 14: +------------------------------------------------------------------------------+ [2015-06-02 10:23:48.413755] I [glusterd.c:146:glusterd_uuid_init] 0-management: retrieved UUID: 0475e11a-3c43-4b76-99dd-1d2e 814c5f15 [2015-06-02 10:23:48.414345] I [glusterd-rpc-ops.c:245:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 04f 22ee8-8e00-4c32-a924-b40a0e413aa6, host: glusterfs05.sh2.ctripcorp.com [2015-06-02 10:23:48.414883] I [glusterd-rpc-ops.c:387:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2015-06-02 10:23:50.088645] I [glusterd-rpc-ops.c:437:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 04f22ee 8-8e00-4c32-a924-b40a0e413aa6, host: glusterfs05.sh2.ctripcorp.com, port: 0 [2015-06-02 10:23:50.090245] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 04 f22ee8-8e00-4c32-a924-b40a0e413aa6 [2015-06-02 10:23:50.107013] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.212:1016 [2015-06-02 10:23:50.107805] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.211:1018 [2015-06-02 10:23:50.108358] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.210:1020 [2015-06-02 10:23:50.139963] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.213:1008 [2015-06-02 10:23:53.286070] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.213:1006 [2015-06-02 10:23:53.893225] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.210:1019 [2015-06-02 10:23:53.928314] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.211:1017 [2015-06-02 10:23:54.013229] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.212:1014 [2015-06-02 10:23:54.273022] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-02 10:23:55.531073] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-02 10:23:56.293171] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.213:1004 [2015-06-02 10:23:56.900676] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.210:1018 [2015-06-02 10:23:56.935113] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed [2015-06-02 10:23:56.936892] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.211:1016 [2015-06-02 10:23:57.020995] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.212:1012 [2015-06-02 10:23:57.659165] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-02 10:23:58.332111] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-02 10:23:58.929018] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2015-06-02 10:23:59.100566] I [glusterd-handshake.c:1119:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-vers ion 2 [2015-06-02 10:23:59.103665] I [glusterd-handler.c:2216:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe fro m uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 [2015-06-02 10:23:59.110710] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick resides [2015-06-02 10:23:59.110735] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct ripcorp.com:/export/sdb/brick for JQStore2 [2015-06-02 10:23:59.131973] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick3 resides [2015-06-02 10:23:59.131994] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct ripcorp.com:/export/sdb/brick3 for Webresource [2015-06-02 10:23:59.152949] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick2 resides [2015-06-02 10:23:59.152969] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct ripcorp.com:/export/sdb/brick2 for ccim [2015-06-02 10:23:59.172948] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick4 resides [2015-06-02 10:23:59.172968] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs07.sh2.ct ripcorp.com:/export/sdb/brick4 for cloudimage [2015-06-02 10:23:59.189033] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick1 resides [2015-06-02 10:23:59.189053] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs07.sh2.ct ripcorp.com:/export/sdb/brick1 for filews [2015-06-02 10:23:59.203985] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick1 resides [2015-06-02 10:23:59.204005] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct ripcorp.com:/export/sdb/brick1 for filex [2015-06-02 10:23:59.212727] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick4 resides [2015-06-02 10:23:59.212747] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct ripcorp.com:/export/sdb/brick4 for pubmail [2015-06-02 10:23:59.237408] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-02 10:23:59.243681] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-02 10:23:59.243947] I [glusterd-handler.c:3334:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glusterfs05.s h2.ctripcorp.com (0), ret: 0 [2015-06-02 10:23:59.245638] W [socket.c:620:__socket_rwv] 0-management: readv on /var/run/5dd5697669073fb549309773b211e62a.s ocket failed (Invalid argument) [2015-06-02 10:23:59.245679] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: nfs has disconnected from glusterd. [2015-06-02 10:23:59.245703] W [socket.c:620:__socket_rwv] 0-management: readv on /var/run/1578ba566e5a93f452fe8ad2e91b8682.s ocket failed (Invalid argument) [2015-06-02 10:23:59.245719] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: glusters hd has disconnected from glusterd. [2015-06-02 10:23:59.245748] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 04 f22ee8-8e00-4c32-a924-b40a0e413aa6 [2015-06-02 10:23:59.300666] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.213:1002 [2015-06-02 10:23:59.569403] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-02 10:23:59.570725] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-02 10:23:59.571481] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-02 10:23:59.572218] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-02 10:23:59.572858] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-02 10:23:59.573422] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-02 10:23:59.574062] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-02 10:23:59.574655] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2015-06-02 10:23:59.908305] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.210:1017 [2015-06-02 10:23:59.944630] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.211:1014 [2015-06-02 10:24:00.028556] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.212:1011 [2015-06-02 10:24:02.309238] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.213:1000 [2015-06-02 10:24:02.915960] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.210:1016 [2015-06-02 10:24:02.952056] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.211:1013 [2015-06-02 10:24:03.040358] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.212:1010 [2015-06-02 10:24:05.317047] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.213:996 [2015-06-02 10:24:05.923400] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.210:1015 [2015-06-02 10:24:05.959144] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.211:1012 [2015-06-02 10:24:06.044976] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.212:1009 [2015-06-02 10:24:08.324821] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.213:994 [2015-06-02 10:24:08.932287] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.210:1014 [2015-06-02 10:24:08.967433] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.211:1011 [2015-06-02 10:24:09.052451] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.212:1006 [2015-06-02 10:24:11.332558] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.213:993 [2015-06-02 10:24:11.939689] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.210:1013 [2015-06-02 10:24:11.975001] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.211:1010 [2015-06-02 10:24:12.060625] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.212:1001 [2015-06-02 10:24:14.339808] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.213:992 [2015-06-02 10:24:14.947317] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.210:1012 [2015-06-02 10:24:14.983732] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.211:1007 [2015-06-02 10:24:15.068444] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands hake request from unknown peer 10.8.230.212:1000 [root at SVR6994HW2285 glusterfs]# Best Regards Yuyang Yang -----邮件原件----- 发件人: Atin Mukherjee [mailto:amukherj at redhat.com] 发送时间: Tuesday, June 02, 2015 5:49 PM 收件人: vyyy杨雨阳; Gluster-users at gluster.org 主题: Re: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start On 06/02/2015 12:23 PM, vyyy杨雨阳 wrote: Attached glusterd log files for 10 11 [2015-06-02 06:33:42.668268] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management handshake request from unknown peer 10.8.230.212:1002 From the above log it looks like node 11 tried to handshake with 12, but how come 12 is part of the cluster? Could you run gluster peer status in 12 and share glusterd log? I am still under the same opinion which I had earlier - you are trying to add node which is already a member of another cluster. Best Regards Yuyang Yang -----邮件原件----- 发件人: Atin Mukherjee [mailto:amukherj at redhat.com] 发送时间: Tuesday, June 02, 2015 2:42 PM 收件人: vyyy杨雨阳; Gluster-users at gluster.org 主题: Re: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start On 06/02/2015 12:04 PM, vyyy杨雨阳 wrote: Glusterfs05~glusterfs10 are clustered for 2 years, recently upgrade to 3.6.3 Glusterfs11~glusterfs14 are new nodes need to join the cluster On glusterfs09: [root at SH02SVR5952 ~]# gluster peer status Number of Peers: 6 Hostname: glusterfs06.sh2.ctripcorp.com Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f State: Peer in Cluster (Connected) Hostname: glusterfs07.sh2.ctripcorp.com Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23 State: Peer in Cluster (Connected) Hostname: glusterfs08.sh2.ctripcorp.com Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207 State: Peer in Cluster (Connected) Hostname: 10.8.230.209 Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 State: Peer in Cluster (Connected) Hostname: glusterfs10.sh2.ctripcorp.com Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c State: Peer in Cluster (Disconnected) Hostname: glusterfs11.sh2.ctripcorp.com Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0 State: Peer Rejected (Connected) [root at SH02SVR5952 ~]# Can you attach glusterd log files for 10 11? [root at SH02SVR5952 ~]# gluster volume status Status of volume: JQStore2 Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 2782 Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 2744 Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 5307 Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 3986 NFS Server on localhost 2049 Y 51697 Self-heal Daemon on localhost N/A Y 51710 NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 NFS Server on 10.8.230.209 2049 Y 4091 Self-heal Daemon on 10.8.230.209 N/A Y 4104 Task Status of Volume JQStore2 ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: Webresource Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 2787 Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 2753 Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 5313 Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 3992 NFS Server on localhost 2049 Y 51697 Self-heal Daemon on localhost N/A Y 51710 NFS Server on 10.8.230.209 2049 Y 4091 Self-heal Daemon on 10.8.230.209 N/A Y 4104 NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 Task Status of Volume Webresource ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: ccim Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 2793 Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 2745 Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 5320 Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 3999 NFS Server on localhost 2049 Y 51697 Self-heal Daemon on localhost N/A Y 51710 NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 NFS Server on 10.8.230.209 2049 Y 4091 Self-heal Daemon on 10.8.230.209 N/A Y 4104 Task Status of Volume ccim ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: cloudimage Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 5325 Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 4005 NFS Server on localhost 2049 Y 51697 Self-heal Daemon on localhost N/A Y 51710 NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 NFS Server on 10.8.230.209 2049 Y 4091 Self-heal Daemon on 10.8.230.209 N/A Y 4104 NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 Task Status of Volume cloudimage ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: filews Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 5330 Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 4010 NFS Server on localhost 2049 Y 51697 Self-heal Daemon on localhost N/A Y 51710 NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 NFS Server on 10.8.230.209 2049 Y 4091 Self-heal Daemon on 10.8.230.209 N/A Y 4104 NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 Task Status of Volume filews ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: filex Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 2794 Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 2755 NFS Server on localhost 2049 Y 51697 Self-heal Daemon on localhost N/A Y 51710 NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 NFS Server on 10.8.230.209 2049 Y 4091 Self-heal Daemon on 10.8.230.209 N/A Y 4104 Task Status of Volume filex ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: pubmail Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 2792 Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 2754 NFS Server on localhost 2049 Y 51697 Self-heal Daemon on localhost N/A Y 51710 NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894 Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905 NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185 Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192 NFS Server on 10.8.230.209 2049 Y 4091 Self-heal Daemon on 10.8.230.209 N/A Y 4104 Task Status of Volume pubmail ------------------------------------------------------------------------------ There are no active volume tasks [root at SH02SVR5952 ~]# Best Regards 杨雨阳 Site Operations Center Ext.15554 Seat.1#3F049 We are the F.I.R.S.T. Fast, Innovation, Reliability , Secure, Team -----邮件原件----- 发件人: Atin Mukherjee [mailto:amukherj at redhat.com] 发送时间: Tuesday, June 02, 2015 2:20 PM 收件人: vyyy杨雨阳; Gluster-users at gluster.org 主题: Re: 答复: [Gluster-users] Gluster peer rejected and failed to start On 06/02/2015 11:33 AM, vyyy杨雨阳 wrote: Actually I have 2 problems 1、New nodes can't add to the clusters I cleaned /var/lib/glusterd, now status is State: Accepted peer request (Connected) 2、One of clusterd nodes shown 'Peer rejected' and glusterd failed to start. The log is attached in pre-mail This is a product cluster, this problem is more egent From the log I can clearly see the problematic node is glusterfs09.sh2.ctripcorp.com. Your existing cluster configuration has bricks hosted in glusterfs09.sh2.ctripcorp.com however the same is not part of the cluster. Could you paste the output of gluster peer status and gluster volume status ? Best Regards Yuyang Yang -----邮件原件----- 发件人: Atin Mukherjee [mailto:amukherj at redhat.com] 发送时间: Tuesday, June 02, 2015 12:52 PM 收件人: vyyy杨雨阳; Gluster-users at gluster.org 主题: Re: [Gluster-users] Gluster peer rejected and failed to start On 06/02/2015 10:00 AM, vyyy杨雨阳 wrote: Hi We have a gluster (Version 3.6.3) cluster with 6 nodes, I tried to add 4 more nodes, but ‘Peer Rejected’, then I tried to resolve it by dump /var/lib/glusterd and probe again, not success, this is a question, But strange thing is: A node already in cluster also shown “Peer Reject” I tried to restart glusterd, It failed I found that /var/lib/glusterd/peers is empty, I copied the files from other nodes, still can’t start glusterd It seems like you are trying to peer probe nodes which are either either part of some other clusters (uncleaned nodes). Could you check whether the nodes which you are adding have empty /var/lib/glusterd? If not clean them and retry. ~Atin etc-glusterfs-glusterd.vol.log shown that cluster member as “unknown peer ” [2015-06-02 01:52:14.650635] C [glusterd-handler.c:2369:__glusterd_handle_friend_update] 0-: Received friend update request from unknown peer 04f22ee8-8e00-4c32-a924-b40a0e413aa6 [2015-06-02 01:52:14.650786] C [glusterd-handler.c:2369:__glusterd_handle_friend_update] 0-: Received friend update request from unknown peer 674a78b5-0590-48d4-8752-d4608832ed1d [2015-06-02 01:52:14.657881] C [glusterd-handler.c:2369:__glusterd_handle_friend_update] 0-: Received friend update request from unknown peer 83e1a9db-3134-45e4-acd2-387b12b5b207 [2015-06-02 01:52:17.747865] W [glusterd-handler.c:697:__glusterd_handle_cluster_lock] 0-management: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 doesn't belong to the cluster. Ignoring request. [2015-06-02 01:52:17.747908] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2015-06-02 01:52:40.338885] W [glusterd-handler.c:697:__glusterd_handle_cluster_lock] 0-management: 674a78b5-0590-48d4-8752-d4608832ed1d doesn't belong to the cluster. Ignoring request. [2015-06-02 01:52:40.338929] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully [2015-06-02 01:52:41.310451] W [glusterd-handler.c:697:__glusterd_handle_cluster_lock] 0-management: 674a78b5-0590-48d4-8752-d4608832ed1d doesn't belong to the cluster. Ignoring request. [2015-06-02 01:52:41.310486] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully Debug info is as following, /usr/sbin/glusterd [root at SH02SVR5951 peers]# /usr/sbin/glusterd --debug [2015-06-02 04:09:24.626690] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.6.3 (args: /usr/sbin/glusterd --debug) [2015-06-02 04:09:24.626739] D [logging.c:1763:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5 [2015-06-02 04:09:24.627052] D [MSGID: 0] [glusterfsd.c:613:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol [2015-06-02 04:09:24.629683] I [glusterd.c:1214:init] 0-management: Maximum allowed open file descriptors set to 65536 [2015-06-02 04:09:24.629706] I [glusterd.c:1259:init] 0-management: Using /var/lib/glusterd as working directory [2015-06-02 04:09:24.629764] D [glusterd.c:391:glusterd_rpcsvc_options_build] 0-: listen-backlog value: 128 [2015-06-02 04:09:24.629895] D [rpcsvc.c:2198:rpcsvc_init] 0-rpc-service: RPC service inited. [2015-06-02 04:09:24.629904] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [2015-06-02 04:09:24.629930] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so [2015-06-02 04:09:24.631989] D [socket.c:3807:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled [2015-06-02 04:09:24.632005] D [socket.c:3810:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled [2015-06-02 04:09:24.632013] D [socket.c:3827:socket_init] 0-socket.management: using system polling thread [2015-06-02 04:09:24.632024] D [name.c:550:server_fill_address_family] 0-socket.management: option address-family not specified, defaulting to inet [2015-06-02 04:09:24.632072] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/rdma.so [2015-06-02 04:09:24.632102] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-transport/rdma.so: cannot open shared object file: No such file or directory [2015-06-02 04:09:24.632112] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine [2015-06-02 04:09:24.632122] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2015-06-02 04:09:24.632132] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, Ver: 2, Port: 0 [2015-06-02 04:09:24.632140] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: 1238463, Ver: 2, Port: 0 [2015-06-02 04:09:24.632148] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, Ver: 2, Port: 0 [2015-06-02 04:09:24.632155] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433, Ver: 3, Port: 0 [2015-06-02 04:09:24.632163] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: 1, Port: 0 [2015-06-02 04:09:24.632170] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0 [2015-06-02 04:09:24.632177] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 1239873, Ver: 1, Port: 0 [2015-06-02 04:09:24.632212] D [rpcsvc.c:2198:rpcsvc_init] 0-rpc-service: RPC service inited. [2015-06-02 04:09:24.632220] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0 [2015-06-02 04:09:24.632236] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so [2015-06-02 04:09:24.632302] D [socket.c:3692:socket_init] 0-socket.management: disabling nodelay [2015-06-02 04:09:24.632314] D [socket.c:3807:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled [2015-06-02 04:09:24.632321] D [socket.c:3810:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled [2015-06-02 04:09:24.632329] D [socket.c:3827:socket_init] 0-socket.management: using system polling thread [2015-06-02 04:09:24.632402] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0 [2015-06-02 04:09:24.632414] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), Num: 14398633, Ver: 2, Port: 0 [2015-06-02 04:09:24.632443] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 [2015-06-02 04:09:24.632455] D [glusterd.c:1484:init] 0-management: cannot get run-with-valgrind value [2015-06-02 04:09:24.633086] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [2015-06-02 04:09:24.633518] D [glusterd.c:446:glusterd_check_gsync_present] 0-glusterd: Returning -1 [2015-06-02 04:09:24.634031] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.634057] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.634115] D [store.c:304:gf_store_retrieve_value] 0-: key operating-version found [2015-06-02 04:09:24.634141] I [glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2 [2015-06-02 04:09:24.634196] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.634436] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.634446] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.634469] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.634488] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634496] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2 [2015-06-02 04:09:24.634510] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634517] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 6 [2015-06-02 04:09:24.634528] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634534] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1 [2015-06-02 04:09:24.634544] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634551] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2 [2015-06-02 04:09:24.634562] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634569] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1 [2015-06-02 04:09:24.634580] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634587] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2 [2015-06-02 04:09:24.634597] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634603] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 24 [2015-06-02 04:09:24.634613] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634620] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0 [2015-06-02 04:09:24.634631] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634638] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = 7bf7d81a-f835-4c3b-9c58-4d4ad7ec6e07 [2015-06-02 04:09:24.634652] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634659] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = c7374917-2344-4c42-aeae-1d0bdb7ece0a [2015-06-02 04:09:24.634678] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634689] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = 703c3b9d-4343-4546-9ca3-350aecdd52fe [2015-06-02 04:09:24.634700] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634707] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2 [2015-06-02 04:09:24.634719] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634726] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2 [2015-06-02 04:09:24.634736] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634742] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.latency-measurement value = on [2015-06-02 04:09:24.634773] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.latency-measurement,value:on [2015-06-02 04:09:24.634785] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634791] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.count-fop-hits value = on [2015-06-02 04:09:24.634803] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.count-fop-hits,value:on [2015-06-02 04:09:24.634814] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634821] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.* [2015-06-02 04:09:24.634838] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.* [2015-06-02 04:09:24.634849] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634856] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs05.sh2.ctripcorp.com:-export-sdb-brick3 [2015-06-02 04:09:24.634868] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634874] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs06.sh2.ctripcorp.com:-export-sdb-brick3 [2015-06-02 04:09:24.634885] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634891] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick3 [2015-06-02 04:09:24.634902] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634908] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick3 [2015-06-02 04:09:24.634919] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634925] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-4 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick3 [2015-06-02 04:09:24.634935] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.634942] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-5 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick3 [2015-06-02 04:09:24.634954] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.634984] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.635002] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.635019] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635030] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635039] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635048] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635056] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635066] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635074] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635083] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635092] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635101] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635110] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635118] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635127] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635136] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635144] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635153] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635161] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635368] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.635380] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.635396] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.635416] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635426] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635436] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635447] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635456] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635465] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.635488] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.635501] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635748] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.635766] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.635786] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.635807] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635818] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635828] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635838] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635847] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.635856] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.635909] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.635922] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636158] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.636167] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.636180] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.636202] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636213] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636222] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636232] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636241] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636250] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.636274] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.636286] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636471] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.636479] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.636493] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.636514] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636524] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636533] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636542] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636552] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636561] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.636586] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.636598] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636856] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.636889] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.636922] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.636943] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636954] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636964] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636975] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636984] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.636994] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.637020] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.637036] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.637307] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.637325] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.637346] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.637367] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.637378] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.637388] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.819347] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.819374] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.819385] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.819413] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0 [2015-06-02 04:09:24.819707] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.820187] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.820222] D [store.c:304:gf_store_retrieve_value] 0-: key version found [2015-06-02 04:09:24.820239] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.820532] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.820585] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.820959] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.821064] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.821090] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.837290] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.837540] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.837551] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.837568] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.837589] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.837600] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.837862] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.837880] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.837896] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.837918] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.837929] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.837939] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.837964] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.838216] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.838257] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.838273] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.838295] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838303] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2 [2015-06-02 04:09:24.838314] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838320] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 6 [2015-06-02 04:09:24.838330] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838336] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1 [2015-06-02 04:09:24.838346] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838352] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2 [2015-06-02 04:09:24.838361] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838367] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1 [2015-06-02 04:09:24.838377] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838383] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2 [2015-06-02 04:09:24.838393] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838399] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 6 [2015-06-02 04:09:24.838408] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838414] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0 [2015-06-02 04:09:24.838423] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838430] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = cdc279f4-6cd0-4087-8d2a-440bccb51e33 [2015-06-02 04:09:24.838443] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838449] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = 0b6d3496-0865-45e4-a45b-c221b22d3d92 [2015-06-02 04:09:24.838459] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838465] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = ecedfb4d-3308-4f6c-b188-4d5b868958e1 [2015-06-02 04:09:24.838475] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838482] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2 [2015-06-02 04:09:24.838493] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838499] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2 [2015-06-02 04:09:24.838508] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838515] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.68.* [2015-06-02 04:09:24.838533] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.68.* [2015-06-02 04:09:24.838545] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838552] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs05.sh2.ctripcorp.com:-export-sdb-brick2 [2015-06-02 04:09:24.838563] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838570] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs06.sh2.ctripcorp.com:-export-sdb-brick2 [2015-06-02 04:09:24.838580] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838587] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick2 [2015-06-02 04:09:24.838597] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838604] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick2 [2015-06-02 04:09:24.838614] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838621] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-4 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick2 [2015-06-02 04:09:24.838631] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838637] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-5 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick2 [2015-06-02 04:09:24.838649] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.838697] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.838719] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.838738] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838747] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838756] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838764] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838772] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838781] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838797] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838806] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838815] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838823] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838832] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838840] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838848] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.838857] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839051] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.839063] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.839084] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.839102] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839113] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839123] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839133] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839142] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839151] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.839177] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.839189] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839382] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.839390] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.839403] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.839421] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839431] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839440] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839450] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839459] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839468] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.839491] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.839502] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839700] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.839712] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.839732] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.839750] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839760] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839770] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839780] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.839799] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.839822] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.839834] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840057] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.840075] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.840096] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.840117] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840128] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840137] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840147] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840156] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840166] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.840192] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.840206] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840402] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.840410] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.840424] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.840440] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840450] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840459] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840469] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840478] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840487] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.840513] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.840525] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840799] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.840818] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.840839] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.840859] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840871] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840880] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840891] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840900] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.840910] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.840931] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0 [2015-06-02 04:09:24.841118] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.841359] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.841396] D [store.c:304:gf_store_retrieve_value] 0-: key version found [2015-06-02 04:09:24.841413] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.841557] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.841611] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.841799] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.841875] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.841901] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.842350] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.842469] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.842477] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.842489] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.842514] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842525] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.842636] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.842645] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.842656] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.842696] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842708] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842718] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.842738] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.842848] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.842859] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.842872] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.842891] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842898] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2 [2015-06-02 04:09:24.842908] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842914] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 4 [2015-06-02 04:09:24.842924] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842930] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1 [2015-06-02 04:09:24.842939] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842945] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2 [2015-06-02 04:09:24.842954] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842960] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1 [2015-06-02 04:09:24.842969] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842975] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2 [2015-06-02 04:09:24.842984] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.842990] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 3 [2015-06-02 04:09:24.842999] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843005] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0 [2015-06-02 04:09:24.843015] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843021] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = a3369f63-9fce-4a56-a921-f7206cb8d3ae [2015-06-02 04:09:24.843033] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843039] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = c6012954-928a-43c1-b7ea-336d6ee0bea9 [2015-06-02 04:09:24.843050] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843056] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = bfbd7007-0e0c-45f9-8d59-05e1aee66728 [2015-06-02 04:09:24.843066] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843072] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2 [2015-06-02 04:09:24.843082] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843089] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2 [2015-06-02 04:09:24.843098] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843104] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.83.* [2015-06-02 04:09:24.843119] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.83.* [2015-06-02 04:09:24.843130] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843136] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick4 [2015-06-02 04:09:24.843147] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843154] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick4 [2015-06-02 04:09:24.843164] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843171] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick4 [2015-06-02 04:09:24.843181] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843188] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick4 [2015-06-02 04:09:24.843198] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.843221] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.843238] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.843255] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843264] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843273] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843281] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843290] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843298] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843306] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843314] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843323] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843332] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843340] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843348] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843357] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843365] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843374] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843476] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.843484] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.843499] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.843519] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843720] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843739] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843751] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843761] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843771] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.843802] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.843817] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.843984] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.843996] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.844013] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.844030] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844041] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844050] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844060] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844069] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844078] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.844104] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.844116] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844222] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.844230] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.844243] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.844266] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844276] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844285] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844294] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844303] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844312] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.844334] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.844346] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844443] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.844451] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.844468] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.844487] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844497] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844506] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844516] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844525] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.844534] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.844550] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0 [2015-06-02 04:09:24.844753] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.845000] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.845033] D [store.c:304:gf_store_retrieve_value] 0-: key version found [2015-06-02 04:09:24.845049] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.845199] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.845244] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.845396] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.845448] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.845471] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.846053] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.846195] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.846206] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.846220] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.846241] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846252] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.846362] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.846371] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.846382] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.846404] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846414] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846424] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.846442] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.846543] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.846552] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.846562] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.846585] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846592] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2 [2015-06-02 04:09:24.846603] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846609] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 6 [2015-06-02 04:09:24.846618] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846624] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1 [2015-06-02 04:09:24.846633] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846639] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2 [2015-06-02 04:09:24.846648] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846654] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1 [2015-06-02 04:09:24.846663] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846690] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2 [2015-06-02 04:09:24.846702] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846708] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 3 [2015-06-02 04:09:24.846717] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846723] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0 [2015-06-02 04:09:24.846733] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846739] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = 8869b21b-1bf1-4eba-a52a-1158abde7d23 [2015-06-02 04:09:24.846752] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846758] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = 27298b95-919d-43f7-9507-8cbfbe5292e3 [2015-06-02 04:09:24.846769] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846775] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = 47f52166-020b-4bef-ada1-7e65fcd2834f [2015-06-02 04:09:24.846785] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846792] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2 [2015-06-02 04:09:24.846802] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846808] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2 [2015-06-02 04:09:24.846818] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846824] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.count-fop-hits value = on [2015-06-02 04:09:24.846837] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.count-fop-hits,value:on [2015-06-02 04:09:24.846848] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846854] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.latency-measurement value = on [2015-06-02 04:09:24.846864] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.latency-measurement,value:on [2015-06-02 04:09:24.846875] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846881] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs05.sh2.ctripcorp.com:-export-sdb-brick [2015-06-02 04:09:24.846892] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846899] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs06.sh2.ctripcorp.com:-export-sdb-brick [2015-06-02 04:09:24.846910] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846916] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick [2015-06-02 04:09:24.846927] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846934] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick [2015-06-02 04:09:24.846944] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846952] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-4 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick [2015-06-02 04:09:24.846962] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.846969] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-5 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick [2015-06-02 04:09:24.846981] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.847003] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.847027] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.847045] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847055] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847064] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847072] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847081] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847089] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847098] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847106] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847115] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847124] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847133] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847141] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847150] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847159] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847167] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847176] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847286] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.847294] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.847308] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.847328] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847339] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847348] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847358] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847367] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847376] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.847402] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.847414] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847516] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.847523] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.847537] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.847559] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847569] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847579] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847588] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847597] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847607] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.847628] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.847640] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847747] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.847756] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.847771] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.847794] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847804] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847814] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847823] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847832] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847841] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.847862] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.847873] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.847969] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.847977] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.847990] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.848012] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848022] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848031] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848040] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848049] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848058] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.848080] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.848091] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848187] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.848195] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.848208] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.848230] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848240] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848249] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848259] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848267] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848277] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.848299] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.848310] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848411] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.848419] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.848432] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.848455] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848465] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848474] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848484] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848493] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.848503] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.848519] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0 [2015-06-02 04:09:24.848727] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.848975] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.849002] D [store.c:304:gf_store_retrieve_value] 0-: key version found [2015-06-02 04:09:24.849019] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.849167] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.849212] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.849367] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.849419] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.849442] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.849880] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.850003] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.850012] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.850024] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.850043] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850054] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.850157] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.850166] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.850177] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.850192] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850202] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850211] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.850228] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.850402] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.850414] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.850426] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.850444] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850451] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2 [2015-06-02 04:09:24.850461] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850467] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 4 [2015-06-02 04:09:24.850476] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850482] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1 [2015-06-02 04:09:24.850491] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850497] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2 [2015-06-02 04:09:24.850506] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850512] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1 [2015-06-02 04:09:24.850522] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850528] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2 [2015-06-02 04:09:24.850537] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850543] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = disperse_count value = 0 [2015-06-02 04:09:24.850552] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850558] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = redundancy_count value = 0 [2015-06-02 04:09:24.850567] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850573] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 11 [2015-06-02 04:09:24.850582] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850588] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0 [2015-06-02 04:09:24.850598] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850604] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = ea46609c-be9a-4283-8357-8fdc2e5a190e [2015-06-02 04:09:24.850616] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850622] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = 21b247ce-58ce-4087-8c32-3d578b4cc22d [2015-06-02 04:09:24.850632] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850638] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = 0724171a-c7ce-4528-b54b-2d983f969537 [2015-06-02 04:09:24.850648] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850654] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2 [2015-06-02 04:09:24.850664] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850692] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2 [2015-06-02 04:09:24.850703] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850709] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.92.*,10.8.230.*,192.168.98.* [2015-06-02 04:09:24.850724] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.92.*,10.8.230.*,192.168.98.* [2015-06-02 04:09:24.850736] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850743] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick1 [2015-06-02 04:09:24.850754] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850760] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick1 [2015-06-02 04:09:24.850772] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850778] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick1 [2015-06-02 04:09:24.850789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850795] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick1 [2015-06-02 04:09:24.850807] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.850830] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.850850] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.850867] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850877] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850885] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850894] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850902] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850911] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850919] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850928] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850936] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850945] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850953] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850962] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850971] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850980] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850988] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.850996] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851005] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851164] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.851173] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.851187] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.851203] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851213] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851223] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851232] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851242] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851251] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851261] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.851283] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.851295] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851457] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.851465] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.851478] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.851493] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851504] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851513] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851522] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851531] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851540] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851549] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.851571] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.851583] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851732] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.851741] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.851755] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.851770] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851780] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851799] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851807] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851816] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851826] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.851850] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0 [2015-06-02 04:09:24.851863] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.851959] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.851974] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.852005] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.852026] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.852037] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.852047] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.852061] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.852070] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.852080] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.852090] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.852111] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0 [2015-06-02 04:09:24.852298] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.852524] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.852556] D [store.c:304:gf_store_retrieve_value] 0-: key version found [2015-06-02 04:09:24.852576] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.852723] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.852776] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0 [2015-06-02 04:09:24.852930] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.852982] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.853006] D [store.c:372:gf_store_save_value] 0-: returning: 0 [2015-06-02 04:09:24.853412] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0 [2015-06-02 04:09:24.853524] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.853539] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.853553] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.853573] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.853583] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.853699] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.853718] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.853735] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.853757] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.853768] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.853778] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.853787] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.853798] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.853820] D [glusterd-store.c:3094:glusterd_store_retrieve_volumes] 0-: Returning with 0 [2015-06-02 04:09:24.853962] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.853972] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.853983] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.854000] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854011] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 [2015-06-02 04:09:24.854023] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854032] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854043] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0 [2015-06-02 04:09:24.854054] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.854070] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2015-06-02 04:09:24.854162] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.854171] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.854181] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.854197] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854205] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 [2015-06-02 04:09:24.854216] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854225] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854232] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0 [2015-06-02 04:09:24.854241] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.854255] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2015-06-02 04:09:24.854345] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.854354] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.854364] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.854380] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854390] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 [2015-06-02 04:09:24.854402] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854411] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854417] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0 [2015-06-02 04:09:24.854427] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.854439] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2015-06-02 04:09:24.854527] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.854536] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.854546] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.854561] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854569] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 [2015-06-02 04:09:24.854580] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854589] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854596] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0 [2015-06-02 04:09:24.854605] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.854617] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2015-06-02 04:09:24.854704] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.854713] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.854725] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.854740] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854749] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 [2015-06-02 04:09:24.854760] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854769] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.854776] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0 [2015-06-02 04:09:24.854789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.854803] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2015-06-02 04:09:24.855064] D [store.c:418:gf_store_handle_new] 0-: Returning 0 [2015-06-02 04:09:24.855072] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0 [2015-06-02 04:09:24.855083] D [store.c:499:gf_store_iter_new] 0-: Returning with 0 [2015-06-02 04:09:24.855098] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.855106] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0 [2015-06-02 04:09:24.855116] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.855125] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0 [2015-06-02 04:09:24.855132] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0 [2015-06-02 04:09:24.855141] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1 [2015-06-02 04:09:24.855154] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2015-06-02 04:09:24.855186] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 [2015-06-02 04:09:24.855210] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-02 04:09:24.855218] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30 [2015-06-02 04:09:24.855231] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so [2015-06-02 04:09:24.855324] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled [2015-06-02 04:09:24.855333] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled [2015-06-02 04:09:24.855340] D [socket.c:3827:socket_init] 0-management: using system polling thread [2015-06-02 04:09:24.859610] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.211 (port-24007) for hostname: glusterfs07.sh2.ctripcorp.com and port: 24007 [2015-06-02 04:09:24.859628] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection [2015-06-02 04:09:24.859707] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0 [2015-06-02 04:09:24.859733] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 [2015-06-02 04:09:24.859772] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-02 04:09:24.859781] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30 [2015-06-02 04:09:24.859794] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so [2015-06-02 04:09:24.859905] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled [2015-06-02 04:09:24.859916] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled [2015-06-02 04:09:24.859923] D [socket.c:3827:socket_init] 0-management: using system polling thread [2015-06-02 04:09:24.863505] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.231.160 (port-24007) for hostname: glusterfs11.sh2.ctripcorp.com and port: 24007 [2015-06-02 04:09:24.863521] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection [2015-06-02 04:09:24.863562] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0 [2015-06-02 04:09:24.863585] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 [2015-06-02 04:09:24.863607] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-02 04:09:24.863615] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30 [2015-06-02 04:09:24.863625] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so [2015-06-02 04:09:24.863754] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled [2015-06-02 04:09:24.863765] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled [2015-06-02 04:09:24.863772] D [socket.c:3827:socket_init] 0-management: using system polling thread [2015-06-02 04:09:24.867255] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.210 (port-24007) for hostname: glusterfs06.sh2.ctripcorp.com and port: 24007 [2015-06-02 04:09:24.867271] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection [2015-06-02 04:09:24.867314] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0 [2015-06-02 04:09:24.867337] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 [2015-06-02 04:09:24.867358] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-02 04:09:24.867366] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30 [2015-06-02 04:09:24.867377] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so [2015-06-02 04:09:24.867463] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled [2015-06-02 04:09:24.867476] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled [2015-06-02 04:09:24.867483] D [socket.c:3827:socket_init] 0-management: using system polling thread [2015-06-02 04:09:24.870920] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.214 (port-24007) for hostname: glusterfs10.sh2.ctripcorp.com and port: 24007 [2015-06-02 04:09:24.870935] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection [2015-06-02 04:09:24.870997] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0 [2015-06-02 04:09:24.871021] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 [2015-06-02 04:09:24.871042] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-02 04:09:24.871050] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30 [2015-06-02 04:09:24.871061] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so [2015-06-02 04:09:24.871129] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled [2015-06-02 04:09:24.871138] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled [2015-06-02 04:09:24.871144] D [socket.c:3827:socket_init] 0-management: using system polling thread [2015-06-02 04:09:24.874102] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.209 (port-24007) for hostname: 10.8.230.209 and port: 24007 [2015-06-02 04:09:24.874116] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection [2015-06-02 04:09:24.874159] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0 [2015-06-02 04:09:24.874181] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0 [2015-06-02 04:09:24.874202] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2015-06-02 04:09:24.874210] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30 [2015-06-02 04:09:24.874219] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so [2015-06-02 04:09:24.874292] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled [2015-06-02 04:09:24.874301] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled [2015-06-02 04:09:24.874326] D [socket.c:3827:socket_init] 0-management: using system polling thread [2015-06-02 04:09:24.877759] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.212 (port-24007) for hostname: glusterfs08.sh2.ctripcorp.com and port: 24007 [2015-06-02 04:09:24.877774] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection [2015-06-02 04:09:24.877818] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0 [2015-06-02 04:09:24.877831] D [glusterd-store.c:4106:glusterd_store_retrieve_peers] 0-: Returning with 0 [2015-06-02 04:09:24.877863] I [glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [2015-06-02 04:09:24.877873] D [glusterd-store.c:3612:glusterd_store_retrieve_snaps] 0-: Returning with 0 [2015-06-02 04:09:24.881125] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0 [2015-06-02 04:09:24.881140] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0 [2015-06-02 04:09:24.881148] D [glusterd-peer-utils.c:600:gd_peerinfo_find_from_hostname] 0-management: Friend glusterfs06.sh2.ctripcorp.com found.. state: 3 [2015-06-02 04:09:24.881156] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0 [2015-06-02 04:09:24.881162] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0 [2015-06-02 04:09:24.881169] D [glusterd-peer-utils.c:600:gd_peerinfo_find_from_hostname] 0-management: Friend glusterfs07.sh2.ctripcorp.com found.. state: 3 [2015-06-02 04:09:24.881176] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0 [2015-06-02 04:09:24.881182] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0 [2015-06-02 04:09:24.881188] D [glusterd-peer-utils.c:600:gd_peerinfo_find_from_hostname] 0-management: Friend glusterfs08.sh2.ctripcorp.com found.. state: 3 [2015-06-02 04:09:24.881195] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0 [2015-06-02 04:09:24.881201] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0 [2015-06-02 04:09:24.890739] D [glusterd-peer-utils.c:121:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: glusterfs09.sh2.ctripcorp.com [2015-06-02 04:09:24.891607] D [common-utils.c:2897:gf_is_local_addr] 0-management: 10.8.230.213 [2015-06-02 04:09:24.891750] D [common-utils.c:2897:gf_is_local_addr] 0-management: 10.8.230.213 [2015-06-02 04:09:24.891820] D [common-utils.c:2897:gf_is_local_addr] 0-management: 10.8.230.213 [2015-06-02 04:09:24.891881] D [common-utils.c:2913:gf_is_local_addr] 0-management: glusterfs09.sh2.ctripcorp.com is not local [2015-06-02 04:09:24.891890] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning -1 [2015-06-02 04:09:24.891898] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning -1 [2015-06-02 04:09:24.891904] E [glusterd-store.c:4244:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore [2015-06-02 04:09:24.891911] D [glusterd-store.c:4321:glusterd_restore] 0-: Returning -1 [2015-06-02 04:09:24.891923] E [xlator.c:425:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again [2015-06-02 04:09:24.891933] E [graph.c:322:glusterfs_graph_init] 0-management: initializing translator failed [2015-06-02 04:09:24.891941] E [graph.c:525:glusterfs_graph_activate] 0-graph: init failed [2015-06-02 04:09:24.892164] D [logging.c:1740:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size reduced. About to flush 2 extra log messages [2015-06-02 04:09:24.892178] D [logging.c:1743:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 2 extra log messages [2015-06-02 04:09:24.892325] W [glusterfsd.c:1194:cleanup_and_exit] (-- 0-: received signum (0), shutting down [2015-06-02 04:09:24.892345] D [glusterfsd-mgmt.c:2244:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given [root at SH02SVR5951 peers]# Best Regards Yuyang Yang _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://www.gluster.org/mailman/listinfo/gluster-users -- ~Atin _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150604/6d8de576/attachment.html>
More information about the Gluster-users
mailing list