[Gluster-users] 答复: 答复: 答复: Gluster peer rejected and failed to start
vyyy杨雨阳
yuyangyang at Ctrip.com
Thu Jun 4 03:00:01 UTC 2015
The problem 2 has resolved.
From glusterd log file, I find Cksums error as following:
[2015-06-04 01:25:03.137798] E [MSGID: 106010] [glusterd-utils.c:3297:glusterd_compare_friend_volume] 0-management: Version of Cksums JQStore2 differ. local cksum = 2617028277, remote cksum = 2210669223 on peer glusterfs09.sh2.ctripcorp.com
The solution for this problem was to do the following, on the BAD node:
/etc/init.d/glusterd stop
rsync -havP --delete 172.26.177.115:/var/lib/glusterd/vols/ /var/lib/glusterd/vols/
/etc/init.d/glusterd start
The problem 1 is still under inverstigate:
New nodes can not join the cluster, log file shown: failed to get the 'versions' from peer
I have checked the file /var/lib/glusterd/ glusterd.info, all nodes are " operating-version=2"
[2015-06-04 02:13:49.199416] E [glusterd-handshake.c:1702:__glusterd_mgmt_hndsk_version_cbk] 0-management: failed to get the
'versions' from peer (10.8.230.211:24007)
[2015-06-04 02:14:07.112577] I [glusterd-handler.c:1015:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req glust
erfs05.sh2.ctripcorp.com 24007
[2015-06-04 02:14:07.113778] I [glusterd-handler.c:3165:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: g
lusterfs05.sh2.ctripcorp.com (24007)
[2015-06-04 02:14:07.131023] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-06-04 02:14:07.134829] I [glusterd-handler.c:3098:glusterd_friend_add] 0-management: connect returned 0
[
Best Regards
杨雨阳
Site Operations Center
Ext.15554 Seat.1#3F049
We are the F.I.R.S.T.
Fast, Innovation, Reliability , Secure, Team
-----邮件原件-----
发件人: vyyy杨雨阳
发送时间: Wednesday, June 03, 2015 5:37 PM
收件人: 'Atin Mukherjee'; Gluster-users at gluster.org
主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start
Hi Atin and everybody,
I have forced peer detach the news nodes ,The peer status seems even strange, glusterfs05 and glusterfs10 are peer in cluster, Can you help me? Any advice is appreciated.
Glusterfs05
[root at sh02svr5956 ~]# gluster peer status
Number of Peers: 5
Hostname: glusterfs06.sh2.ctripcorp.com
Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
State: Peer Rejected (Connected)
Hostname: glusterfs07.sh2.ctripcorp.com
Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
State: Peer Rejected (Connected)
Hostname: glusterfs08.sh2.ctripcorp.com
Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
State: Peer Rejected (Connected)
Hostname: glusterfs10.sh2.ctripcorp.com
Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
State: Peer in Cluster (Connected)
Hostname: glusterfs09.sh2.ctripcorp.com
Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
State: Peer Rejected (Connected)
[root at sh02svr5956 ~]#
Glusterfs06
[root at SH02SVR5955 ~]# gluster peer status
Number of Peers: 5
Hostname: glusterfs10.sh2.ctripcorp.com
Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
State: Peer Rejected (Connected)
Hostname: glusterfs08.sh2.ctripcorp.com
Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
State: Peer in Cluster (Connected)
Hostname: glusterfs07.sh2.ctripcorp.com
Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
State: Peer in Cluster (Connected)
Hostname: 10.8.230.209
Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
State: Peer Rejected (Connected)
Hostname: glusterfs09.sh2.ctripcorp.com
Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
State: Peer in Cluster (Connected)
[root at SH02SVR5955 ~]#
Glusterfs07
[root at SH02SVR5954 ~]# gluster peer status
Number of Peers: 5
Hostname: glusterfs10.sh2.ctripcorp.com
Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
State: Peer Rejected (Connected)
Hostname: 10.8.230.209
Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
State: Peer Rejected (Connected)
Hostname: glusterfs06.sh2.ctripcorp.com
Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
State: Peer in Cluster (Connected)
Hostname: glusterfs08.sh2.ctripcorp.com
Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
State: Peer in Cluster (Connected)
Hostname: glusterfs09.sh2.ctripcorp.com
Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
State: Peer in Cluster (Connected)
[root at SH02SVR5954 ~]#
Glusterfs08
[root at SH02SVR5953 ~]# gluster peer status
Number of Peers: 5
Hostname: glusterfs07.sh2.ctripcorp.com
Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
State: Peer in Cluster (Connected)
Hostname: glusterfs10.sh2.ctripcorp.com
Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
State: Peer Rejected (Connected)
Hostname: glusterfs06.sh2.ctripcorp.com
Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
State: Peer in Cluster (Connected)
Hostname: 10.8.230.209
Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
State: Peer Rejected (Connected)
Hostname: glusterfs09.sh2.ctripcorp.com
Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
State: Peer in Cluster (Connected)
[root at SH02SVR5953 ~]#
[root at SH02SVR5952 ~]# gluster peer status
Number of Peers: 5
Hostname: glusterfs06.sh2.ctripcorp.com
Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
State: Peer in Cluster (Connected)
Hostname: glusterfs07.sh2.ctripcorp.com
Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
State: Peer in Cluster (Connected)
Hostname: glusterfs08.sh2.ctripcorp.com
Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
State: Peer in Cluster (Connected)
Hostname: 10.8.230.209
Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
State: Peer Rejected (Connected)
Hostname: glusterfs10.sh2.ctripcorp.com
Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
State: Peer Rejected (Connected)
[root at SH02SVR5952 ~]#
Glusterfs10
[root at SH02SVR5951 ~]# gluster peer status
Number of Peers: 6
Hostname: glusterfs07.sh2.ctripcorp.com
Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
State: Peer Rejected (Connected)
Hostname: glusterfs11.sh2.ctripcorp.com
Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
State: Peer Rejected (Connected)
Hostname: glusterfs09.sh2.ctripcorp.com
Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
State: Peer Rejected (Connected)
Hostname: glusterfs06.sh2.ctripcorp.com
Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
State: Peer Rejected (Connected)
Hostname: 10.8.230.209
Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
State: Peer in Cluster (Connected)
Hostname: glusterfs08.sh2.ctripcorp.com
Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
State: Peer Rejected (Connected)
[root at SH02SVR5951 ~]#
Best Regards
杨雨阳
Site Operations Center
Ext.15554 Seat.1#3F049
We are the F.I.R.S.T.
Fast, Innovation, Reliability , Secure, Team
-----邮件原件-----
发件人: vyyy杨雨阳
发送时间: Wednesday, June 03, 2015 2:55 PM
收件人: 'Atin Mukherjee'
主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start
Can I delete the files in /var/lib/glusterd/peers which state is not =3 on all nodes, then restart glusterd?
[root at sh02svr5956 peers]# more 0475e11a-3c43-4b76-99dd-1d2e814c5f15
uuid=0475e11a-3c43-4b76-99dd-1d2e814c5f15
state=6
hostname1=glusterfs12.sh2.ctripcorp.com
Best Regards
Yang
-----邮件原件-----
发件人: vyyy杨雨阳
发送时间: Wednesday, June 03, 2015 2:39 PM
收件人: 'Atin Mukherjee'
主题: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start
Since it's intranet, You can only control my desktop, and then access these node.
Should I apply a webex room?
Best Regards
杨雨阳
Site Operations Center
Ext.15554 Seat.1#3F049
We are the F.I.R.S.T.
Fast, Innovation, Reliability , Secure, Team
-----邮件原件-----
发件人: Atin Mukherjee [mailto:amukherj at redhat.com]
发送时间: Wednesday, June 03, 2015 2:36 PM
收件人: vyyy杨雨阳
主题: Re: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start
Is it possible to share your ip/user/passwd for few of these nodes?
On 06/03/2015 11:50 AM, vyyy杨雨阳 wrote:
> Can you remote control my desktop (like webex)?
> The nodes host bricks. And on the new nodes, I did "clean /var/lib/glusterd and restart" several times, but have no effect.
> Is there any other work around?
>
>
> Best Regards
> Yang
>
> -----邮件原件-----
> 发件人: Atin Mukherjee [mailto:amukherj at redhat.com]
> 发送时间: Wednesday, June 03, 2015 12:33 PM
> 收件人: vyyy杨雨阳; Gluster-users at gluster.org
> 主题: Re: 答复: [Gluster-users] Gluster peer rejected and failed to start
>
> I can't comment on what went wrong till I can have visibility on the
> entire cluster. Does the node which fails to start host any brick? If
> not then work around could be to clean /var/lib/glusterd and restart
> glusterd, that should sync up data and bring this node in consistency.
>
> HTH,
> Atin
>
> On 06/03/2015 08:34 AM, vyyy杨雨阳 wrote:
>> I found that /var/lib/glusterd/peers can't include node's own peer, I delete this file. Glusterd started finally.
>> But the peer status still not normal.
>> Seems that glusterfs10 leave the cluster and join the new nodes alone.
>>
>> Peer status on glusterfs10
>> [root at SH02SVR5951 ~]# gluster peer status
>> Number of Peers: 7
>>
>> Hostname: glusterfs07.sh2.ctripcorp.com
>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs11.sh2.ctripcorp.com
>> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
>> State: Accepted peer request (Connected)
>>
>> Hostname: glusterfs09.sh2.ctripcorp.com
>> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs06.sh2.ctripcorp.com
>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>> State: Peer Rejected (Connected)
>>
>> Hostname: 10.8.230.209
>> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs08.sh2.ctripcorp.com
>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs12.sh2.ctripcorp.com
>> Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15
>> State: Peer in Cluster (Connected)
>>
>>
>>
>> Peer status on glusterfs10
>>
>> [root at SVR6993HW2285 ~]# gluster peer status
>> Number of Peers: 6
>>
>> Hostname: glusterfs05.sh2.ctripcorp.com
>> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs09.sh2.ctripcorp.com
>> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs07.sh2.ctripcorp.com
>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs08.sh2.ctripcorp.com
>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs06.sh2.ctripcorp.com
>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs10.sh2.ctripcorp.com
>> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>> State: Peer in Cluster (Connected)
>>
>>
>> Peer status on glusterfs05, this is the node that I tried to peer probe new nodes (glusterfs11 to glusterfs14)
>>
>> [root at sh02svr5956 ~]# gluster peer status
>> Number of Peers: 9
>>
>> Hostname: glusterfs06.sh2.ctripcorp.com
>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs07.sh2.ctripcorp.com
>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs08.sh2.ctripcorp.com
>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs10.sh2.ctripcorp.com
>> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs09.sh2.ctripcorp.com
>> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs11.sh2.ctripcorp.com
>> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs12.sh2.ctripcorp.com
>> Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs13.sh2.ctripcorp.com
>> Uuid: 43e96d31-6c6d-4096-832a-440e544554ca
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs14.sh2.ctripcorp.com
>> Uuid: 45c11af9-7850-4598-ad62-4bb343b62524
>> State: Peer Rejected (Connected)
>> [root at sh02svr5956 ~]#
>>
>>
>>
>>
>> Best Regards
>> Yang
>>
>>
>> -----邮件原件-----
>> 发件人: vyyy杨雨阳
>> 发送时间: Wednesday, June 03, 2015 10:04 AM
>> 收件人: 'Atin Mukherjee'; 'Gluster-users at gluster.org'
>> 主题: [Gluster-users] Gluster peer rejected and failed to start
>>
>> I am still messed, can you help me figure out problem 2 (A node of cluster failed to start glusterd) priority
>>
>> The glusterd log on the node gluster10 shown ' resolve brick failed in restore'
>> I checked /var/lib/glusterd/vols, seems ok
>>
>> Glusterd log file Error message:
>>
>> [2015-06-03 01:31:32.564010] E [glusterd-store.c:4244:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore
>> [2015-06-03 01:31:32.564036] E [xlator.c:425:xlator_init] 0-management: Initialization of volume 'management' failed, review
>> your volfile again
>>
>>
>>
>>
>> Best Regards
>> Yang
>>
>> -----邮件原件-----
>> 发件人: vyyy杨雨阳
>> 发送时间: Tuesday, June 02, 2015 6:27 PM
>> 收件人: 'Atin Mukherjee'; Gluster-users at gluster.org
>> 主题: 答复: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start
>>
>> Gluster11 to gluster14 are new nodes I tried to join exit cluster but failed
>>
>>
>> On gluster12:
>> [root at SVR6994HW2285 ~]# gluster peer status
>> Number of Peers: 1
>>
>> Hostname: glusterfs05.sh2.ctripcorp.com
>> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>> State: Accepted peer request (Connected)
>> [root at SVR6994HW2285 ~]#
>>
>>
>>
>> On gluster05: (peer status seems strange on all those nodes):
>>
>> [root at sh02svr5956 ~]# gluster peer status
>> Number of Peers: 9
>>
>> Hostname: glusterfs06.sh2.ctripcorp.com
>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs07.sh2.ctripcorp.com
>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs08.sh2.ctripcorp.com
>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs10.sh2.ctripcorp.com
>> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>> State: Peer in Cluster (Disconnected)
>>
>> Hostname: glusterfs09.sh2.ctripcorp.com
>> Uuid: 674a78b5-0590-48d4-8752-d4608832ed1d
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs11.sh2.ctripcorp.com
>> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs12.sh2.ctripcorp.com
>> Uuid: 0475e11a-3c43-4b76-99dd-1d2e814c5f15
>> State: Peer in Cluster (Connected)
>>
>> Hostname: glusterfs13.sh2.ctripcorp.com
>> Uuid: 43e96d31-6c6d-4096-832a-440e544554ca
>> State: Peer Rejected (Connected)
>>
>> Hostname: glusterfs14.sh2.ctripcorp.com
>> Uuid: 45c11af9-7850-4598-ad62-4bb343b62524
>> State: Peer Rejected (Connected)
>> [root at sh02svr5956 ~]#
>>
>>
>>
>> The glusterd log file
>>
>> [root at SVR6994HW2285 glusterfs]# more etc-glusterfs-glusterd.vol.log
>> [2015-06-02 10:23:48.368697] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glust
>> erd version 3.6.3 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
>> [2015-06-02 10:23:48.372587] I [glusterd.c:1214:init] 0-management: Maximum allowed open file descriptors set to 65536
>> [2015-06-02 10:23:48.372625] I [glusterd.c:1259:init] 0-management: Using /var/lib/glusterd as working directory
>> [2015-06-02 10:23:48.374890] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-trans
>> port/rdma.so: cannot open shared object file: No such file or directory
>> [2015-06-02 10:23:48.374908] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-
>> type 'rdma' is not valid or not found on this machine
>> [2015-06-02 10:23:48.374920] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the tra
>> nsport failed
>> [2015-06-02 10:23:48.376539] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed
>> in the system
>> [2015-06-02 10:23:48.377464] I [glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2
>> [2015-06-02 10:23:48.378122] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
>> [2015-06-02 10:23:48.378191] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>> [2015-06-02 10:23:48.383072] I [glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snap
>> s list.
>> Final graph:
>> +------------------------------------------------------------------------------+
>> 1: volume management
>> 2: type mgmt/glusterd
>> 3: option rpc-auth.auth-glusterfs on
>> 4: option rpc-auth.auth-unix on
>> 5: option rpc-auth.auth-null on
>> 6: option transport.socket.listen-backlog 128
>> 7: option ping-timeout 30
>> 8: option transport.socket.read-fail-log off
>> 9: option transport.socket.keepalive-interval 2
>> 10: option transport.socket.keepalive-time 10
>> 11: option transport-type rdma
>> 12: option working-directory /var/lib/glusterd
>> 13: end-volume
>> 14:
>> +------------------------------------------------------------------------------+
>> [2015-06-02 10:23:48.413755] I [glusterd.c:146:glusterd_uuid_init] 0-management: retrieved UUID: 0475e11a-3c43-4b76-99dd-1d2e
>> 814c5f15
>> [2015-06-02 10:23:48.414345] I [glusterd-rpc-ops.c:245:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 04f
>> 22ee8-8e00-4c32-a924-b40a0e413aa6, host: glusterfs05.sh2.ctripcorp.com
>> [2015-06-02 10:23:48.414883] I [glusterd-rpc-ops.c:387:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req
>> [2015-06-02 10:23:50.088645] I [glusterd-rpc-ops.c:437:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 04f22ee
>> 8-8e00-4c32-a924-b40a0e413aa6, host: glusterfs05.sh2.ctripcorp.com, port: 0
>> [2015-06-02 10:23:50.090245] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 04
>> f22ee8-8e00-4c32-a924-b40a0e413aa6
>> [2015-06-02 10:23:50.107013] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.212:1016
>> [2015-06-02 10:23:50.107805] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.211:1018
>> [2015-06-02 10:23:50.108358] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.210:1020
>> [2015-06-02 10:23:50.139963] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.213:1008
>> [2015-06-02 10:23:53.286070] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.213:1006
>> [2015-06-02 10:23:53.893225] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.210:1019
>> [2015-06-02 10:23:53.928314] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.211:1017
>> [2015-06-02 10:23:54.013229] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.212:1014
>> [2015-06-02 10:23:54.273022] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>> [2015-06-02 10:23:55.531073] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>> [2015-06-02 10:23:56.293171] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.213:1004
>> [2015-06-02 10:23:56.900676] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.210:1018
>> [2015-06-02 10:23:56.935113] W [glusterd-op-sm.c:4060:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
>> [2015-06-02 10:23:56.936892] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.211:1016
>> [2015-06-02 10:23:57.020995] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.212:1012
>> [2015-06-02 10:23:57.659165] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>> [2015-06-02 10:23:58.332111] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>> [2015-06-02 10:23:58.929018] I [glusterd-handler.c:1225:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>> [2015-06-02 10:23:59.100566] I [glusterd-handshake.c:1119:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-vers
>> ion 2
>> [2015-06-02 10:23:59.103665] I [glusterd-handler.c:2216:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe fro
>> m uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>> [2015-06-02 10:23:59.110710] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick
>> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick resides
>> [2015-06-02 10:23:59.110735] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct
>> ripcorp.com:/export/sdb/brick for JQStore2
>> [2015-06-02 10:23:59.131973] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick
>> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick3 resides
>> [2015-06-02 10:23:59.131994] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct
>> ripcorp.com:/export/sdb/brick3 for Webresource
>> [2015-06-02 10:23:59.152949] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick
>> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick2 resides
>> [2015-06-02 10:23:59.152969] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct
>> ripcorp.com:/export/sdb/brick2 for ccim
>> [2015-06-02 10:23:59.172948] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick
>> glusterfs07.sh2.ctripcorp.com:/export/sdb/brick4 resides
>> [2015-06-02 10:23:59.172968] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs07.sh2.ct
>> ripcorp.com:/export/sdb/brick4 for cloudimage
>> [2015-06-02 10:23:59.189033] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick
>> glusterfs07.sh2.ctripcorp.com:/export/sdb/brick1 resides
>> [2015-06-02 10:23:59.189053] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs07.sh2.ct
>> ripcorp.com:/export/sdb/brick1 for filews
>> [2015-06-02 10:23:59.203985] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick
>> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick1 resides
>> [2015-06-02 10:23:59.204005] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct
>> ripcorp.com:/export/sdb/brick1 for filex
>> [2015-06-02 10:23:59.212727] E [glusterd-utils.c:6861:glusterd_brick_start] 0-management: Could not find peer on which brick
>> glusterfs06.sh2.ctripcorp.com:/export/sdb/brick4 resides
>> [2015-06-02 10:23:59.212747] E [glusterd-op-sm.c:1782:glusterd_start_bricks] 0-management: Failed to start glusterfs06.sh2.ct
>> ripcorp.com:/export/sdb/brick4 for pubmail
>> [2015-06-02 10:23:59.237408] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>> [2015-06-02 10:23:59.243681] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>> [2015-06-02 10:23:59.243947] I [glusterd-handler.c:3334:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to glusterfs05.s
>> h2.ctripcorp.com (0), ret: 0
>> [2015-06-02 10:23:59.245638] W [socket.c:620:__socket_rwv] 0-management: readv on /var/run/5dd5697669073fb549309773b211e62a.s
>> ocket failed (Invalid argument)
>> [2015-06-02 10:23:59.245679] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: nfs has
>> disconnected from glusterd.
>> [2015-06-02 10:23:59.245703] W [socket.c:620:__socket_rwv] 0-management: readv on /var/run/1578ba566e5a93f452fe8ad2e91b8682.s
>> ocket failed (Invalid argument)
>> [2015-06-02 10:23:59.245719] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: glusters
>> hd has disconnected from glusterd.
>> [2015-06-02 10:23:59.245748] I [glusterd-rpc-ops.c:634:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 04
>> f22ee8-8e00-4c32-a924-b40a0e413aa6
>> [2015-06-02 10:23:59.300666] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.213:1002
>> [2015-06-02 10:23:59.569403] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>> [2015-06-02 10:23:59.570725] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>> [2015-06-02 10:23:59.571481] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>> [2015-06-02 10:23:59.572218] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>> [2015-06-02 10:23:59.572858] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>> [2015-06-02 10:23:59.573422] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>> [2015-06-02 10:23:59.574062] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>> [2015-06-02 10:23:59.574655] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>> [2015-06-02 10:23:59.908305] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.210:1017
>> [2015-06-02 10:23:59.944630] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.211:1014
>> [2015-06-02 10:24:00.028556] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.212:1011
>> [2015-06-02 10:24:02.309238] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.213:1000
>> [2015-06-02 10:24:02.915960] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.210:1016
>> [2015-06-02 10:24:02.952056] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.211:1013
>> [2015-06-02 10:24:03.040358] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.212:1010
>> [2015-06-02 10:24:05.317047] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.213:996
>> [2015-06-02 10:24:05.923400] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.210:1015
>> [2015-06-02 10:24:05.959144] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.211:1012
>> [2015-06-02 10:24:06.044976] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.212:1009
>> [2015-06-02 10:24:08.324821] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.213:994
>> [2015-06-02 10:24:08.932287] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.210:1014
>> [2015-06-02 10:24:08.967433] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.211:1011
>> [2015-06-02 10:24:09.052451] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.212:1006
>> [2015-06-02 10:24:11.332558] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.213:993
>> [2015-06-02 10:24:11.939689] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.210:1013
>> [2015-06-02 10:24:11.975001] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.211:1010
>> [2015-06-02 10:24:12.060625] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.212:1001
>> [2015-06-02 10:24:14.339808] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.213:992
>> [2015-06-02 10:24:14.947317] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.210:1012
>> [2015-06-02 10:24:14.983732] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.211:1007
>> [2015-06-02 10:24:15.068444] E [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management: Rejecting management hands
>> hake request from unknown peer 10.8.230.212:1000
>> [root at SVR6994HW2285 glusterfs]#
>>
>>
>>
>>
>> Best Regards
>> Yuyang Yang
>>
>>
>> -----邮件原件-----
>> 发件人: Atin Mukherjee [mailto:amukherj at redhat.com]
>> 发送时间: Tuesday, June 02, 2015 5:49 PM
>> 收件人: vyyy杨雨阳; Gluster-users at gluster.org
>> 主题: Re: 答复: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start
>>
>>
>>
>> On 06/02/2015 12:23 PM, vyyy杨雨阳 wrote:
>>> Attached glusterd log files for 10 & 11
>> [2015-06-02 06:33:42.668268] E
>> [glusterd-handshake.c:972:gd_validate_mgmt_hndsk_req] 0-management:
>> Rejecting management handshake request from unknown peer 10.8.230.212:1002
>>
>> From the above log it looks like node 11 tried to handshake with 12, but
>> how come 12 is part of the cluster? Could you run gluster peer status in
>> 12 and share glusterd log? I am still under the same opinion which I had
>> earlier - you are trying to add node which is already a member of
>> another cluster.
>>>
>>> Best Regards
>>> Yuyang Yang
>>>
>>> -----邮件原件-----
>>> 发件人: Atin Mukherjee [mailto:amukherj at redhat.com]
>>> 发送时间: Tuesday, June 02, 2015 2:42 PM
>>> 收件人: vyyy杨雨阳; Gluster-users at gluster.org
>>> 主题: Re: 答复: 答复: [Gluster-users] Gluster peer rejected and failed to start
>>>
>>>
>>>
>>> On 06/02/2015 12:04 PM, vyyy杨雨阳 wrote:
>>>> Glusterfs05~glusterfs10 are clustered for 2 years, recently upgrade to 3.6.3
>>>> Glusterfs11~glusterfs14 are new nodes need to join the cluster
>>>>
>>>> On glusterfs09:
>>>>
>>>> [root at SH02SVR5952 ~]# gluster peer status
>>>> Number of Peers: 6
>>>>
>>>> Hostname: glusterfs06.sh2.ctripcorp.com
>>>> Uuid: 2cb15023-28b0-4d0d-8a43-b8c6e570776f
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: glusterfs07.sh2.ctripcorp.com
>>>> Uuid: 5357c40d-7e34-41f0-a96b-9aa76e52ad23
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: glusterfs08.sh2.ctripcorp.com
>>>> Uuid: 83e1a9db-3134-45e4-acd2-387b12b5b207
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: 10.8.230.209
>>>> Uuid: 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: glusterfs10.sh2.ctripcorp.com
>>>> Uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>>>> State: Peer in Cluster (Disconnected)
>>>>
>>>> Hostname: glusterfs11.sh2.ctripcorp.com
>>>> Uuid: 2d703550-92b5-4f5e-af90-ff2fbf3366f0
>>>> State: Peer Rejected (Connected)
>>>> [root at SH02SVR5952 ~]#
>>> Can you attach glusterd log files for 10 & 11?
>>>>
>>>> [root at SH02SVR5952 ~]# gluster volume status
>>>> Status of volume: JQStore2
>>>> Gluster process Port Online Pid
>>>> ------------------------------------------------------------------------------
>>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 2782
>>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 2744
>>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 5307
>>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick 49152 Y 3986
>>>> NFS Server on localhost 2049 Y 51697
>>>> Self-heal Daemon on localhost N/A Y 51710
>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894
>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905
>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185
>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192
>>>> NFS Server on 10.8.230.209 2049 Y 4091
>>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104
>>>>
>>>> Task Status of Volume JQStore2
>>>> ------------------------------------------------------------------------------
>>>> There are no active volume tasks
>>>>
>>>> Status of volume: Webresource
>>>> Gluster process Port Online Pid
>>>> ------------------------------------------------------------------------------
>>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 2787
>>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 2753
>>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 5313
>>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick3 49155 Y 3992
>>>> NFS Server on localhost 2049 Y 51697
>>>> Self-heal Daemon on localhost N/A Y 51710
>>>> NFS Server on 10.8.230.209 2049 Y 4091
>>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104
>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185
>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192
>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894
>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905
>>>>
>>>> Task Status of Volume Webresource
>>>> ------------------------------------------------------------------------------
>>>> There are no active volume tasks
>>>>
>>>> Status of volume: ccim
>>>> Gluster process Port Online Pid
>>>> ------------------------------------------------------------------------------
>>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 2793
>>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 2745
>>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 5320
>>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick2 49154 Y 3999
>>>> NFS Server on localhost 2049 Y 51697
>>>> Self-heal Daemon on localhost N/A Y 51710
>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185
>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192
>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894
>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905
>>>> NFS Server on 10.8.230.209 2049 Y 4091
>>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104
>>>>
>>>> Task Status of Volume ccim
>>>> ------------------------------------------------------------------------------
>>>> There are no active volume tasks
>>>>
>>>> Status of volume: cloudimage
>>>> Gluster process Port Online Pid
>>>> ------------------------------------------------------------------------------
>>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 5325
>>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 4005
>>>> NFS Server on localhost 2049 Y 51697
>>>> Self-heal Daemon on localhost N/A Y 51710
>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185
>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192
>>>> NFS Server on 10.8.230.209 2049 Y 4091
>>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104
>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894
>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905
>>>>
>>>> Task Status of Volume cloudimage
>>>> ------------------------------------------------------------------------------
>>>> There are no active volume tasks
>>>>
>>>> Status of volume: filews
>>>> Gluster process Port Online Pid
>>>> ------------------------------------------------------------------------------
>>>> Brick glusterfs07.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 5330
>>>> Brick glusterfs09.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 4010
>>>> NFS Server on localhost 2049 Y 51697
>>>> Self-heal Daemon on localhost N/A Y 51710
>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185
>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192
>>>> NFS Server on 10.8.230.209 2049 Y 4091
>>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104
>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894
>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905
>>>>
>>>> Task Status of Volume filews
>>>> ------------------------------------------------------------------------------
>>>> There are no active volume tasks
>>>>
>>>> Status of volume: filex
>>>> Gluster process Port Online Pid
>>>> ------------------------------------------------------------------------------
>>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 2794
>>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick1 49153 Y 2755
>>>> NFS Server on localhost 2049 Y 51697
>>>> Self-heal Daemon on localhost N/A Y 51710
>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894
>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905
>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185
>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192
>>>> NFS Server on 10.8.230.209 2049 Y 4091
>>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104
>>>>
>>>> Task Status of Volume filex
>>>> ------------------------------------------------------------------------------
>>>> There are no active volume tasks
>>>>
>>>> Status of volume: pubmail
>>>> Gluster process Port Online Pid
>>>> ------------------------------------------------------------------------------
>>>> Brick glusterfs05.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 2792
>>>> Brick glusterfs06.sh2.ctripcorp.com:/export/sdb/brick4 49156 Y 2754
>>>> NFS Server on localhost 2049 Y 51697
>>>> Self-heal Daemon on localhost N/A Y 51710
>>>> NFS Server on glusterfs07.sh2.ctripcorp.com 2049 Y 110894
>>>> Self-heal Daemon on glusterfs07.sh2.ctripcorp.com N/A Y 110905
>>>> NFS Server on glusterfs06.sh2.ctripcorp.com 2049 Y 22185
>>>> Self-heal Daemon on glusterfs06.sh2.ctripcorp.com N/A Y 22192
>>>> NFS Server on 10.8.230.209 2049 Y 4091
>>>> Self-heal Daemon on 10.8.230.209 N/A Y 4104
>>>>
>>>> Task Status of Volume pubmail
>>>> ------------------------------------------------------------------------------
>>>> There are no active volume tasks
>>>>
>>>> [root at SH02SVR5952 ~]#
>>>>
>>>>
>>>>
>>>> Best Regards
>>>> 杨雨阳
>>>> Site Operations Center
>>>> Ext.15554 Seat.1#3F049
>>>> We are the F.I.R.S.T.
>>>> Fast, Innovation, Reliability , Secure, Team
>>>>
>>>>
>>>>
>>>> -----邮件原件-----
>>>> 发件人: Atin Mukherjee [mailto:amukherj at redhat.com]
>>>> 发送时间: Tuesday, June 02, 2015 2:20 PM
>>>> 收件人: vyyy杨雨阳; Gluster-users at gluster.org
>>>> 主题: Re: 答复: [Gluster-users] Gluster peer rejected and failed to start
>>>>
>>>>
>>>>
>>>> On 06/02/2015 11:33 AM, vyyy杨雨阳 wrote:
>>>>> Actually I have 2 problems
>>>>> 1、New nodes can't add to the clusters
>>>>> I cleaned /var/lib/glusterd, now status is
>>>>> State: Accepted peer request (Connected)
>>>>>
>>>>> 2、One of clusterd nodes shown 'Peer rejected' and glusterd failed to start.
>>>>> The log is attached in pre-mail
>>>>> This is a product cluster, this problem is more egent
>>>> From the log I can clearly see the problematic node is
>>>> glusterfs09.sh2.ctripcorp.com. Your existing cluster configuration has
>>>> bricks hosted in glusterfs09.sh2.ctripcorp.com however the same is not
>>>> part of the cluster. Could you paste the output of gluster peer status
>>>> and gluster volume status ?
>>>>>
>>>>>
>>>>>
>>>>> Best Regards
>>>>> Yuyang Yang
>>>>>
>>>>> -----邮件原件-----
>>>>> 发件人: Atin Mukherjee [mailto:amukherj at redhat.com]
>>>>> 发送时间: Tuesday, June 02, 2015 12:52 PM
>>>>> 收件人: vyyy杨雨阳; Gluster-users at gluster.org
>>>>> 主题: Re: [Gluster-users] Gluster peer rejected and failed to start
>>>>>
>>>>>
>>>>>
>>>>> On 06/02/2015 10:00 AM, vyyy杨雨阳 wrote:
>>>>>> Hi
>>>>>>
>>>>>> We have a gluster (Version 3.6.3) cluster with 6 nodes, I tried to add 4 more nodes, but ‘Peer Rejected’, then I tried to resolve it by dump /var/lib/glusterd and probe again, not success, this is a question, But strange thing is:
>>>>>>
>>>>>> A node already in cluster also shown “Peer Reject”
>>>>>>
>>>>>> I tried to restart glusterd, It failed
>>>>>>
>>>>>> I found that /var/lib/glusterd/peers is empty, I copied the files from other nodes, still can’t start glusterd
>>>>> It seems like you are trying to peer probe nodes which are either either
>>>>> part of some other clusters (uncleaned nodes). Could you check whether
>>>>> the nodes which you are adding have empty /var/lib/glusterd? If not
>>>>> clean them and retry.
>>>>>
>>>>> ~Atin
>>>>>>
>>>>>>
>>>>>>
>>>>>> etc-glusterfs-glusterd.vol.log shown that cluster member as “unknown peer ”
>>>>>>
>>>>>> [2015-06-02 01:52:14.650635] C [glusterd-handler.c:2369:__glusterd_handle_friend_update] 0-: Received friend update request from unknown peer 04f22ee8-8e00-4c32-a924-b40a0e413aa6
>>>>>> [2015-06-02 01:52:14.650786] C [glusterd-handler.c:2369:__glusterd_handle_friend_update] 0-: Received friend update request from unknown peer 674a78b5-0590-48d4-8752-d4608832ed1d
>>>>>> [2015-06-02 01:52:14.657881] C [glusterd-handler.c:2369:__glusterd_handle_friend_update] 0-: Received friend update request from unknown peer 83e1a9db-3134-45e4-acd2-387b12b5b207
>>>>>> [2015-06-02 01:52:17.747865] W [glusterd-handler.c:697:__glusterd_handle_cluster_lock] 0-management: 04f22ee8-8e00-4c32-a924-b40a0e413aa6 doesn't belong to the cluster. Ignoring request.
>>>>>> [2015-06-02 01:52:17.747908] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
>>>>>> [2015-06-02 01:52:40.338885] W [glusterd-handler.c:697:__glusterd_handle_cluster_lock] 0-management: 674a78b5-0590-48d4-8752-d4608832ed1d doesn't belong to the cluster. Ignoring request.
>>>>>> [2015-06-02 01:52:40.338929] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
>>>>>> [2015-06-02 01:52:41.310451] W [glusterd-handler.c:697:__glusterd_handle_cluster_lock] 0-management: 674a78b5-0590-48d4-8752-d4608832ed1d doesn't belong to the cluster. Ignoring request.
>>>>>> [2015-06-02 01:52:41.310486] E [rpcsvc.c:544:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
>>>>>>
>>>>>>
>>>>>>
>>>>>> Debug info is as following,
>>>>>>
>>>>>>
>>>>>>
>>>>>> /usr/sbin/glusterd
>>>>>> [root at SH02SVR5951 peers]# /usr/sbin/glusterd --debug
>>>>>> [2015-06-02 04:09:24.626690] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.6.3 (args: /usr/sbin/glusterd --debug)
>>>>>> [2015-06-02 04:09:24.626739] D [logging.c:1763:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5
>>>>>> [2015-06-02 04:09:24.627052] D [MSGID: 0] [glusterfsd.c:613:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol
>>>>>> [2015-06-02 04:09:24.629683] I [glusterd.c:1214:init] 0-management: Maximum allowed open file descriptors set to 65536
>>>>>> [2015-06-02 04:09:24.629706] I [glusterd.c:1259:init] 0-management: Using /var/lib/glusterd as working directory
>>>>>> [2015-06-02 04:09:24.629764] D [glusterd.c:391:glusterd_rpcsvc_options_build] 0-: listen-backlog value: 128
>>>>>> [2015-06-02 04:09:24.629895] D [rpcsvc.c:2198:rpcsvc_init] 0-rpc-service: RPC service inited.
>>>>>> [2015-06-02 04:09:24.629904] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0
>>>>>> [2015-06-02 04:09:24.629930] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>>>>>> [2015-06-02 04:09:24.631989] D [socket.c:3807:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled
>>>>>> [2015-06-02 04:09:24.632005] D [socket.c:3810:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled
>>>>>> [2015-06-02 04:09:24.632013] D [socket.c:3827:socket_init] 0-socket.management: using system polling thread
>>>>>> [2015-06-02 04:09:24.632024] D [name.c:550:server_fill_address_family] 0-socket.management: option address-family not specified, defaulting to inet
>>>>>> [2015-06-02 04:09:24.632072] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/rdma.so
>>>>>> [2015-06-02 04:09:24.632102] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.6.3/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
>>>>>> [2015-06-02 04:09:24.632112] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
>>>>>> [2015-06-02 04:09:24.632122] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
>>>>>> [2015-06-02 04:09:24.632132] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, Ver: 2, Port: 0
>>>>>> [2015-06-02 04:09:24.632140] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: 1238463, Ver: 2, Port: 0
>>>>>> [2015-06-02 04:09:24.632148] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, Ver: 2, Port: 0
>>>>>> [2015-06-02 04:09:24.632155] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433, Ver: 3, Port: 0
>>>>>> [2015-06-02 04:09:24.632163] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: 1, Port: 0
>>>>>> [2015-06-02 04:09:24.632170] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0
>>>>>> [2015-06-02 04:09:24.632177] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 1239873, Ver: 1, Port: 0
>>>>>> [2015-06-02 04:09:24.632212] D [rpcsvc.c:2198:rpcsvc_init] 0-rpc-service: RPC service inited.
>>>>>> [2015-06-02 04:09:24.632220] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0
>>>>>> [2015-06-02 04:09:24.632236] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>>>>>> [2015-06-02 04:09:24.632302] D [socket.c:3692:socket_init] 0-socket.management: disabling nodelay
>>>>>> [2015-06-02 04:09:24.632314] D [socket.c:3807:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled
>>>>>> [2015-06-02 04:09:24.632321] D [socket.c:3810:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled
>>>>>> [2015-06-02 04:09:24.632329] D [socket.c:3827:socket_init] 0-socket.management: using system polling thread
>>>>>> [2015-06-02 04:09:24.632402] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0
>>>>>> [2015-06-02 04:09:24.632414] D [rpcsvc.c:1801:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), Num: 14398633, Ver: 2, Port: 0
>>>>>> [2015-06-02 04:09:24.632443] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>>>>>> [2015-06-02 04:09:24.632455] D [glusterd.c:1484:init] 0-management: cannot get run-with-valgrind value
>>>>>> [2015-06-02 04:09:24.633086] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system
>>>>>> [2015-06-02 04:09:24.633518] D [glusterd.c:446:glusterd_check_gsync_present] 0-glusterd: Returning -1
>>>>>> [2015-06-02 04:09:24.634031] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.634057] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.634115] D [store.c:304:gf_store_retrieve_value] 0-: key operating-version found
>>>>>> [2015-06-02 04:09:24.634141] I [glusterd-store.c:2043:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2
>>>>>> [2015-06-02 04:09:24.634196] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.634436] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.634446] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.634469] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634488] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634496] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2
>>>>>> [2015-06-02 04:09:24.634510] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634517] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 6
>>>>>> [2015-06-02 04:09:24.634528] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634534] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1
>>>>>> [2015-06-02 04:09:24.634544] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634551] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2
>>>>>> [2015-06-02 04:09:24.634562] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634569] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1
>>>>>> [2015-06-02 04:09:24.634580] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634587] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2
>>>>>> [2015-06-02 04:09:24.634597] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634603] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 24
>>>>>> [2015-06-02 04:09:24.634613] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634620] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0
>>>>>> [2015-06-02 04:09:24.634631] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634638] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = 7bf7d81a-f835-4c3b-9c58-4d4ad7ec6e07
>>>>>> [2015-06-02 04:09:24.634652] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634659] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = c7374917-2344-4c42-aeae-1d0bdb7ece0a
>>>>>> [2015-06-02 04:09:24.634678] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634689] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = 703c3b9d-4343-4546-9ca3-350aecdd52fe
>>>>>> [2015-06-02 04:09:24.634700] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634707] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2
>>>>>> [2015-06-02 04:09:24.634719] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634726] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2
>>>>>> [2015-06-02 04:09:24.634736] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634742] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.latency-measurement value = on
>>>>>> [2015-06-02 04:09:24.634773] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.latency-measurement,value:on
>>>>>> [2015-06-02 04:09:24.634785] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634791] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.count-fop-hits value = on
>>>>>> [2015-06-02 04:09:24.634803] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.count-fop-hits,value:on
>>>>>> [2015-06-02 04:09:24.634814] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634821] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.*
>>>>>> [2015-06-02 04:09:24.634838] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.*
>>>>>> [2015-06-02 04:09:24.634849] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634856] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs05.sh2.ctripcorp.com:-export-sdb-brick3
>>>>>> [2015-06-02 04:09:24.634868] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634874] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs06.sh2.ctripcorp.com:-export-sdb-brick3
>>>>>> [2015-06-02 04:09:24.634885] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634891] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick3
>>>>>> [2015-06-02 04:09:24.634902] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634908] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick3
>>>>>> [2015-06-02 04:09:24.634919] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634925] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-4 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick3
>>>>>> [2015-06-02 04:09:24.634935] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.634942] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-5 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick3
>>>>>> [2015-06-02 04:09:24.634954] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.634984] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635002] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.635019] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635030] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635039] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635048] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635056] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635066] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635074] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635083] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635092] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635101] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635110] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635118] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635127] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635136] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635144] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635153] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635161] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635368] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.635380] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.635396] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635416] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635426] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635436] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635447] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635456] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635465] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.635488] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.635501] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635748] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.635766] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.635786] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635807] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635818] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635828] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635838] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635847] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.635856] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.635909] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.635922] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636158] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.636167] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.636180] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636202] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636213] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636222] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636232] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636241] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636250] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.636274] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.636286] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636471] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.636479] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.636493] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636514] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636524] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636533] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636542] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636552] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636561] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.636586] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.636598] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636856] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.636889] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.636922] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636943] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636954] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636964] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636975] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636984] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.636994] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.637020] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.637036] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.637307] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.637325] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.637346] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.637367] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.637378] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.637388] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.819347] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.819374] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.819385] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.819413] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.819707] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>>>>>> [2015-06-02 04:09:24.820187] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.820222] D [store.c:304:gf_store_retrieve_value] 0-: key version found
>>>>>> [2015-06-02 04:09:24.820239] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.820532] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.820585] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>>>>>> [2015-06-02 04:09:24.820959] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.821064] D [store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.821090] D [store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.837290] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.837540] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.837551] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.837568] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.837589] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.837600] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.837862] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.837880] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.837896] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.837918] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.837929] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.837939] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.837964] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.838216] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.838257] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.838273] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838295] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838303] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2
>>>>>> [2015-06-02 04:09:24.838314] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838320] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 6
>>>>>> [2015-06-02 04:09:24.838330] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838336] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1
>>>>>> [2015-06-02 04:09:24.838346] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838352] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2
>>>>>> [2015-06-02 04:09:24.838361] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838367] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1
>>>>>> [2015-06-02 04:09:24.838377] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838383] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2
>>>>>> [2015-06-02 04:09:24.838393] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838399] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 6
>>>>>> [2015-06-02 04:09:24.838408] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838414] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0
>>>>>> [2015-06-02 04:09:24.838423] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838430] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = cdc279f4-6cd0-4087-8d2a-440bccb51e33
>>>>>> [2015-06-02 04:09:24.838443] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838449] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = 0b6d3496-0865-45e4-a45b-c221b22d3d92
>>>>>> [2015-06-02 04:09:24.838459] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838465] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = ecedfb4d-3308-4f6c-b188-4d5b868958e1
>>>>>> [2015-06-02 04:09:24.838475] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838482] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2
>>>>>> [2015-06-02 04:09:24.838493] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838499] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2
>>>>>> [2015-06-02 04:09:24.838508] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838515] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.68.*
>>>>>> [2015-06-02 04:09:24.838533] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.68.*
>>>>>> [2015-06-02 04:09:24.838545] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838552] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs05.sh2.ctripcorp.com:-export-sdb-brick2
>>>>>> [2015-06-02 04:09:24.838563] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838570] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs06.sh2.ctripcorp.com:-export-sdb-brick2
>>>>>> [2015-06-02 04:09:24.838580] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838587] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick2
>>>>>> [2015-06-02 04:09:24.838597] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838604] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick2
>>>>>> [2015-06-02 04:09:24.838614] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838621] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-4 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick2
>>>>>> [2015-06-02 04:09:24.838631] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838637] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-5 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick2
>>>>>> [2015-06-02 04:09:24.838649] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.838697] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838719] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.838738] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838747] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838756] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838764] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838772] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838781] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838797] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838806] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838815] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838823] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838832] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838840] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838848] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.838857] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839051] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.839063] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.839084] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839102] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839113] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839123] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839133] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839142] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839151] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.839177] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.839189] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839382] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.839390] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.839403] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839421] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839431] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839440] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839450] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839459] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839468] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.839491] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.839502] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839700] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.839712] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.839732] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839750] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839760] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839770] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839780] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.839799] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.839822] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.839834] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840057] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.840075] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.840096] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840117] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840128] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840137] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840147] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840156] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840166] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.840192] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.840206] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840402] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.840410] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.840424] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840440] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840450] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840459] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840469] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840478] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840487] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.840513] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.840525] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840799] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.840818] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.840839] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840859] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840871] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840880] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840891] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840900] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.840910] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.840931] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.841118] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>>>>>> [2015-06-02 04:09:24.841359] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.841396] D [store.c:304:gf_store_retrieve_value] 0-: key version found
>>>>>> [2015-06-02 04:09:24.841413] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.841557] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.841611] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>>>>>> [2015-06-02 04:09:24.841799] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.841875] D [store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.841901] D [store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.842350] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.842469] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.842477] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.842489] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842514] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842525] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.842636] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.842645] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.842656] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842696] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842708] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842718] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.842738] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.842848] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.842859] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.842872] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842891] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842898] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2
>>>>>> [2015-06-02 04:09:24.842908] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842914] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 4
>>>>>> [2015-06-02 04:09:24.842924] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842930] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1
>>>>>> [2015-06-02 04:09:24.842939] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842945] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2
>>>>>> [2015-06-02 04:09:24.842954] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842960] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1
>>>>>> [2015-06-02 04:09:24.842969] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842975] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2
>>>>>> [2015-06-02 04:09:24.842984] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.842990] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 3
>>>>>> [2015-06-02 04:09:24.842999] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843005] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0
>>>>>> [2015-06-02 04:09:24.843015] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843021] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = a3369f63-9fce-4a56-a921-f7206cb8d3ae
>>>>>> [2015-06-02 04:09:24.843033] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843039] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = c6012954-928a-43c1-b7ea-336d6ee0bea9
>>>>>> [2015-06-02 04:09:24.843050] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843056] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = bfbd7007-0e0c-45f9-8d59-05e1aee66728
>>>>>> [2015-06-02 04:09:24.843066] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843072] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2
>>>>>> [2015-06-02 04:09:24.843082] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843089] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2
>>>>>> [2015-06-02 04:09:24.843098] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843104] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.83.*
>>>>>> [2015-06-02 04:09:24.843119] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.83.*
>>>>>> [2015-06-02 04:09:24.843130] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843136] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick4
>>>>>> [2015-06-02 04:09:24.843147] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843154] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick4
>>>>>> [2015-06-02 04:09:24.843164] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843171] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick4
>>>>>> [2015-06-02 04:09:24.843181] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843188] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick4
>>>>>> [2015-06-02 04:09:24.843198] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.843221] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843238] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.843255] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843264] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843273] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843281] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843290] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843298] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843306] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843314] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843323] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843332] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843340] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843348] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843357] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843365] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843374] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843476] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.843484] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.843499] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843519] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843720] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843739] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843751] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843761] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843771] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.843802] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.843817] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.843984] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.843996] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.844013] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844030] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844041] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844050] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844060] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844069] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844078] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.844104] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.844116] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844222] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.844230] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.844243] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844266] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844276] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844285] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844294] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844303] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844312] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.844334] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.844346] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844443] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.844451] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.844468] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844487] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844497] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844506] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844516] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844525] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844534] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.844550] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.844753] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>>>>>> [2015-06-02 04:09:24.845000] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.845033] D [store.c:304:gf_store_retrieve_value] 0-: key version found
>>>>>> [2015-06-02 04:09:24.845049] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.845199] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.845244] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>>>>>> [2015-06-02 04:09:24.845396] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.845448] D [store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.845471] D [store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.846053] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.846195] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.846206] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.846220] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846241] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846252] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.846362] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.846371] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.846382] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846404] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846414] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846424] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.846442] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.846543] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.846552] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.846562] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846585] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846592] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2
>>>>>> [2015-06-02 04:09:24.846603] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846609] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 6
>>>>>> [2015-06-02 04:09:24.846618] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846624] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1
>>>>>> [2015-06-02 04:09:24.846633] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846639] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2
>>>>>> [2015-06-02 04:09:24.846648] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846654] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1
>>>>>> [2015-06-02 04:09:24.846663] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846690] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2
>>>>>> [2015-06-02 04:09:24.846702] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846708] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 3
>>>>>> [2015-06-02 04:09:24.846717] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846723] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0
>>>>>> [2015-06-02 04:09:24.846733] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846739] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = 8869b21b-1bf1-4eba-a52a-1158abde7d23
>>>>>> [2015-06-02 04:09:24.846752] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846758] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = 27298b95-919d-43f7-9507-8cbfbe5292e3
>>>>>> [2015-06-02 04:09:24.846769] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846775] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = 47f52166-020b-4bef-ada1-7e65fcd2834f
>>>>>> [2015-06-02 04:09:24.846785] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846792] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2
>>>>>> [2015-06-02 04:09:24.846802] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846808] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2
>>>>>> [2015-06-02 04:09:24.846818] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846824] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.count-fop-hits value = on
>>>>>> [2015-06-02 04:09:24.846837] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.count-fop-hits,value:on
>>>>>> [2015-06-02 04:09:24.846848] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846854] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = diagnostics.latency-measurement value = on
>>>>>> [2015-06-02 04:09:24.846864] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=diagnostics.latency-measurement,value:on
>>>>>> [2015-06-02 04:09:24.846875] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846881] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs05.sh2.ctripcorp.com:-export-sdb-brick
>>>>>> [2015-06-02 04:09:24.846892] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846899] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs06.sh2.ctripcorp.com:-export-sdb-brick
>>>>>> [2015-06-02 04:09:24.846910] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846916] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick
>>>>>> [2015-06-02 04:09:24.846927] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846934] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick
>>>>>> [2015-06-02 04:09:24.846944] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846952] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-4 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick
>>>>>> [2015-06-02 04:09:24.846962] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.846969] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-5 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick
>>>>>> [2015-06-02 04:09:24.846981] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.847003] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847027] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.847045] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847055] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847064] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847072] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847081] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847089] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847098] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847106] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847115] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847124] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847133] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847141] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847150] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847159] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847167] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847176] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847286] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.847294] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.847308] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847328] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847339] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847348] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847358] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847367] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847376] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.847402] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.847414] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847516] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.847523] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.847537] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847559] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847569] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847579] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847588] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847597] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847607] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.847628] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.847640] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847747] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.847756] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.847771] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847794] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847804] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847814] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847823] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847832] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847841] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.847862] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.847873] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.847969] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.847977] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.847990] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848012] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848022] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848031] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848040] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848049] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848058] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.848080] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.848091] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848187] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.848195] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.848208] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848230] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848240] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848249] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848259] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848267] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848277] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.848299] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.848310] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848411] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.848419] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.848432] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848455] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848465] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848474] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848484] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848493] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848503] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.848519] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.848727] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>>>>>> [2015-06-02 04:09:24.848975] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.849002] D [store.c:304:gf_store_retrieve_value] 0-: key version found
>>>>>> [2015-06-02 04:09:24.849019] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.849167] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.849212] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>>>>>> [2015-06-02 04:09:24.849367] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.849419] D [store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.849442] D [store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.849880] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.850003] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.850012] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.850024] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850043] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850054] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.850157] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.850166] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.850177] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850192] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850202] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850211] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.850228] D [glusterd-utils.c:513:glusterd_volinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.850402] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.850414] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.850426] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850444] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850451] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = type value = 2
>>>>>> [2015-06-02 04:09:24.850461] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850467] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = count value = 4
>>>>>> [2015-06-02 04:09:24.850476] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850482] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = status value = 1
>>>>>> [2015-06-02 04:09:24.850491] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850497] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = sub_count value = 2
>>>>>> [2015-06-02 04:09:24.850506] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850512] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1
>>>>>> [2015-06-02 04:09:24.850522] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850528] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = replica_count value = 2
>>>>>> [2015-06-02 04:09:24.850537] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850543] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = disperse_count value = 0
>>>>>> [2015-06-02 04:09:24.850552] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850558] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = redundancy_count value = 0
>>>>>> [2015-06-02 04:09:24.850567] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850573] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = version value = 11
>>>>>> [2015-06-02 04:09:24.850582] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850588] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = transport-type value = 0
>>>>>> [2015-06-02 04:09:24.850598] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850604] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = volume-id value = ea46609c-be9a-4283-8357-8fdc2e5a190e
>>>>>> [2015-06-02 04:09:24.850616] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850622] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = username value = 21b247ce-58ce-4087-8c32-3d578b4cc22d
>>>>>> [2015-06-02 04:09:24.850632] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850638] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = password value = 0724171a-c7ce-4528-b54b-2d983f969537
>>>>>> [2015-06-02 04:09:24.850648] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850654] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = op-version value = 2
>>>>>> [2015-06-02 04:09:24.850664] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850692] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2
>>>>>> [2015-06-02 04:09:24.850703] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850709] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = nfs.rpc-auth-allow value = 10.8.92.*,10.8.230.*,192.168.98.*
>>>>>> [2015-06-02 04:09:24.850724] D [glusterd-store.c:2766:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=nfs.rpc-auth-allow,value:10.8.92.*,10.8.230.*,192.168.98.*
>>>>>> [2015-06-02 04:09:24.850736] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850743] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-0 value = glusterfs07.sh2.ctripcorp.com:-export-sdb-brick1
>>>>>> [2015-06-02 04:09:24.850754] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850760] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-1 value = glusterfs08.sh2.ctripcorp.com:-export-sdb-brick1
>>>>>> [2015-06-02 04:09:24.850772] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850778] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-2 value = glusterfs09.sh2.ctripcorp.com:-export-sdb-brick1
>>>>>> [2015-06-02 04:09:24.850789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850795] D [glusterd-store.c:2642:glusterd_store_update_volinfo] 0-: key = brick-3 value = glusterfs10.sh2.ctripcorp.com:-export-sdb-brick1
>>>>>> [2015-06-02 04:09:24.850807] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.850830] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850850] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.850867] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850877] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850885] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850894] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850902] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850911] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850919] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850928] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850936] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850945] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850953] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850962] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850971] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850980] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850988] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.850996] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851005] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851164] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.851173] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.851187] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851203] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851213] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851223] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851232] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851242] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851251] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851261] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.851283] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.851295] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851457] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.851465] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.851478] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851493] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851504] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851513] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851522] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851531] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851540] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851549] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.851571] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.851583] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851732] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.851741] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.851755] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851770] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851780] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851799] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851807] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851816] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851826] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.851850] D [glusterd-utils.c:1019:glusterd_brickinfo_new] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.851863] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.851959] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.851974] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.852005] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.852026] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.852037] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.852047] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.852061] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.852070] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.852080] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.852090] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.852111] D [glusterd-store.c:2382:glusterd_store_retrieve_bricks] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.852298] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>>>>>> [2015-06-02 04:09:24.852524] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.852556] D [store.c:304:gf_store_retrieve_value] 0-: key version found
>>>>>> [2015-06-02 04:09:24.852576] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.852723] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.852776] D [glusterd-utils.c:2268:glusterd_volume_compute_cksum] 0-management: Returning with 0
>>>>>> [2015-06-02 04:09:24.852930] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.852982] D [store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.853006] D [store.c:372:gf_store_save_value] 0-: returning: 0
>>>>>> [2015-06-02 04:09:24.853412] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.853524] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.853539] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.853553] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.853573] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.853583] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.853699] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.853718] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.853735] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.853757] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.853768] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.853778] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.853787] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.853798] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.853820] D [glusterd-store.c:3094:glusterd_store_retrieve_volumes] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.853962] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.853972] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.853983] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854000] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854011] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>>>>>> [2015-06-02 04:09:24.854023] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854032] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854043] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0
>>>>>> [2015-06-02 04:09:24.854054] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.854070] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
>>>>>> [2015-06-02 04:09:24.854162] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.854171] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.854181] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854197] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854205] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>>>>>> [2015-06-02 04:09:24.854216] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854225] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854232] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0
>>>>>> [2015-06-02 04:09:24.854241] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.854255] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
>>>>>> [2015-06-02 04:09:24.854345] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.854354] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.854364] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854380] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854390] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>>>>>> [2015-06-02 04:09:24.854402] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854411] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854417] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0
>>>>>> [2015-06-02 04:09:24.854427] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.854439] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
>>>>>> [2015-06-02 04:09:24.854527] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.854536] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.854546] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854561] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854569] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>>>>>> [2015-06-02 04:09:24.854580] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854589] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854596] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0
>>>>>> [2015-06-02 04:09:24.854605] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.854617] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
>>>>>> [2015-06-02 04:09:24.854704] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.854713] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.854725] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854740] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854749] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>>>>>> [2015-06-02 04:09:24.854760] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854769] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.854776] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0
>>>>>> [2015-06-02 04:09:24.854789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.854803] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
>>>>>> [2015-06-02 04:09:24.855064] D [store.c:418:gf_store_handle_new] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.855072] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
>>>>>> [2015-06-02 04:09:24.855083] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.855098] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.855106] D [glusterd-utils.c:8326:glusterd_sm_tr_log_init] 0-: returning 0
>>>>>> [2015-06-02 04:09:24.855116] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.855125] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.855132] D [glusterd-peer-utils.c:422:glusterd_peer_hostname_new] 0-glusterd: Returning 0
>>>>>> [2015-06-02 04:09:24.855141] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
>>>>>> [2015-06-02 04:09:24.855154] I [glusterd-handler.c:3146:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
>>>>>> [2015-06-02 04:09:24.855186] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0
>>>>>> [2015-06-02 04:09:24.855210] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>>>>>> [2015-06-02 04:09:24.855218] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30
>>>>>> [2015-06-02 04:09:24.855231] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>>>>>> [2015-06-02 04:09:24.855324] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled
>>>>>> [2015-06-02 04:09:24.855333] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled
>>>>>> [2015-06-02 04:09:24.855340] D [socket.c:3827:socket_init] 0-management: using system polling thread
>>>>>> [2015-06-02 04:09:24.859610] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.211 (port-24007) for hostname: glusterfs07.sh2.ctripcorp.com and port: 24007
>>>>>> [2015-06-02 04:09:24.859628] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection
>>>>>> [2015-06-02 04:09:24.859707] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.859733] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0
>>>>>> [2015-06-02 04:09:24.859772] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>>>>>> [2015-06-02 04:09:24.859781] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30
>>>>>> [2015-06-02 04:09:24.859794] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>>>>>> [2015-06-02 04:09:24.859905] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled
>>>>>> [2015-06-02 04:09:24.859916] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled
>>>>>> [2015-06-02 04:09:24.859923] D [socket.c:3827:socket_init] 0-management: using system polling thread
>>>>>> [2015-06-02 04:09:24.863505] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.231.160 (port-24007) for hostname: glusterfs11.sh2.ctripcorp.com and port: 24007
>>>>>> [2015-06-02 04:09:24.863521] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection
>>>>>> [2015-06-02 04:09:24.863562] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.863585] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0
>>>>>> [2015-06-02 04:09:24.863607] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>>>>>> [2015-06-02 04:09:24.863615] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30
>>>>>> [2015-06-02 04:09:24.863625] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>>>>>> [2015-06-02 04:09:24.863754] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled
>>>>>> [2015-06-02 04:09:24.863765] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled
>>>>>> [2015-06-02 04:09:24.863772] D [socket.c:3827:socket_init] 0-management: using system polling thread
>>>>>> [2015-06-02 04:09:24.867255] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.210 (port-24007) for hostname: glusterfs06.sh2.ctripcorp.com and port: 24007
>>>>>> [2015-06-02 04:09:24.867271] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection
>>>>>> [2015-06-02 04:09:24.867314] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.867337] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0
>>>>>> [2015-06-02 04:09:24.867358] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>>>>>> [2015-06-02 04:09:24.867366] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30
>>>>>> [2015-06-02 04:09:24.867377] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>>>>>> [2015-06-02 04:09:24.867463] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled
>>>>>> [2015-06-02 04:09:24.867476] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled
>>>>>> [2015-06-02 04:09:24.867483] D [socket.c:3827:socket_init] 0-management: using system polling thread
>>>>>> [2015-06-02 04:09:24.870920] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.214 (port-24007) for hostname: glusterfs10.sh2.ctripcorp.com and port: 24007
>>>>>> [2015-06-02 04:09:24.870935] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection
>>>>>> [2015-06-02 04:09:24.870997] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.871021] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0
>>>>>> [2015-06-02 04:09:24.871042] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>>>>>> [2015-06-02 04:09:24.871050] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30
>>>>>> [2015-06-02 04:09:24.871061] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>>>>>> [2015-06-02 04:09:24.871129] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled
>>>>>> [2015-06-02 04:09:24.871138] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled
>>>>>> [2015-06-02 04:09:24.871144] D [socket.c:3827:socket_init] 0-management: using system polling thread
>>>>>> [2015-06-02 04:09:24.874102] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.209 (port-24007) for hostname: 10.8.230.209 and port: 24007
>>>>>> [2015-06-02 04:09:24.874116] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection
>>>>>> [2015-06-02 04:09:24.874159] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.874181] D [glusterd-handler.c:2984:glusterd_transport_inet_options_build] 0-glusterd: Returning 0
>>>>>> [2015-06-02 04:09:24.874202] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>>>>>> [2015-06-02 04:09:24.874210] D [rpc-clnt.c:981:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30
>>>>>> [2015-06-02 04:09:24.874219] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.3/rpc-transport/socket.so
>>>>>> [2015-06-02 04:09:24.874292] D [socket.c:3807:socket_init] 0-management: SSL support on the I/O path is NOT enabled
>>>>>> [2015-06-02 04:09:24.874301] D [socket.c:3810:socket_init] 0-management: SSL support for glusterd is NOT enabled
>>>>>> [2015-06-02 04:09:24.874326] D [socket.c:3827:socket_init] 0-management: using system polling thread
>>>>>> [2015-06-02 04:09:24.877759] D [common-utils.c:253:gf_resolve_ip6] 0-resolver: returning ip-10.8.230.212 (port-24007) for hostname: glusterfs08.sh2.ctripcorp.com and port: 24007
>>>>>> [2015-06-02 04:09:24.877774] D [socket.c:2826:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection
>>>>>> [2015-06-02 04:09:24.877818] D [glusterd-handler.c:2923:glusterd_rpc_create] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.877831] D [glusterd-store.c:4106:glusterd_store_retrieve_peers] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.877863] I [glusterd-store.c:3497:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list.
>>>>>> [2015-06-02 04:09:24.877873] D [glusterd-store.c:3612:glusterd_store_retrieve_snaps] 0-: Returning with 0
>>>>>> [2015-06-02 04:09:24.881125] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.881140] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.881148] D [glusterd-peer-utils.c:600:gd_peerinfo_find_from_hostname] 0-management: Friend glusterfs06.sh2.ctripcorp.com found.. state: 3
>>>>>> [2015-06-02 04:09:24.881156] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.881162] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.881169] D [glusterd-peer-utils.c:600:gd_peerinfo_find_from_hostname] 0-management: Friend glusterfs07.sh2.ctripcorp.com found.. state: 3
>>>>>> [2015-06-02 04:09:24.881176] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.881182] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.881188] D [glusterd-peer-utils.c:600:gd_peerinfo_find_from_hostname] 0-management: Friend glusterfs08.sh2.ctripcorp.com found.. state: 3
>>>>>> [2015-06-02 04:09:24.881195] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning 0
>>>>>> [2015-06-02 04:09:24.881201] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning 0
>>>>>> [2015-06-02 04:09:24.890739] D [glusterd-peer-utils.c:121:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: glusterfs09.sh2.ctripcorp.com
>>>>>> [2015-06-02 04:09:24.891607] D [common-utils.c:2897:gf_is_local_addr] 0-management: 10.8.230.213
>>>>>> [2015-06-02 04:09:24.891750] D [common-utils.c:2897:gf_is_local_addr] 0-management: 10.8.230.213
>>>>>> [2015-06-02 04:09:24.891820] D [common-utils.c:2897:gf_is_local_addr] 0-management: 10.8.230.213
>>>>>> [2015-06-02 04:09:24.891881] D [common-utils.c:2913:gf_is_local_addr] 0-management: glusterfs09.sh2.ctripcorp.com is not local
>>>>>> [2015-06-02 04:09:24.891890] D [glusterd-peer-utils.c:156:glusterd_hostname_to_uuid] 0-management: returning -1
>>>>>> [2015-06-02 04:09:24.891898] D [glusterd-utils.c:1059:glusterd_resolve_brick] 0-management: Returning -1
>>>>>> [2015-06-02 04:09:24.891904] E [glusterd-store.c:4244:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore
>>>>>> [2015-06-02 04:09:24.891911] D [glusterd-store.c:4321:glusterd_restore] 0-: Returning -1
>>>>>> [2015-06-02 04:09:24.891923] E [xlator.c:425:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again
>>>>>> [2015-06-02 04:09:24.891933] E [graph.c:322:glusterfs_graph_init] 0-management: initializing translator failed
>>>>>> [2015-06-02 04:09:24.891941] E [graph.c:525:glusterfs_graph_activate] 0-graph: init failed
>>>>>> [2015-06-02 04:09:24.892164] D [logging.c:1740:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size reduced. About to flush 2 extra log messages
>>>>>> [2015-06-02 04:09:24.892178] D [logging.c:1743:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 2 extra log messages
>>>>>> [2015-06-02 04:09:24.892325] W [glusterfsd.c:1194:cleanup_and_exit] (--> 0-: received signum (0), shutting down
>>>>>> [2015-06-02 04:09:24.892345] D [glusterfsd-mgmt.c:2244:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given
>>>>>> [root at SH02SVR5951 peers]#
>>>>>>
>>>>>>
>>>>>>
>>>>>> Best Regards
>>>>>> Yuyang Yang
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>
>>>>>
>>>>
>>>
>>
>
--
~Atin
More information about the Gluster-users
mailing list