[Bugs] [Bug 1162479] replace-brick doesn't work fine .
bugzilla at redhat.com
bugzilla at redhat.com
Fri Nov 14 05:25:13 UTC 2014
https://bugzilla.redhat.com/show_bug.cgi?id=1162479
wangqy <wangqy8848 at 163.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Flags|needinfo?(wangqy8848 at 163.co |
|m) |
--- Comment #2 from wangqy <wangqy8848 at 163.com> ---
(In reply to Atin Mukherjee from comment #1)
> If glusterd is crashed, then till you restart the service cli requests will
> get timed out. What we need to look at here is why glusterd crashed. Can you
> share the sequence of glusterd logs during replace brick transaction, that
> would help in root causing the problem.
I try it again and get the gluster log by add --log-level=DEBUG into
/etc/init.d/glusterd. And here is the debug log of glusterd:
[2014-11-14 03:33:59.116911] I
[glusterd-replace-brick.c:98:__glusterd_handle_replace_brick] 0-management:
Received replace brick req
[2014-11-14 03:33:59.116939] D
[glusterd-replace-brick.c:140:__glusterd_handle_replace_brick] 0-management:
src brick=data-node3:/brick/testrep
[2014-11-14 03:33:59.116946] D
[glusterd-replace-brick.c:151:__glusterd_handle_replace_brick] 0-management:
dst brick=data-node5:/brick/testrep
[2014-11-14 03:33:59.116952] I
[glusterd-replace-brick.c:153:__glusterd_handle_replace_brick] 0-management:
Received replace brick start request
[2014-11-14 03:33:59.116962] D [glusterd-utils.c:161:glusterd_lock]
0-management: Cluster lock held by 0f05057d-4ff1-4443-b35d-7db852b493fb
[2014-11-14 03:33:59.116969] D [glusterd-handler.c:617:glusterd_op_txn_begin]
0-management: Acquired lock on localhost
[2014-11-14 03:33:59.116976] D
[glusterd-op-sm.c:5640:glusterd_op_sm_inject_event] 0-management: Enqueue
event: 'GD_OP_EVENT_START_LOCK'
[2014-11-14 03:33:59.116982] D [glusterd-handler.c:635:glusterd_op_txn_begin]
0-management: Returning 0
[2014-11-14 03:33:59.116989] D [glusterd-op-sm.c:5717:glusterd_op_sm]
0-management: Dequeued event of type: 'GD_OP_EVENT_START_LOCK'
[2014-11-14 03:33:59.117052] D [glusterd-rpc-ops.c:1214:glusterd_cluster_lock]
0-management: Returning 0
[2014-11-14 03:33:59.117072] D [glusterd-rpc-ops.c:1214:glusterd_cluster_lock]
0-management: Returning 0
[2014-11-14 03:33:59.117079] D [glusterd-op-sm.c:2412:glusterd_op_ac_send_lock]
0-management: Returning with 0
[2014-11-14 03:33:59.117086] D
[glusterd-utils.c:6111:glusterd_sm_tr_log_transition_add] 0-management:
Transitioning from 'Default' to 'Lock sent' due to event
'GD_OP_EVENT_START_LOCK'
[2014-11-14 03:33:59.117092] D
[glusterd-utils.c:6113:glusterd_sm_tr_log_transition_add] 0-management:
returning 0
[2014-11-14 03:33:59.118719] D
[glusterd-rpc-ops.c:602:__glusterd_cluster_lock_cbk] 0-management: Received
lock ACC from uuid: dcaa270d-d46f-43c2-b9a2-ad60197fd912
[2014-11-14 03:33:59.118767] D
[glusterd-utils.c:5439:glusterd_friend_find_by_uuid] 0-management: Friend
found... state: Peer in Cluster
[2014-11-14 03:33:59.118776] D
[glusterd-op-sm.c:5640:glusterd_op_sm_inject_event] 0-management: Enqueue
event: 'GD_OP_EVENT_RCVD_ACC'
[2014-11-14 03:33:59.118784] D [glusterd-op-sm.c:5717:glusterd_op_sm]
0-management: Dequeued event of type: 'GD_OP_EVENT_RCVD_ACC'
[2014-11-14 03:33:59.118791] D
[glusterd-utils.c:6111:glusterd_sm_tr_log_transition_add] 0-management:
Transitioning from 'Lock sent' to 'Lock sent' due to event
'GD_OP_EVENT_RCVD_ACC'
[2014-11-14 03:33:59.118812] D
[glusterd-utils.c:6113:glusterd_sm_tr_log_transition_add] 0-management:
returning 0
[2014-11-14 03:33:59.118891] D
[glusterd-rpc-ops.c:602:__glusterd_cluster_lock_cbk] 0-management: Received
lock ACC from uuid: bac58735-7d09-4d5c-8187-eedbdbdf30a7
[2014-11-14 03:33:59.118900] D
[glusterd-utils.c:5439:glusterd_friend_find_by_uuid] 0-management: Friend
found... state: Peer in Cluster
[2014-11-14 03:33:59.118907] D
[glusterd-op-sm.c:5640:glusterd_op_sm_inject_event] 0-management: Enqueue
event: 'GD_OP_EVENT_RCVD_ACC'
[2014-11-14 03:33:59.118913] D [glusterd-op-sm.c:5717:glusterd_op_sm]
0-management: Dequeued event of type: 'GD_OP_EVENT_RCVD_ACC'
[2014-11-14 03:33:59.118918] D
[glusterd-op-sm.c:5640:glusterd_op_sm_inject_event] 0-management: Enqueue
event: 'GD_OP_EVENT_ALL_ACC'
[2014-11-14 03:33:59.118924] D
[glusterd-op-sm.c:2572:glusterd_op_ac_rcvd_lock_acc] 0-management: Returning 0
[2014-11-14 03:33:59.118930] D
[glusterd-utils.c:6111:glusterd_sm_tr_log_transition_add] 0-management:
Transitioning from 'Lock sent' to 'Lock sent' due to event
'GD_OP_EVENT_RCVD_ACC'
[2014-11-14 03:33:59.118935] D
[glusterd-utils.c:6113:glusterd_sm_tr_log_transition_add] 0-management:
returning 0
[2014-11-14 03:33:59.118941] D [glusterd-op-sm.c:5717:glusterd_op_sm]
0-management: Dequeued event of type: 'GD_OP_EVENT_ALL_ACC'
[2014-11-14 03:33:59.118953] D [glusterd-utils.c:1156:glusterd_volinfo_find]
0-management: Volume testrep found
[2014-11-14 03:33:59.118960] D [glusterd-utils.c:1163:glusterd_volinfo_find]
0-management: Returning 0
[2014-11-14 03:33:59.118978] D [glusterd-utils.c:1156:glusterd_volinfo_find]
0-management: Volume testrep found
[2014-11-14 03:33:59.118984] D [glusterd-utils.c:1163:glusterd_volinfo_find]
0-management: Returning 0
[2014-11-14 03:33:59.118993] D
[glusterd-replace-brick.c:238:glusterd_op_stage_replace_brick] 0-management:
src brick=data-node3:/brick/testrep
[2014-11-14 03:33:59.119012] D
[glusterd-replace-brick.c:247:glusterd_op_stage_replace_brick] 0-management:
dst brick=data-node5:/brick/testrep
[2014-11-14 03:33:59.119021] D [glusterd-utils.c:1156:glusterd_volinfo_find]
0-management: Volume testrep found
014-11-14 03:33:59.119027] D [glusterd-utils.c:1163:glusterd_volinfo_find]
0-management: Returning 0
[2014-11-14 03:33:59.119042] D [glusterd-utils.c:669:glusterd_brickinfo_new]
0-management: Returning 0
[2014-11-14 03:33:59.119062] D
[glusterd-utils.c:743:glusterd_brickinfo_new_from_brick] 0-management:
Returning 0
[2014-11-14 03:33:59.119071] D [glusterd-utils.c:518:glusterd_volinfo_new]
0-management: Returning 0
[2014-11-14 03:33:59.119097] D
[glusterd-utils.c:602:glusterd_volume_brickinfos_delete] 0-management:
Returning 0
[2014-11-14 03:33:59.119109] D [store.c:434:gf_store_handle_destroy] 0-:
Returning 0
[2014-11-14 03:33:59.119115] D [glusterd-utils.c:643:glusterd_volinfo_delete]
0-management: Returning 0
[2014-11-14 03:33:59.119123] D [glusterd-utils.c:669:glusterd_brickinfo_new]
0-management: Returning 0
[2014-11-14 03:33:59.119129] D
[glusterd-utils.c:743:glusterd_brickinfo_new_from_brick] 0-management:
Returning 0
[2014-11-14 03:33:59.119136] D [glusterd-utils.c:518:glusterd_volinfo_new]
0-management: Returning 0
[2014-11-14 03:33:59.119143] D
[glusterd-utils.c:602:glusterd_volume_brickinfos_delete] 0-management:
Returning 0
[2014-11-14 03:33:59.119149] D [store.c:434:gf_store_handle_destroy] 0-:
Returning 0
[2014-11-14 03:33:59.119154] D [glusterd-utils.c:643:glusterd_volinfo_delete]
0-management: Returning 0
[2014-11-14 03:33:59.119163] D [glusterd-utils.c:5726:glusterd_is_rb_started]
0-: is_rb_started:status=0
[2014-11-14 03:33:59.119187] I
[glusterd-utils.c:8850:glusterd_generate_and_set_task_id] 0-management:
Generated task-id ee10c393-3ece-4928-9563-642a4bc637fc for key replace-brick-id
[2014-11-14 03:33:59.119197] D [glusterd-utils.c:669:glusterd_brickinfo_new]
0-management: Returning 0
[2014-11-14 03:33:59.119203] D
[glusterd-utils.c:743:glusterd_brickinfo_new_from_brick] 0-management:
Returning 0
[2014-11-14 03:33:59.119214] D
[glusterd-utils.c:5483:glusterd_friend_find_by_hostname] 0-management: Friend
data-node3 found.. state: 3
[2014-11-14 03:33:59.119220] D
[glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management: returning 0
[2014-11-14 03:33:59.119226] D
[glusterd-utils.c:1026:glusterd_volume_brickinfo_get] 0-management: Found brick
data-node3:/brick/testrep in volume testrep
[2014-11-14 03:33:59.119232] D
[glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management: Returning 0
[2014-11-14 03:33:59.119237] D
[glusterd-utils.c:1059:glusterd_volume_brickinfo_get_by_brick] 0-: Returning 0
[2014-11-14 03:33:59.119342] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.153
[2014-11-14 03:33:59.119419] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.153
[2014-11-14 03:33:59.119455] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.153
[2014-11-14 03:33:59.119498] D [common-utils.c:2946:gf_is_local_addr]
0-management: data-node3 is not local
[2014-11-14 03:33:59.119510] D [glusterd-utils.c:669:glusterd_brickinfo_new]
0-management: Returning 0
[2014-11-14 03:33:59.119517] D
[glusterd-utils.c:743:glusterd_brickinfo_new_from_brick] 0-management:
Returning 0
[2014-11-14 03:33:59.119524] D
[glusterd-utils.c:5483:glusterd_friend_find_by_hostname] 0-management: Friend
data-node5 found.. state: 3
[2014-11-14 03:33:59.119530] D
[glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management: returning 0
[2014-11-14 03:33:59.119535] D [glusterd-utils.c:685:glusterd_resolve_brick]
0-management: Returning 0
[2014-11-14 03:33:59.119541] D
[glusterd-utils.c:5439:glusterd_friend_find_by_uuid] 0-management: Friend
found... state: Peer in Cluster
[2014-11-14 03:33:59.119547] D
[glusterd-utils.c:5718:glusterd_new_brick_validate] 0-management: returning 0
[2014-11-14 03:33:59.119553] D [glusterd-utils.c:5726:glusterd_is_rb_started]
0-: is_rb_started:status=0
[2014-11-14 03:33:59.119558] D [glusterd-utils.c:5735:glusterd_is_rb_paused]
0-: is_rb_paused:status=0
[2014-11-14 03:33:59.119564] D [glusterd-utils.c:5726:glusterd_is_rb_started]
0-: is_rb_started:status=0
[2014-11-14 03:33:59.119569] D [glusterd-utils.c:5735:glusterd_is_rb_paused]
0-: is_rb_paused:status=0
[2014-11-14 03:33:59.119605] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.155
[2014-11-14 03:33:59.119641] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.155
[2014-11-14 03:33:59.119673] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.155
[2014-11-14 03:33:59.119704] D [common-utils.c:2946:gf_is_local_addr]
0-management: data-node5 is not local
[2014-11-14 03:33:59.119739] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.155
[2014-11-14 03:33:59.119774] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.155
[2014-11-14 03:33:59.119807] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.155
[2014-11-14 03:33:59.119838] D [common-utils.c:2946:gf_is_local_addr]
0-management: data-node5 is not local
[2014-11-14 03:33:59.119846] D
[glusterd-utils.c:5483:glusterd_friend_find_by_hostname] 0-management: Friend
data-node5 found.. state: 3
[2014-11-14 03:33:59.119915] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.155
[2014-11-14 03:33:59.119946] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.155
[2014-11-14 03:33:59.119978] D [common-utils.c:2946:gf_is_local_addr]
0-management: data-node5 is not local
[2014-11-14 03:33:59.119985] D
[glusterd-replace-brick.c:573:glusterd_op_stage_replace_brick] 0-management:
Returning 0
[2014-11-14 03:33:59.119991] D
[glusterd-op-sm.c:4151:glusterd_op_stage_validate] 0-management: OP = 10.
Returning 0
[2014-11-14 03:33:59.120040] D [glusterd-rpc-ops.c:1306:glusterd_stage_op]
0-management: Returning 0
[2014-11-14 03:33:59.120061] D [glusterd-rpc-ops.c:1306:glusterd_stage_op]
0-management: Returning 0
[2014-11-14 03:33:59.120069] D
[glusterd-op-sm.c:3008:glusterd_op_ac_send_stage_op] 0-management: Sent stage
op request for 'Volume Replace brick' to 2 peers
[2014-11-14 03:33:59.120078] D
[glusterd-op-sm.c:3013:glusterd_op_ac_send_stage_op] 0-management: Returning
with 0
[2014-11-14 03:33:59.120085] D
[glusterd-utils.c:6111:glusterd_sm_tr_log_transition_add] 0-management:
Transitioning from 'Lock sent' to 'Stage op sent' due to event
'GD_OP_EVENT_ALL_ACC'
[2014-11-14 03:33:59.120091] D
[glusterd-utils.c:6113:glusterd_sm_tr_log_transition_add] 0-management:
returning 0
[2014-11-14 03:33:59.121626] D [glusterd-rpc-ops.c:773:__glusterd_stage_op_cbk]
0-management: Received stage ACC from uuid:
dcaa270d-d46f-43c2-b9a2-ad60197fd912
[2014-11-14 03:33:59.121678] D
[glusterd-utils.c:5439:glusterd_friend_find_by_uuid] 0-management: Friend
found... state: Peer in Cluster
[2014-11-14 03:33:59.121692] D [glusterd-utils.c:7542:glusterd_rb_use_rsp_dict]
0-: src-brick-port=49152 found
[2014-11-14 03:33:59.121704] D
[glusterd-op-sm.c:5640:glusterd_op_sm_inject_event] 0-management: Enqueue
event: 'GD_OP_EVENT_RCVD_ACC'
[2014-11-14 03:33:59.121715] D [glusterd-op-sm.c:5717:glusterd_op_sm]
0-management: Dequeued event of type: 'GD_OP_EVENT_RCVD_ACC'
[2014-11-14 03:33:59.121722] D
[glusterd-op-sm.c:3581:glusterd_op_ac_rcvd_stage_op_acc] 0-management:
Returning 0
[2014-11-14 03:33:59.121729] D
[glusterd-utils.c:6111:glusterd_sm_tr_log_transition_add] 0-management:
Transitioning from 'Stage op sent' to 'Stage op sent' due to event
'GD_OP_EVENT_RCVD_ACC'
[2014-11-14 03:33:59.121736] D
[glusterd-utils.c:6113:glusterd_sm_tr_log_transition_add] 0-management:
returning 0
[2014-11-14 03:33:59.134657] D [glusterd-rpc-ops.c:773:__glusterd_stage_op_cbk]
0-management: Received stage ACC from uuid:
bac58735-7d09-4d5c-8187-eedbdbdf30a7
[2014-11-14 03:33:59.134692] D
[glusterd-utils.c:5439:glusterd_friend_find_by_uuid] 0-management: Friend
found... state: Peer in Cluster
[2014-11-14 03:33:59.134703] D [glusterd-utils.c:7548:glusterd_rb_use_rsp_dict]
0-: dst-brick-port=49153 found
[2014-11-14 03:33:59.134713] D
[glusterd-op-sm.c:5640:glusterd_op_sm_inject_event] 0-management: Enqueue
event: 'GD_OP_EVENT_RCVD_ACC'
[2014-11-14 03:33:59.134720] D [glusterd-op-sm.c:5717:glusterd_op_sm]
0-management: Dequeued event of type: 'GD_OP_EVENT_RCVD_ACC'
[2014-11-14 03:33:59.134726] D
[glusterd-op-sm.c:5640:glusterd_op_sm_inject_event] 0-management: Enqueue
event: 'GD_OP_EVENT_STAGE_ACC'
[2014-11-14 03:33:59.134731] D
[glusterd-op-sm.c:3581:glusterd_op_ac_rcvd_stage_op_acc] 0-management:
Returning 0
[2014-11-14 03:33:59.134738] D
[glusterd-utils.c:6111:glusterd_sm_tr_log_transition_add] 0-management:
Transitioning from 'Stage op sent' to 'Stage op sent' due to event
'GD_OP_EVENT_RCVD_ACC'
[2014-11-14 03:33:59.134743] D
[glusterd-utils.c:6113:glusterd_sm_tr_log_transition_add] 0-management:
returning 0
[2014-11-14 03:33:59.134749] D [glusterd-op-sm.c:5717:glusterd_op_sm]
0-management: Dequeued event of type: 'GD_OP_EVENT_STAGE_ACC'
[2014-11-14 03:33:59.134758] D [glusterd-utils.c:1156:glusterd_volinfo_find]
0-management: Volume testrep found
[2014-11-14 03:33:59.134764] D [glusterd-utils.c:1163:glusterd_volinfo_find]
0-management: Returning 0
[2014-11-14 03:33:59.134777] D
[glusterd-op-sm.c:5307:glusterd_op_bricks_select] 0-management: Returning 0
[2014-11-14 03:33:59.134783] D [glusterd-rpc-ops.c:1585:glusterd_brick_op]
0-management: Sent brick op req for operation 'Volume Replace brick' to 0
bricks
[2014-11-14 03:33:59.134789] D [glusterd-rpc-ops.c:1593:glusterd_brick_op]
0-management: Returning 0
[2014-11-14 03:33:59.134795] D
[glusterd-op-sm.c:5640:glusterd_op_sm_inject_event] 0-management: Enqueue
event: 'GD_OP_EVENT_ALL_ACK'
[2014-11-14 03:33:59.134800] D
[glusterd-op-sm.c:5200:glusterd_op_ac_send_brick_op] 0-management: Returning
with 0
[2014-11-14 03:33:59.134806] D
[glusterd-utils.c:6111:glusterd_sm_tr_log_transition_add] 0-management:
Transitioning from 'Stage op sent' to 'Brick op sent' due to event
'GD_OP_EVENT_STAGE_ACC'
[2014-11-14 03:33:59.134811] D
[glusterd-utils.c:6113:glusterd_sm_tr_log_transition_add] 0-management:
returning 0
[2014-11-14 03:33:59.134829] D [glusterd-op-sm.c:5717:glusterd_op_sm]
0-management: Dequeued event of type: 'GD_OP_EVENT_ALL_ACK'
[2014-11-14 03:33:59.134836] D [glusterd-utils.c:1156:glusterd_volinfo_find]
0-management: Volume testrep found
[2014-11-14 03:33:59.134841] D [glusterd-utils.c:1163:glusterd_volinfo_find]
0-management: Returning 0
[2014-11-14 03:33:59.134858] D
[glusterd-replace-brick.c:1571:glusterd_op_replace_brick] 0-management: src
brick=data-node3:/brick/testrep
[2014-11-14 03:33:59.134865] D
[glusterd-replace-brick.c:1579:glusterd_op_replace_brick] 0-management: dst
brick=data-node5:/brick/testrep
[2014-11-14 03:33:59.134871] D [glusterd-utils.c:1156:glusterd_volinfo_find]
0-management: Volume testrep found
[2014-11-14 03:33:59.134876] D [glusterd-utils.c:1163:glusterd_volinfo_find]
0-management: Returning 0
[2014-11-14 03:33:59.134893] D
[glusterd-utils.c:743:glusterd_brickinfo_new_from_brick] 0-management:
Returning 0
[2014-11-14 03:33:59.134900] D
[glusterd-utils.c:5483:glusterd_friend_find_by_hostname] 0-management: Friend
data-node3 found.. state: 3
[2014-11-14 03:33:59.134906] D
[glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management: returning 0
[2014-11-14 03:33:59.134912] D
[glusterd-utils.c:1026:glusterd_volume_brickinfo_get] 0-management: Found brick
data-node3:/brick/testrep in volume testrep
[2014-11-14 03:33:59.134917] D
[glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management: Returning 0
[2014-11-14 03:33:59.134923] D
[glusterd-utils.c:1059:glusterd_volume_brickinfo_get_by_brick] 0-: Returning 0
[2014-11-14 03:33:59.134930] D
[glusterd-utils.c:5483:glusterd_friend_find_by_hostname] 0-management: Friend
data-node5 found.. state: 3
[2014-11-14 03:33:59.134936] D
[glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management: returning 0
[2014-11-14 03:33:59.134941] D [glusterd-utils.c:685:glusterd_resolve_brick]
0-management: Returning 0
[2014-11-14 03:33:59.135057] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.153
[2014-11-14 03:33:59.135112] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.153
[2014-11-14 03:33:59.135145] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.153
[2014-11-14 03:33:59.135176] D [common-utils.c:2946:gf_is_local_addr]
0-management: data-node3 is not local
[2014-11-14 03:33:59.135214] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.155
[2014-11-14 03:33:59.135248] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.155
[2014-11-14 03:33:59.135280] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.155
[2014-11-14 03:33:59.135311] D [common-utils.c:2946:gf_is_local_addr]
0-management: data-node5 is not local
[2014-11-14 03:33:59.135349] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.155
[2014-11-14 03:33:59.135409] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.155
[2014-11-14 03:33:59.135442] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.155
[2014-11-14 03:33:59.135485] D [common-utils.c:2946:gf_is_local_addr]
0-management: data-node5 is not local
[2014-11-14 03:33:59.135531] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.153
[2014-11-14 03:33:59.135566] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.153
[2014-11-14 03:33:59.135598] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.153
[2014-11-14 03:33:59.135628] D [common-utils.c:2946:gf_is_local_addr]
0-management: data-node3 is not local
[2014-11-14 03:33:59.135664] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.155
[2014-11-14 03:33:59.135698] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.155
[2014-11-14 03:33:59.135729] D [common-utils.c:2930:gf_is_local_addr]
0-management: 192.168.1.155
[2014-11-14 03:33:59.135759] D [common-utils.c:2946:gf_is_local_addr]
0-management: data-node5 is not local
[2014-11-14 03:33:59.135770] D [glusterd-utils.c:5746:glusterd_set_rb_status]
0-: setting status from 0 to 1
[2014-11-14 03:33:59.135783] D
[glusterd-store.c:629:glusterd_store_create_volume_dir] 0-management: Returning
with 0
[2014-11-14 03:33:59.164271] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.189402] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.214469] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.239646] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.264680] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.289899] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.315100] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.340133] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.365339] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.382124] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.398932] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.415818] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.432559] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.432600] D
[glusterd-store.c:653:glusterd_store_volinfo_write] 0-management: Returning 0
[2014-11-14 03:33:59.449350] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.466321] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.483142] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.508273] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.533441] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.550232] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.550275] D
[glusterd-store.c:250:glusterd_store_brickinfo_write] 0-management: Returning 0
[2014-11-14 03:33:59.550290] D
[glusterd-store.c:276:glusterd_store_perform_brick_store] 0-management:
Returning 0
[2014-11-14 03:33:59.550296] D [glusterd-store.c:306:glusterd_store_brickinfo]
0-management: Returning with 0
[2014-11-14 03:33:59.567061] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.584035] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.600833] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.617505] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.634455] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.651231] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.651272] D
[glusterd-store.c:250:glusterd_store_brickinfo_write] 0-management: Returning 0
[2014-11-14 03:33:59.651286] D
[glusterd-store.c:276:glusterd_store_perform_brick_store] 0-management:
Returning 0
[2014-11-14 03:33:59.651293] D [glusterd-store.c:306:glusterd_store_brickinfo]
0-management: Returning with 0
[2014-11-14 03:33:59.651298] D [glusterd-store.c:793:glusterd_store_brickinfos]
0-management: Returning 0
[2014-11-14 03:33:59.651305] D
[glusterd-store.c:993:glusterd_store_perform_volume_store] 0-management:
Returning 0
[2014-11-14 03:33:59.835933] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.919508] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.945196] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.970424] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.995582] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:33:59.995624] D
[glusterd-store.c:852:glusterd_store_rbstate_write] 0-management: Returning 0
[2014-11-14 03:34:00.012546] D
[glusterd-store.c:882:glusterd_store_perform_rbstate_store] 0-management:
Returning 0
[2014-11-14 03:34:00.037861] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:34:00.062935] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:34:00.088125] D [store.c:348:gf_store_save_value] 0-: returning:
0
[2014-11-14 03:34:00.088176] D
[glusterd-store.c:933:glusterd_store_node_state_write] 0-management: Returning
0
[2014-11-14 03:34:00.105106] D
[glusterd-store.c:963:glusterd_store_perform_node_state_store] 0-management:
Returning 0
[2014-11-14 03:34:00.105354] D
[glusterd-utils.c:1784:glusterd_volume_compute_cksum] 0-management: Returning
with 0
[2014-11-14 03:34:00.105364] D [glusterd-store.c:1143:glusterd_store_volinfo]
0-management: Returning 0
[2014-11-14 03:34:00.105394] D
[glusterd-op-sm.c:4266:glusterd_op_commit_perform] 0-management: Returning 0
[2014-11-14 03:34:00.105488] D [glusterd-rpc-ops.c:1362:glusterd_commit_op]
0-management: Returning 0
[2014-11-14 03:34:00.105512] D [glusterd-rpc-ops.c:1362:glusterd_commit_op]
0-management: Returning 0
[2014-11-14 03:34:00.105519] D
[glusterd-op-sm.c:3538:glusterd_op_ac_send_commit_op] 0-management: Sent commit
op req for 'Volume Replace brick' to 2 peers
[2014-11-14 03:34:00.105527] D
[glusterd-op-sm.c:3559:glusterd_op_ac_send_commit_op] 0-management: Returning
with 0
[2014-11-14 03:34:00.105534] D
[glusterd-utils.c:6111:glusterd_sm_tr_log_transition_add] 0-management:
Transitioning from 'Brick op sent' to 'Commit op sent' due to event
'GD_OP_EVENT_ALL_ACK'
[2014-11-14 03:34:00.105540] D
[glusterd-utils.c:6113:glusterd_sm_tr_log_transition_add] 0-management:
returning 0
[2014-11-14 03:34:00.109741] D [socket.c:492:__socket_rwv] 0-socket.management:
EOF on socket
[2014-11-14 03:34:00.109758] D [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2014-11-14 03:34:00.111882] D [socket.c:492:__socket_rwv] 0-socket.management:
EOF on socket
[2014-11-14 03:34:00.111930] D [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2014-11-14 03:34:01.089999] D [socket.c:492:__socket_rwv] 0-socket.management:
EOF on socket
[2014-11-14 03:34:01.090040] D [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2014-11-14 03:34:01.348813] D
[glusterd-rpc-ops.c:901:__glusterd_commit_op_cbk] 0-management: Received commit
ACC from uuid: bac58735-7d09-4d5c-8187-eedbdbdf30a7
[2014-11-14 03:34:01.348858] D
[glusterd-utils.c:5439:glusterd_friend_find_by_uuid] 0-management: Friend
found... state: Peer in Cluster
[2014-11-14 03:34:01.348870] D [glusterd-utils.c:7548:glusterd_rb_use_rsp_dict]
0-: dst-brick-port=49153 found
[2014-11-14 03:34:01.348881] D
[glusterd-op-sm.c:5640:glusterd_op_sm_inject_event] 0-management: Enqueue
event: 'GD_OP_EVENT_RCVD_ACC'
[2014-11-14 03:34:01.348888] D [glusterd-op-sm.c:5717:glusterd_op_sm]
0-management: Dequeued event of type: 'GD_OP_EVENT_RCVD_ACC'
[2014-11-14 03:34:01.348896] D
[glusterd-utils.c:6111:glusterd_sm_tr_log_transition_add] 0-management:
Transitioning from 'Commit op sent' to 'Commit op sent' due to event
'GD_OP_EVENT_RCVD_ACC'
[2014-11-14 03:34:01.348902] D
[glusterd-utils.c:6113:glusterd_sm_tr_log_transition_add] 0-management:
returning 0
[2014-11-14 03:34:01.657424] D [socket.c:492:__socket_rwv] 0-socket.management:
EOF on socket
[2014-11-14 03:34:01.657459] D [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2014-11-14 03:34:02.124285] D [socket.c:492:__socket_rwv] 0-socket.management:
EOF on socket
[2014-11-14 03:34:02.124322] D [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2014-11-14 03:34:02.664488] D [socket.c:492:__socket_rwv] 0-socket.management:
EOF on socket
[2014-11-14 03:34:02.664527] D [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2014-11-14 03:34:02.676091] D [socket.c:492:__socket_rwv] 0-socket.management:
EOF on socket
[2014-11-14 03:34:02.676137] D [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2014-11-14 03:34:04.132191] D [socket.c:492:__socket_rwv] 0-socket.management:
EOF on socket
[2014-11-14 03:34:04.132229] D [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2014-11-14 03:34:04.183220] D
[glusterd-rpc-ops.c:901:__glusterd_commit_op_cbk] 0-management: Received commit
ACC from uuid: dcaa270d-d46f-43c2-b9a2-ad60197fd912
[2014-11-14 03:34:04.183317] D
[glusterd-utils.c:5439:glusterd_friend_find_by_uuid] 0-management: Friend
found... state: Peer in Cluster
[2014-11-14 03:34:04.183331] D [glusterd-utils.c:7542:glusterd_rb_use_rsp_dict]
0-: src-brick-port=49152 found
[2014-11-14 03:34:04.183342] D
[glusterd-op-sm.c:5640:glusterd_op_sm_inject_event] 0-management: Enqueue
event: 'GD_OP_EVENT_RCVD_ACC'
[2014-11-14 03:34:04.183349] D [glusterd-op-sm.c:5717:glusterd_op_sm]
0-management: Dequeued event of type: 'GD_OP_EVENT_RCVD_ACC'
[2014-11-14 03:34:04.183367] D
[glusterd-utils.c:6111:glusterd_sm_tr_log_transition_add] 0-management:
Transitioning from 'Commit op sent' to 'Commit op sent' due to event
'GD_OP_EVENT_RCVD_ACC'
[2014-11-14 03:34:04.183409] D
[glusterd-utils.c:6113:glusterd_sm_tr_log_transition_add] 0-management:
returning 0
[2014-11-14 03:34:04.267906] D [socket.c:492:__socket_rwv] 0-socket.management:
EOF on socket
[2014-11-14 03:34:04.267953] D [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2014-11-14 03:34:04.686770] D [socket.c:492:__socket_rwv] 0-socket.management:
EOF on socket
[2014-11-14 03:34:04.686809] D [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2014-11-14 03:34:05.139493] D [socket.c:492:__socket_rwv] 0-socket.management:
EOF on socket
[2014-11-14 03:34:05.139538] D [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2014-11-14 03:34:05.693804] D [socket.c:492:__socket_rwv] 0-socket.management:
EOF on socket
[2014-11-14 03:34:05.693843] D [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2014-11-14 03:34:05.709951] D [socket.c:492:__socket_rwv] 0-socket.management:
EOF on socket
[2014-11-14 03:34:05.709994] D [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2014-11-14 03:34:07.147269] D [socket.c:492:__socket_rwv] 0-socket.management:
EOF on socket
[2014-11-14 03:34:07.147307] D [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2014-11-14 03:34:07.761150] D [socket.c:492:__socket_rwv] 0-socket.management:
EOF on socket
[2014-11-14 03:34:07.761189] D [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2014-11-14 03:34:08.157998] D [socket.c:492:__socket_rwv] 0-socket.management:
EOF on socket
[2014-11-14 03:34:08.158039] D [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2014-11-14 03:34:08.773222] D [socket.c:492:__socket_rwv] 0-socket.management:
EOF on socket
[2014-11-14 03:34:08.773265] D [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2014-11-14 03:34:08.785388] D [socket.c:492:__socket_rwv] 0-socket.management:
EOF on socket
[2014-11-14 03:34:08.785435] D [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2014-11-14 03:34:09.464325] D
[glusterd-replace-brick.c:1910:glusterd_do_replace_brick] 0-: Cancelling timer
thread
[2014-11-14 03:34:09.464392] D
[glusterd-replace-brick.c:1914:glusterd_do_replace_brick] 0-: Replace brick
operation detected
[2014-11-14 03:34:09.464419] D
[glusterd-replace-brick.c:1929:glusterd_do_replace_brick] 0-: src
brick=data-node3:/brick/testrep
[2014-11-14 03:34:09.464428] D
[glusterd-replace-brick.c:1938:glusterd_do_replace_brick] 0-: dst
brick=data-node5:/brick/testrep
[2014-11-14 03:34:09.464435] D [glusterd-utils.c:1156:glusterd_volinfo_find]
0-management: Volume testrep found
[2014-11-14 03:34:09.464435] D [glusterd-utils.c:1156:glusterd_volinfo_find]
0-management: Volume testrep found
[2014-11-14 03:34:09.464441] D [glusterd-utils.c:1163:glusterd_volinfo_find]
0-management: Returning 0
[2014-11-14 03:34:09.464467] D [glusterd-utils.c:669:glusterd_brickinfo_new]
0-management: Returning 0
[2014-11-14 03:34:09.464476] D
[glusterd-utils.c:743:glusterd_brickinfo_new_from_brick] 0-management:
Returning 0
[2014-11-14 03:34:09.464484] D
[glusterd-utils.c:5483:glusterd_friend_find_by_hostname] 0-management: Friend
data-node3 found.. state: 3
[2014-11-14 03:34:09.464491] D
[glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management: returning 0
[2014-11-14 03:34:09.464498] D
[glusterd-utils.c:1026:glusterd_volume_brickinfo_get] 0-management: Found brick
data-node3:/brick/testrep in volume testrep
[2014-11-14 03:34:09.464516] D
[glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management: Returning 0
[2014-11-14 03:34:09.464523] D
[glusterd-utils.c:1059:glusterd_volume_brickinfo_get_by_brick] 0-: Returning 0
[2014-11-14 03:34:09.464529] D
[glusterd-utils.c:5483:glusterd_friend_find_by_hostname] 0-management: Friend
data-node5 found.. state: 3
[2014-11-14 03:34:09.464536] D
[glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management: returning 0
[2014-11-14 03:34:09.464541] D [glusterd-utils.c:685:glusterd_resolve_brick]
0-management: Returning 0
[2014-11-14 03:34:09.464549] D
[glusterd-replace-brick.c:872:rb_generate_client_volfile] 0-management:
Creating volfile
[2014-11-14 03:34:09.506554] D [run.c:190:runner_log] 0-management:
Successfully started glusterfs: /usr/sbin/glusterfs
-f/var/lib/glusterd/vols/testrep/rb_client.vol
/var/run/gluster/testrep-rb_mount
[2014-11-14 03:34:09.539425] D [run.c:190:runner_log] 0-management:
Successfully unmounted maintenance client: /bin/umount -l
/var/run/gluster/testrep-rb_mount
[2014-11-14 03:34:09.545275] D
[glusterd-op-sm.c:5640:glusterd_op_sm_inject_event] 0-management: Enqueue
event: 'GD_OP_EVENT_COMMIT_ACC'
[2014-11-14 03:34:09.545301] D [glusterd-op-sm.c:5717:glusterd_op_sm]
0-management: Dequeued event of type: 'GD_OP_EVENT_COMMIT_ACC'
[2014-11-14 03:34:09.545436] D
[glusterd-rpc-ops.c:1248:glusterd_cluster_unlock] 0-management: Returning 0
[2014-11-14 03:34:09.545540] D
[glusterd-rpc-ops.c:1248:glusterd_cluster_unlock] 0-management: Returning 0
[2014-11-14 03:34:09.545552] D
[glusterd-op-sm.c:2465:glusterd_op_ac_send_unlock] 0-management: Returning with
0
[2014-11-14 03:34:09.545560] D
[glusterd-utils.c:6111:glusterd_sm_tr_log_transition_add] 0-management:
Transitioning from 'Commit op sent' to 'Unlock sent' due to event
'GD_OP_EVENT_COMMIT_ACC'
[2014-11-14 03:34:09.545566] D
[glusterd-utils.c:6113:glusterd_sm_tr_log_transition_add] 0-management:
returning 0
[2014-11-14 03:44:15.653040] E [rpc-clnt.c:208:call_bail] 0-management: bailing
out frame type(glusterd mgmt) op(--(2)) xid = 0x1a sent = 2014-11-14
03:34:09.545536. timeout = 600 for 192.168.1.155:24007
And I check the glusterd status, it appears running. after that.
BTW, i try the same test with glusterfs 3.6.1. The replace-brick works well,
except i need commit with "force", otherwise it will commit failed.
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list