[Gluster-users] replace-brick not working

Mon Feb 14 18:05:25 UTC 2011

Hello All,

I need to migrate some data from a pair of mirrored bricks (i.e. a brick 
and it's mirror).  Unfortunately the "... replace-brick ... start" and 
"... replace-brick ... status" commands give a null response.   The 
commands I used followed by the relevant log file errors are shown 
below.  The bdan11 server (where I want the data to go) is new and 
doesn't have any data on it at the moment.  However glusterd is running 
and the host is part of the peer group.  The fuse module is loaded  on 
all servers.  I tried restarting all the gluster daemons on all the 
servers but it didn't make any difference.  I also tried a similar 
operation on another volume using other servers (all with existing 
GlusterFS bricks this time) with the same result.  Any suggestions would 
be much appreciated.

[root at bdan0 ~]# gluster volume replace-brick atmos 
bdan5:/vg2lv1/glusterfs2 bdan11:/atmos/glusterfs start
[root at bdan0 ~]# gluster volume replace-brick atmos 
bdan5:/vg2lv1/glusterfs2 bdan11:/atmos/glusterfs status
[root at bdan0 ~]#

---- on bdan11 (where I want the data to go to) ---

[2011-02-14 16:57:52.296610] I 
[glusterd-handler.c:673:glusterd_handle_cli_list_friends] glusterd: 
Received cli list req
[2011-02-14 16:58:21.56781] I 
[glusterd-handler.c:425:glusterd_handle_cluster_lock] glusterd: Received 
LOCK from uuid: 473c4e80-d55c-45ee-8a52-a4aba4da8317
[2011-02-14 16:58:21.56828] I [glusterd-utils.c:236:glusterd_lock] 
glusterd: Cluster lock held by 473c4e80-d55c-45ee-8a52-a4aba4da8317
[2011-02-14 16:58:21.56868] I 
[glusterd-handler.c:1973:glusterd_op_lock_send_resp] glusterd: 
Responded, ret: 0
[2011-02-14 16:58:21.96404] I 
[glusterd-handler.c:464:glusterd_handle_stage_op] glusterd: Received 
stage op from uuid: 473c4e80-d55c-45ee-8a52-a4aba4da8317
[2011-02-14 16:58:21.96466] I 
[glusterd-utils.c:730:glusterd_volume_brickinfo_get_by_brick] : brick: 
bdan5:/vg2lv1/glusterfs2
[2011-02-14 16:58:21.99679] I 
[glusterd-utils.c:2144:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan5 found.. state: 3
[2011-02-14 16:58:21.99720] I 
[glusterd-utils.c:701:glusterd_volume_brickinfo_get] : Found brick
[2011-02-14 16:58:21.102568] I 
[glusterd-handler.c:2065:glusterd_op_stage_send_resp] glusterd: 
Responded to stage, ret: 0

---- on bdan5 (current location of data) ---

[2011-02-14 16:58:59.159290] I 
[glusterd-handler.c:425:glusterd_handle_cluster_lock] glusterd: Received 
LOCK from uuid: 473c4e80-d55c-45ee-8a52-a4aba4da8317
[2011-02-14 16:58:59.159567] I [glusterd-utils.c:236:glusterd_lock] 
glusterd: Cluster lock held by 473c4e80-d55c-45ee-8a52-a4aba4da8317
[2011-02-14 16:58:59.159621] I 
[glusterd-handler.c:1973:glusterd_op_lock_send_resp] glusterd: 
Responded, ret: 0
[2011-02-14 16:58:59.198752] I 
[glusterd-handler.c:464:glusterd_handle_stage_op] glusterd: Received 
stage op from uuid: 473c4e80-d55c-45ee-8a52-a4aba4da8317
[2011-02-14 16:58:59.198799] I 
[glusterd-utils.c:730:glusterd_volume_brickinfo_get_by_brick] : brick: 
bdan5:/vg2lv1/glusterfs2
[2011-02-14 16:58:59.199123] I 
[glusterd-utils.c:701:glusterd_volume_brickinfo_get] : Found brick
[2011-02-14 16:58:59.199183] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.199201] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.199214] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.199227] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.199240] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.199251] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.199263] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.199275] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.199287] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.199299] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.199312] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.201616] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.201667] I 
[glusterd-handler.c:2065:glusterd_op_stage_send_resp] glusterd: 
Responded to stage, ret: 0

---- on bdan0 (where volume commands were executed)---

[2011-02-14 16:58:59.159586] I 
[glusterd-handler.c:1222:glusterd_handle_replace_brick] glusterd: 
Received replace brick req
[2011-02-14 16:58:59.159694] I [glusterd-utils.c:236:glusterd_lock] 
glusterd: Cluster lock held by 473c4e80-d55c-45ee-8a52-a4aba4da8317
[2011-02-14 16:58:59.159717] I 
[glusterd-handler.c:2832:glusterd_op_txn_begin] glusterd: Acquired local 
lock
[2011-02-14 16:58:59.160085] I 
[glusterd3_1-mops.c:1091:glusterd3_1_cluster_lock] glusterd: Sent lock 
req to 14 peers
[2011-02-14 16:58:59.160168] I 
[glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received 
ACC from uuid: cc81118c-ff50-4823-8e75-f710b88f9d9c
[2011-02-14 16:58:59.160204] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.160259] I 
[glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received 
ACC from uuid: 66f88fb4-5aa7-44c0-870f-a2d41fd88c47
[2011-02-14 16:58:59.160282] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.160323] I 
[glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received 
ACC from uuid: 29566f7b-fa3a-43fd-9751-086479cedd01
[2011-02-14 16:58:59.160349] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.160396] I 
[glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received 
ACC from uuid: 8e037099-d853-4fc3-a4b7-28980fd2b131
[2011-02-14 16:58:59.160419] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.160468] I 
[glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received 
ACC from uuid: 814b274a-8dbe-40d2-a406-c41d347f8ad1
[2011-02-14 16:58:59.160538] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.160577] I 
[glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received 
ACC from uuid: 80363690-44c6-4d86-9607-5af0764e8900
[2011-02-14 16:58:59.160600] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.160638] I 
[glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received 
ACC from uuid: 344ba772-9c6f-4263-a380-79a24fb6f741
[2011-02-14 16:58:59.160660] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.169445] I 
[glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received 
ACC from uuid: 6753c5fa-e33c-4724-930e-53efa6e95563
[2011-02-14 16:58:59.169481] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.172089] I 
[glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received 
ACC from uuid: 39422893-699c-4f4a-a1c4-e038a4c928b8
[2011-02-14 16:58:59.172121] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.172382] I 
[glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received 
ACC from uuid: 681f3a0d-4fad-4b18-a625-da747f551155
[2011-02-14 16:58:59.172406] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.177922] I 
[glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received 
ACC from uuid: fca1393e-ee59-4b4c-9205-b3dcdd2ee1ae
[2011-02-14 16:58:59.177946] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.181466] I 
[glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received 
ACC from uuid: a2cd8e72-151a-4fb1-936d-1fa9ef9c1ca4
[2011-02-14 16:58:59.181506] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.181548] I 
[glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received 
ACC from uuid: fa313e90-3e8e-4f4d-a266-43f34777ed07
[2011-02-14 16:58:59.181570] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.195991] I 
[glusterd3_1-mops.c:395:glusterd3_1_cluster_lock_cbk] glusterd: Received 
ACC from uuid: 551ca306-46a9-423a-92a8-0e33ccbb3b2b
[2011-02-14 16:58:59.196023] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.196072] I 
[glusterd-utils.c:730:glusterd_volume_brickinfo_get_by_brick] : brick: 
bdan5:/vg2lv1/glusterfs2
[2011-02-14 16:58:59.197837] I 
[glusterd-utils.c:2144:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan5 found.. state: 3
[2011-02-14 16:58:59.197875] I 
[glusterd-utils.c:701:glusterd_volume_brickinfo_get] : Found brick
[2011-02-14 16:58:59.197950] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.197976] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.197999] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.198023] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.198046] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.198068] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.198091] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.198155] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.198180] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.198204] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.198228] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.199251] I 
[glusterd-utils.c:2105:glusterd_friend_find_by_hostname] glusterd: 
Friend bdan11 found.. state: 3
[2011-02-14 16:58:59.199580] I 
[glusterd3_1-mops.c:1233:glusterd3_1_stage_op] glusterd: Sent op req to 
14 peers
[2011-02-14 16:58:59.202555] I 
[glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC 
from uuid: 814b274a-8dbe-40d2-a406-c41d347f8ad1
[2011-02-14 16:58:59.202582] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.203934] I 
[glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC 
from uuid: 6753c5fa-e33c-4724-930e-53efa6e95563
[2011-02-14 16:58:59.203957] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.204128] I 
[glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC 
from uuid: 8e037099-d853-4fc3-a4b7-28980fd2b131
[2011-02-14 16:58:59.204156] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.204281] I 
[glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC 
from uuid: fca1393e-ee59-4b4c-9205-b3dcdd2ee1ae
[2011-02-14 16:58:59.204303] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.204528] I 
[glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC 
from uuid: fa313e90-3e8e-4f4d-a266-43f34777ed07
[2011-02-14 16:58:59.204564] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.204608] I 
[glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC 
from uuid: 681f3a0d-4fad-4b18-a625-da747f551155
[2011-02-14 16:58:59.204631] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.204727] I 
[glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC 
from uuid: 66f88fb4-5aa7-44c0-870f-a2d41fd88c47
[2011-02-14 16:58:59.204763] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.204806] I 
[glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC 
from uuid: 344ba772-9c6f-4263-a380-79a24fb6f741
[2011-02-14 16:58:59.204829] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.204939] I 
[glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC 
from uuid: 29566f7b-fa3a-43fd-9751-086479cedd01
[2011-02-14 16:58:59.204970] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.205012] I 
[glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC 
from uuid: a2cd8e72-151a-4fb1-936d-1fa9ef9c1ca4
[2011-02-14 16:58:59.205034] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.206075] I 
[glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC 
from uuid: 80363690-44c6-4d86-9607-5af0764e8900
[2011-02-14 16:58:59.206099] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.206724] I 
[glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC 
from uuid: 551ca306-46a9-423a-92a8-0e33ccbb3b2b
[2011-02-14 16:58:59.206747] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 16:58:59.227190] I 
[glusterd3_1-mops.c:594:glusterd3_1_stage_op_cbk] glusterd: Received ACC 
from uuid: 39422893-699c-4f4a-a1c4-e038a4c928b8
[2011-02-14 16:58:59.227214] I 
[glusterd-utils.c:2066:glusterd_friend_find_by_uuid] glusterd: Friend 
found.. state: Peer in Cluster
[2011-02-14 17:07:16.354075] E 
[glusterd3_1-mops.c:1357:glusterd_handle_rpc_msg] : Unable to set cli op: 16
[2011-02-14 17:07:16.354152] E 
[glusterd-utils.c:399:glusterd_serialize_reply] : Failed to encode message
[2011-02-14 17:07:16.354179] E 
[glusterd-utils.c:441:glusterd_submit_reply] : Failed to serialize reply
[2011-02-14 17:07:16.354200] W 
[glusterd3_1-mops.c:1498:glusterd_handle_rpc_msg] : Returning -1

-Dan.