[Gluster-devel] Brick replace

[Atin Mukherjee]

On 09/07/2014 06:21 AM, Emmanuel Dreyfus wrote:
> Hi
> 
> I try getting tests/basic/pump.t to pass on NetBSD, but after a few
> experiments, it seems the brick replace functionality is just broken.
> 
> I run that steps one by one on a fresh install:
> 
> netbsd0# glusterd
> netbsd0# $CLI volume create $V0 $H0:$B0/${V0}0
> volume create: patchy: success: please start the volume to access data
> netbsd0# $CLI volume start $V0
> volume start: patchy: success
> netbsd0# $GFS --volfile-id=/$V0 --volfile-server=$H0 $M0;  
> netbsd0# cp -r /usr/share/misc/ $M0/
> netbsd0# $CLI volume replace-brick $V0 $H0:$B0/${V0}0 $H0:$B0/${V0}1
> start
> volume replace-brick: success: replace-brick started successfully
> ID: 98030ade-2dab-467e-86cb-cbea2436e85f
> netbsd0# $CLI volume replace-brick $V0 $H0:$B0/${V0}0 $H0:$B0/${V0}1
> commit
> volume replace-brick: failed: Commit failed on localhost. Please check
> the log file for more details.
> 
> Where logs should I look at? There is nothing in the mount log. Here is
> glusterd log:
> 
> [2014-09-07 00:39:01.905900] I
> [glusterd-replace-brick.c:154:__glusterd_handle_replace_brick]
> 0-management: Received replace brick commit request
> [2014-09-07 00:39:01.948259] I
> [glusterd-replace-brick.c:1441:rb_update_srcbrick_port] 0-: adding
> src-brick port no
> [2014-09-07 00:39:01.951238] I
> [glusterd-replace-brick.c:1495:rb_update_dstbrick_port] 0-: adding
> dst-brick port no
> [2014-09-07 00:39:01.974376] E
> [glusterd-replace-brick.c:1780:glusterd_op_replace_brick] 0-management:
> Commit operation failed
> [2014-09-07 00:39:01.974423] E
> [glusterd-op-sm.c:4109:glusterd_op_ac_send_commit_op] 0-management:
> Commit of operation 'Volume Replace brick' failed on localhost  
> 
> The brick log has a lot of errors:
> 
> [2014-09-07 00:44:44.041565] E
> [client-handshake.c:1544:client_query_portmap] 0-patchy-replace-brick:
> remote-subvolume not set in volfile
> [2014-09-07 00:44:44.041636] I [client.c:2215:client_rpc_notify]
> 0-patchy-replace-brick: disconnected from patchy-replace-brick. Client
> process will keep trying to connect to glusterd until brick's port is
> available
> 
> Any hint of where to look at?

Looking at the code, I can see there are lot of errors logged in debug
level, I would suggest you to run glusterd with -LDEBUG and reproduce
the issue such that you can get to the exact problem area looking at the
glusterd log (just check why rb_do_operation returns -1)

Having said that, I believe we should also change the loglevel from
DEBUG to ERROR in few failure cases in rb_do_operation ()

~Atin
> 
>