[Gluster-users] Removing a brick
Sam Giraffe
sam at giraffetech.biz
Tue Feb 24 15:05:10 UTC 2015
# rpm -qa | grep -i gluster
glusterfs-libs-3.6.2-1.el6.x86_64
glusterfs-api-3.6.2-1.el6.x86_64
glusterfs-fuse-3.6.2-1.el6.x86_64
glusterfs-server-3.6.2-1.el6.x86_64
glusterfs-cli-3.6.2-1.el6.x86_64
glusterfs-geo-replication-3.6.2-1.el6.x86_64
glusterfs-3.6.2-1.el6.x86_64
/var/log/glusterfs/cli.log has some interesting data in it:
[2015-02-24 15:01:41.902048] D [cli.c:612:cli_rpc_init] 0-cli:
Connecting to glusterd using default socket
[2015-02-24 15:01:41.902163] D
[rpc-clnt.c:972:rpc_clnt_connection_init] 0-glusterfs: defaulting
frame-timeout to 30mins
[2015-02-24 15:01:41.902190] D
[rpc-clnt.c:986:rpc_clnt_connection_init] 0-glusterfs: disable
ping-timeout
[2015-02-24 15:01:41.902221] D
[rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to
load file /usr/lib64/glusterfs/3.6.2/rpc-transport/socket.so
[2015-02-24 15:01:41.904940] D [socket.c:3684:socket_init]
0-glusterfs: disabling nodelay
[2015-02-24 15:01:41.904973] D [socket.c:3799:socket_init]
0-glusterfs: SSL support on the I/O path is NOT enabled
[2015-02-24 15:01:41.904999] D [socket.c:3802:socket_init]
0-glusterfs: SSL support for glusterd is NOT enabled
[2015-02-24 15:01:41.905014] D [socket.c:3819:socket_init]
0-glusterfs: using system polling thread
[2015-02-24 15:01:41.905044] T [rpc-clnt.c:418:rpc_clnt_reconnect]
0-glusterfs: attempting reconnect
[2015-02-24 15:01:41.905066] T [socket.c:2871:socket_connect]
0-glusterfs: connecting 0x1f33f30, state=0 gen=0 sock=-1
[2015-02-24 15:01:41.905090] T
[name.c:290:af_unix_client_get_remote_sockaddr] 0-glusterfs: using
connect-path /var/run/glusterd.socket
[2015-02-24 15:01:41.905117] T [name.c:106:af_unix_client_bind]
0-glusterfs: bind-path not specified for unix socket, letting connect
to assign default value
[2015-02-24 15:01:41.905278] D
[rpc-clnt.c:972:rpc_clnt_connection_init] 0-glusterfs: defaulting
frame-timeout to 30mins
[2015-02-24 15:01:41.905302] D
[rpc-clnt.c:986:rpc_clnt_connection_init] 0-glusterfs: disable
ping-timeout
[2015-02-24 15:01:41.905326] D
[rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to
load file /usr/lib64/glusterfs/3.6.2/rpc-transport/socket.so
[2015-02-24 15:01:41.905368] D [socket.c:3799:socket_init]
0-glusterfs: SSL support on the I/O path is NOT enabled
[2015-02-24 15:01:41.905387] D [socket.c:3802:socket_init]
0-glusterfs: SSL support for glusterd is NOT enabled
[2015-02-24 15:01:41.905408] D [socket.c:3819:socket_init]
0-glusterfs: using system polling thread
[2015-02-24 15:01:41.905429] T [rpc-clnt.c:418:rpc_clnt_reconnect]
0-glusterfs: attempting reconnect
[2015-02-24 15:01:41.905448] T [socket.c:2871:socket_connect]
0-glusterfs: connecting 0x1f3c3e0, state=0 gen=0 sock=-1
[2015-02-24 15:01:41.905469] T
[name.c:290:af_unix_client_get_remote_sockaddr] 0-glusterfs: using
connect-path /tmp/quotad.socket
[2015-02-24 15:01:41.905501] T [name.c:106:af_unix_client_bind]
0-glusterfs: bind-path not specified for unix socket, letting connect
to assign default value
[2015-02-24 15:01:41.905551] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:41.905592] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:41.905614] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:41.905635] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:41.905656] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:41.905691] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:41.905717] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:41.905741] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:41.905767] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:41.905789] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:41.905810] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:41.905849] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:41.905866] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:41.905883] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:41.905900] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.067441] D
[cli-cmd-volume.c:1787:cli_check_gsync_present] 0-cli: Returning 0
[2015-02-24 15:01:42.067547] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.067576] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.067617] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.067653] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.067680] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.067707] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.067733] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.067752] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.067775] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.067797] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.067821] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.067843] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.067878] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.067898] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.067914] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.067938] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.067962] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.067983] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068002] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068025] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068040] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068055] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068070] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068083] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068099] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068114] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068131] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068153] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068168] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068191] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068216] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068236] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068254] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068275] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068297] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068328] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068350] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068372] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.068395] D [registry.c:408:cli_cmd_register]
0-cli: Returning 0
[2015-02-24 15:01:42.069226] T [cli.c:264:cli_rpc_notify] 0-glusterfs:
got RPC_CLNT_CONNECT
[2015-02-24 15:01:42.069271] T
[cli-quotad-client.c:94:cli_quotad_notify] 0-glusterfs: got
RPC_CLNT_CONNECT
[2015-02-24 15:01:42.069300] I [socket.c:2344:socket_event_handler]
0-transport: disconnecting now
[2015-02-24 15:01:42.069338] T
[cli-quotad-client.c:100:cli_quotad_notify] 0-glusterfs: got
RPC_CLNT_DISCONNECT
[2015-02-24 15:01:42.069455] T [rpc-clnt.c:1381:rpc_clnt_record]
0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner:
[2015-02-24 15:01:42.069499] T
[rpc-clnt.c:1238:rpc_clnt_record_build_header] 0-rpc-clnt: Request
fraglen 220, payload: 156, rpc hdr: 64
[2015-02-24 15:01:42.069915] T [socket.c:2863:socket_connect] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f9d1ceb3550]
(--> /usr/lib64/glusterfs/3.6.2/rpc-transport/socket.so(+0x7293)[0x7f9d1a142293]
(--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x468)[0x7f9d1c60bfe8]
(--> gluster(cli_submit_request+0xdb)[0x40a96b] (-->
gluster(cli_cmd_submit+0x8e)[0x40b76e] ))))) 0-glusterfs: connect ()
called on transport already connected
[2015-02-24 15:01:42.070711] T [rpc-clnt.c:1573:rpc_clnt_submit]
0-rpc-clnt: submitted request (XID: 0x1 Program: Gluster CLI,
ProgVers: 2, Proc: 11) to rpc-transport (glusterfs)
[2015-02-24 15:01:42.070739] D
[rpc-clnt-ping.c:231:rpc_clnt_start_ping] 0-glusterfs: ping timeout is
0, returning
[2015-02-24 15:01:42.078829] T [rpc-clnt.c:660:rpc_clnt_reply_init]
0-glusterfs: received rpc message (RPC XID: 0x1 Program: Gluster CLI,
ProgVers: 2, Proc: 11) from rpc-transport (glusterfs)
[2015-02-24 15:01:42.079096] T
[cli-rpc-ops.c:1358:gf_cli_print_rebalance_status] 0-cli: failed to
get status
[2015-02-24 15:01:42.079118] E
[cli-rpc-ops.c:1973:gf_cli3_remove_brick_status_cbk] 0-cli: Failed to
print remove-brick rebalance status
[2015-02-24 15:01:42.079213] D [cli-cmd.c:384:cli_cmd_submit] 0-cli:
Returning -2
[2015-02-24 15:01:42.079242] D
[cli-rpc-ops.c:3592:gf_cli_remove_brick] 0-cli: Returning -2
[2015-02-24 15:01:42.079260] D
[cli-cmd-volume.c:1557:cli_cmd_volume_remove_brick_cbk] 0-cli:
frame->local is not NULL (0x7f9d0c0009c0)
[2015-02-24 15:01:42.079287] I [input.c:36:cli_batch] 0-: Exiting with: -2
/var/log/glusterfs/etc-glusterfs-glusterd.vol.log is has repeats of:
2015-02-24 14:53:23.852588] W [socket.c:611:__socket_rwv]
0-management: readv on
/var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid
argument)
[2015-02-24 14:53:26.894893] W [socket.c:611:__socket_rwv]
0-management: readv on
/var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid
argument)
The message "I [MSGID: 106004]
[glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management:
Peer a32a46c8-d2d4-4b5d-8f7b-ee6dee3a15e0, in Peer in Cluster state,
has disconnected from glusterd." repeated 34 times between [2015-02-24
14:51:32.621065] and [2015-02-24 14:53:22.852790]
The message "I [MSGID: 106006]
[glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management:
nfs has disconnected from glusterd." repeated 39 times between
[2015-02-24 14:51:27.420758] and [2015-02-24 14:53:26.894928]
[2015-02-24 14:53:28.895747] I [MSGID: 106004]
[glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management:
Peer a32a46c8-d2d4-4b5d-8f7b-ee6dee3a15e0, in Peer in Cluster state,
has disconnected from glusterd.
[2015-02-24 14:53:29.895225] W [socket.c:611:__socket_rwv]
0-management: readv on
/var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid
argument)
[2015-02-24 14:53:29.895264] I [MSGID: 106006]
[glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management:
nfs has disconnected from glusterd.
[2015-02-24 14:53:32.922976] W [socket.c:611:__socket_rwv]
0-management: readv on
/var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid
argument)
[2015-02-24 14:53:35.952776] W [socket.c:611:__socket_rwv]
0-management: readv on
/var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid
argument)
[2015-02-24 14:53:38.983371] W [socket.c:611:__socket_rwv]
0-management: readv on
/var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid
argument)
[2015-02-24 14:53:42.077272] W [socket.c:611:__socket_rwv]
0-management: readv on
/var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid
argument)
[2015-02-24 14:53:45.129492] W [socket.c:611:__socket_rwv]
0-management: readv on
/var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid
argument)
[2015-02-24 14:53:48.327998] W [socket.c:611:__socket_rwv]
0-management: readv on
/var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid
argument)
[2015-02-24 14:53:51.368593] W [socket.c:611:__socket_rwv]
0-management: readv on
/var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid
argument)
[2015-02-24 14:53:54.521800] W [socket.c:611:__socket_rwv]
0-management: readv on
/var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid
argument)
[2015-02-24 14:53:57.560446] W [socket.c:611:__socket_rwv]
0-management: readv on
/var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid
argument)
[2015-02-24 14:54:00.643818] W [socket.c:611:__socket_rwv]
0-management: readv on
/var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid
argument)
[2015-02-24 14:54:03.675623] W [socket.c:611:__socket_rwv]
0-management: readv on
/var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid
argument)
[2015-02-24 14:54:06.708956] W [socket.c:611:__socket_rwv]
0-management: readv on
/var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid
argument)
On Tue, Feb 24, 2015 at 1:18 AM, Atin Mukherjee <amukherj at redhat.com> wrote:
> Could you attach glusterd logs? Which version of glusterfs-server are
> you using?
>
> ~Atin
>
> On 02/24/2015 12:59 PM, Sam Giraffe wrote:
>> I get the same blank message when I try brick status with both bricks:
>>
>> # gluster volume remove-brick art server1:/brick1 server2:/brick2 status
>>
>> Node Rebalanced-files size scanned failures
>> skipped status run time in secs
>> --------- ----------- ----------- ----------- -----------
>> ----------- ------------ --------------
>>
>> On Mon, Feb 23, 2015 at 8:17 PM, Atin Mukherjee <amukherj at redhat.com> wrote:
>>>
>>>
>>> On 02/24/2015 06:41 AM, Sam Giraffe wrote:
>>>> Hi,
>>>>
>>>> On my 20 nodes with 2 replica cluster I was able to run:
>>>>
>>>> # gluster volume remove-brick art server1:/brick1 server2:/brick2 start
>>>>
>>>> I got a message stating:
>>>>
>>>> volume remove-brick start: success
>>>> ID: 16b887bb-e848-4054-a5af-9390055d32c9
>>>>
>>>> However, when I run 'status' I get nothing in the output:
>>>>
>>>> $ gluster volume remove-brick art server1:/brick1 status
>>>> Node Rebalanced-files size scanned failures
>>>> skipped status run time in secs
>>>> --------- -----------
>>>> ----------- ----------- ----------- -----------
>>>> ------------ --------------
>>> In remove-brick start you have couple of bricks to be removed however
>>> the status command has only one, Could you try remove brick status with
>>> correct brick details i.e. appending brick2 in the status query.
>>>>
>>>> And when I try a commit I get an error saying that the removal is in progress:
>>>>
>>>> # gluster volume remove-brick art server1:/brick1 server2:/brick2 commit
>>>> Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
>>>> volume remove-brick commit: failed: Staging failed on server1. Error:
>>>> use 'force' option as migration is in progress
>>>> Staging failed on server2. Error: use 'force' option as migration is in progress
>>>>
>>> This is expected as remove-brick transaction hasn't finished.
>>>> Any clue on how to get status of brick removal?
>>>>
>>>> Thank you
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>
>>> --
>>> ~Atin
>>
>>
>
> --
> ~Atin
More information about the Gluster-users
mailing list