[Gluster-users] remove-brick never finishes

[Branden Timm]

I started a remove-brick operation on a distributed volume roughly eight days ago and it has never finished.  The counters (besides runtime in secs) have been the same for several days now:

Node Rebalanced-files  size scanned failures skipped  status   run time in secs
localhost      5039            169.1GB        851242             0             0          in progress 689421.00

The brick has ~3.5T of data in it, so clearly more data needs to be evacuated, but nothing is moving.

The only errors I've found in the volume log are the following:

[2015-08-12 14:59:16.657643] E [glusterd-hooks.c:328:glusterd_hooks_run_hooks] 0-management: Failed to open dir /var/lib/glusterd/hooks/1/remove-brick/pre, due to No such file or directory
[2015-08-12 14:59:21.920355] E [glusterd-hooks.c:328:glusterd_hooks_run_hooks] 0-management: Failed to open dir /var/lib/glusterd/hooks/1/remove-brick/post, due to No such file or directory

Which is indeed strange, because the rpm reports it should have placed those hooks:
rpm -q --filesbypkg glusterfs-server | grep /var/lib/glusterd/hooks/1/remove-brick
glusterfs-server          /var/lib/glusterd/hooks/1/remove-brick
glusterfs-server          /var/lib/glusterd/hooks/1/remove-brick/post
glusterfs-server          /var/lib/glusterd/hooks/1/remove-brick/pre

What's even stranger is that rpm --verify works even though the files are not there:
rpm --verify glusterfs-server # returns nothing
ls -alh /var/lib/glusterd/hooks/1/remove-brick
ls: cannot access /var/lib/glusterd/hooks/1/remove-brick: No such file or directory

Any help would be greatly appreciated.  Thanks!



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150820/aa786395/attachment.html>