[Gluster-users] volume failing to heal after replace-brick

[Jordan Tomkinson]

Hi,

Using Gluster 3.6.1, I'm trying to replace a brick but after issuing a
volume heal nothing gets healed and my clients see an empty volume.

I have reproduced this on a test volume, shown here.

$ gluster volume status test

Status of volume: test
Gluster process                        Port    Online    Pid
------------------------------------------------------------------------------
Brick ds1:/export/test            49153    Y    7093
Brick ds2:/export/test            49154    Y    11472
NFS Server on localhost                    2049    Y    11484
Self-heal Daemon on localhost                N/A    Y    11491
NFS Server on 10.42.0.207                2049    Y    7110
Self-heal Daemon on 10.42.0.207                N/A    Y    7117
 
Task Status of Volume test
------------------------------------------------------------------------------
There are no active volume tasks

I then mount the volume from a client and store some files.

Now I replace ds2:/export/test with an empty disk mounted on
ds2:/export/test2

$ gluster volume replace-brick test ds2:/export/test ds2:/export/test2
commit force
volume replace-brick: success: replace-brick commit successful

At this point, doing an ls on the volume mounted from a fuse client
shows an empty volume, basically the contents of the new empty brick.

So i issue a volume heal full

$ gluster volume heal test full
Launching heal operation to perform full self heal on volume test has
been successful
Use heal info commands to check status

$ gluster volume heal test info
Gathering list of entries to be healed on volume test has been successful

Brick ds1:/export/test
Number of entries: 0

Brick ds2:/export/test2
Number of entries: 0


Nothing gets healed from ds1:/export/test to ds2:/export/test2 and my
clients still see an empty volume.

I can see the data on ds1:/export/test if i look inside the brick
directory, but nothing on ds2:/export/test2

tailing glustershd.log, nothing is printed after running the heal command.

the only log is one line from etc-glusterfs-glusterd.vol.log
[2015-02-06 02:48:39.906224] I
[glusterd-volume-ops.c:482:__glusterd_handle_cli_heal_volume]
0-management: Received heal vol req for volume test

Any ideas?



-- 
Jordan Tomkinson
System Administrator
Catalyst IT Australia