[Gluster-users] Missing files after add new bricks and remove old ones - how to restore files

[Marco Lorenzo Crociani]

Hi,
we had a 2 node glusterfs cluster.
we had a 2x2 Distributed-Replicate volume on it.

It was:
Brick1: s20gfs.ovirt:/gluster/VOL/brick1
Brick2: s21gfs.ovirt:/gluster/VOL/brick2
Brick3: s20gfs.ovirt:/gluster/VOL/brick3
Brick4: s21gfs.ovirt:/gluster/VOL/brick4

We added more nodes to the cluster. So I wanted to redistribute the 
bricks on the nodes.
I added 2 new bricks to the volume.

gluster volume add-brick VOL s22gfs.ovirt:/gluster/VOL/brick2 
s23gfs.ovirt:/gluster/VOL/brick3

then I removed 2 old bricks
gluster volume remove-brick VOL s20gfs.ovirt:/gluster/VOL/brick1 
s21gfs.ovirt:/gluster/VOL/brick2 start

checked the status
gluster volume remove-brick VOL s20gfs.ovirt:/gluster/VOL/brick1 
s21gfs.ovirt:/gluster/VOL/brick2 status

when it was completed and I saw data on the new bricks I run:
gluster volume remove-brick VOL s20gfs.ovirt:/gluster/VOL/brick1 
s21gfs.ovirt:/gluster/VOL/brick2 commit

Result:
some files missing from the volume. Those file are still on the removed 
bricks.

First question. Have I followed a wrong procedure?

Second question. How to restore those files? I can't re-add back those 
brick to the volume because it tolds me:
volume add-brick: failed: /gluster/VOL/brick1 is already part of a volume
If I use the force option I get the old files or they get erased?
Should I rsync from the  "unmounted" brick to the "mounted" volume or to 
an "umounted" brick that is part of the volume?

Regards,

-- 
Marco Crociani
Prisma Telecom Testing S.r.l.
via Petrocchi, 4  20127 MILANO  ITALY
Phone:  +39 02 26113507
Fax:  +39 02 26113597
e-mail:  marcoc at prismatelecomtesting.com
web:  http://www.prismatelecomtesting.com