[Gluster-users] remove-brick removed unexpected bricks

[Cool]

I'm pretty sure I did "watch ... remove-brick ... status" till it 
mentioned everything is completed before trigger commit, I should make 
it clear in my previous mail.

Actually you can read my mail again - in step #5, files on /sdc1 got 
migrated instead of /sdd1, even though my command was trying to 
remove-brick /sdd1, this is the root cause (to me) that caused the 
problem, as data on /sdc1 migrated to /sdb1 and /sdd1, then commit 
simply remove /sdd1 from gfs_v0. It seems vol definition information got 
some problem in gluster.

-C.B.

On 8/12/2013 9:51 PM, Ravishankar N wrote:
> On 08/13/2013 03:43 AM, Cool wrote:
>> remove-brick in 3.4.0 seems removing wrong bricks, can someone help 
>> to review the environment/steps to see if I did anything stupid?
>>
>> setup - Ubuntu 12.04LTS on gfs11 and gfs12, with following packages 
>> from ppa, both nodes have 3 xfs partitions sdb1, sdc1, sdd1:
>> ii  glusterfs-client 3.4.0final-ubuntu1~precise1 clustered 
>> file-system (client package)
>> ii  glusterfs-common 3.4.0final-ubuntu1~precise1 GlusterFS common 
>> libraries and translator modules
>> ii  glusterfs-server 3.4.0final-ubuntu1~precise1 clustered 
>> file-system (server package)
>>
>> step to reproduce the problem:
>> 1. create volume gfs_v0 in replica 2 with gfs11:/sdb1 and gfs12:/sdb1
>> 2. add-brick gfs11:/sdc1 and gfs12:/sdc1
>> 3. add-brick gfs11:/sdd1 and gfs12:/sdd1
>> 4. rebalance to make files distributed to all three pair of disks
>> 5. remove-brick gfs11:/sdd1 and gfs12:/sdd1 start, files on 
>> ***/sdc1*** are migrating out
>> 6. remove-brick commit led to data loss in gfs_v0
>>
>> If between step 5 and 6 I initiate a remove-brick targeting /sdc1, 
>> then after commit I would not lose anything since all data will be 
>> migrated back to /sdb1.
>>
>
> You should ensure  that a 'remove-brick  start ' has completed and 
> then commit it before initiating the second one. The correct way to do 
> this would be:
> 5.   # gluster volume remove-brick gfs_v0 gfs11:/sdd1 gfs12:/sdd1 start
> 6. Check that the data migration has been completed using the status 
> command:
>       # gluster volume remove-brick gfs_v0 gfs11:/sdd1 gfs12:/sdd1 status
> 7.   #gluster volume remove-brick gfs_v0 gfs11:/sdd1 gfs12:/sdd1 commit
> 8.   # gluster volume remove-brick gfs_v0 gfs11:/sdc1 gfs12:/sdc1 start
> 9.   # gluster volume remove-brick gfs_v0 gfs11:/sdc1 gfs12:/sdc1 status
> 10. # gluster volume remove-brick gfs_v0 gfs11:/sdc1 gfs12:/sdc1 commit
>
> This would leave you with the original replica 2 volume that you had 
> begun with. Hope this helps.
>
> Note:
> The latest version of glusterfs has the check that prevents a second 
> remove-brick operation until the first one has been committed.
> (You would receive a message thus : "volume remove-brick start: 
> failed: An earlier remove-brick task exists for volume <volname>. 
> Either commit it or stop it before starting a new task." )
>
> -Ravi
>
>
>> -C.B.
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>