[Gluster-users] Migrating data from a failing filesystem

[james.bellinger at icecube.wisc.edu]

Thanks for the info!
I started the remove-brick start and, of course, the brick went read-only
in less than an hour.
This morning I checked the status a couple of minutes apart and found:

     Node Rebalanced-files       size     scanned      failures        
status
---------      -----------   --------   ---------   -----------  
------------
gfs-node04             6634   590.7GB       81799         14868    in
progress
...
gfs-node04             6669   596.5GB       86584         15271    in
progress

I'm not sure exactly what it is doing here:  4785 files scanned, 403
failures, and 35 rebalanced.  The used amount on the partition hasn't
changed.  If anything, the _other_ brick on the server is shrinking! 
(Which is related to the question I had before that you mention below.)

gluster volume remove-brick scratch gfs-node04:/sdb start
but...
df /sda
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda             12644872688 10672989432 1844930140  86% /sda
...
/dev/sda             12644872688 10671453672 1846465900  86% /sda

Have I shot myself in the other foot?
jim





> On 09/23/2014 08:56 PM, james.bellinger at icecube.wisc.edu wrote:
>> I inherited a non-replicated gluster system based on antique hardware.
>>
>> One of the brick filesystems is flaking out, and remounts read-only.  I
>> repair it and remount it, but this is only postponing the inevitable.
>>
>> How can I migrate files off a failing brick that intermittently turns
>> read-only?  I have enough space, thanks to a catastrophic failure on
>> another brick; I don't want to present people with another one.  But if
>> I
>> understand migration correctly references have to be deleted, which
>> isn't
>> possible if the filesystem turns read-only.
>
> What you could do is initiate the  migration  with `remove-brick start'
> and monitor the progress with 'remove-brick status`. Irrespective of
> whether the rebalance  completes or fails (due to the brick turning
> read-only), you could anyway update the volume configuration with
> 'remove-brick commit`. Now if the brick still has files left, mount the
> gluster volume on that node and copy the files from the brick to the
> volume via the mount.  You can then safely rebuild the array/ add a
> different brick or whatever.
>
>> What I want to do is migrate the files off, remove it from gluster,
>> rebuild the array, rebuild the filesystem, and then add it back as a
>> brick.  (Actually what I'd really like is to hear that the students are
>> all done with the system and I can turn the whole thing off, but theses
>> aren't complete yet.)
>>
>> Any advice or words of warning will be appreciated.
>
> Looks like your bricks are in trouble for over a year now
> (http://gluster.org/pipermail/gluster-users.old/2013-September/014319.html).
> Better get them fixed sooner than later! :-)

Oddly enough the old XRAID systems are holding up better than the VTRAK
arrays.  That doesn't help me much, though, since they're so small.

> HTH,
> Ravi
>
>> James Bellinger
>>
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>