[Gluster-users] Self-Heal Daemon not Running

[Ravishankar N]

On 09/25/2013 06:16 AM, Andrew Lau wrote:
> That's where I found the 200+ entries
>
> [ root at hv01 ]gluster volume heal STORAGE info split-brain
> Gathering Heal info on volume STORAGE has been successful
>
> Brick hv01:/data1
> Number of entries: 271
> at                    path on brick
>
> 2013-09-25 00:04:29 /6682d31f-39ce-4896-99ef-14e1c9682585/dom_md/ids
> 2013-09-25 00:04:29 
> /6682d31f-39ce-4896-99ef-14e1c9682585/images/5599c7c7-0c25-459a-9d7d-80190a7c739b/0593d351-2ab1-49cd-a9b6-c94c897ebcc7
> 2013-09-24 23:54:29 <gfid:9c83f7e4-6982-4477-816b-172e4e640566>
> 2013-09-24 23:54:29 <gfid:91e98909-c217-417b-a3c1-4cf0f2356e14>
> <snip>
>
> Brick hv02:/data1
> Number of entries: 0
>
> When I run the same command on hv02, it will show the reverse (the 
> other node having 0 entries).
>
> I remember last time having to delete these files individually on 
> another split-brain case, but I was hoping there was a better solution 
> than going through 200+ entries.
>
While I haven't tried it out myself, Jeff Darcy has written a script 
(https://github.com/jdarcy/glusterfs/tree/heal-script/extras/heal_script) which 
helps in automating the process. He has detailed it's usage in his blog 
post http://hekafs.org/index.php/2012/06/healing-split-brain/

Hope this helps.
-Ravi
> Cheers.
>
>
> On Wed, Sep 25, 2013 at 10:39 AM, Mohit Anchlia 
> <mohitanchlia at gmail.com <mailto:mohitanchlia at gmail.com>> wrote:
>
>     What's the output of
>     |gluster volume heal $VOLUME info ||split||-brain|
>
>
>     On Tue, Sep 24, 2013 at 5:33 PM, Andrew Lau <andrew at andrewklau.com
>     <mailto:andrew at andrewklau.com>> wrote:
>
>         Found the BZ
>         https://bugzilla.redhat.com/show_bug.cgi?id=960190 - so I
>         restarted one of the volumes and it seems to have restarted
>         the all daemons again.
>
>         Self heal started again, but I seem to have split-brain issues
>         everywhere. There's over 100 different entries on each node,
>         what's the best way to restore this now? Short of having to
>         manually go through and delete 200+ files. It looks like a
>         full split brain as the file sizes on the different nodes are
>         out of balance by about 100GB or so.
>
>         Any suggestions would be much appreciated!
>
>         Cheers.
>
>         On Tue, Sep 24, 2013 at 10:32 PM, Andrew Lau
>         <andrew at andrewklau.com <mailto:andrew at andrewklau.com>> wrote:
>
>             Hi,
>
>             Right now, I have a 2x1 replica. Ever since I had to
>             reinstall one of the gluster servers, there's been issues
>             with split-brain. The self-heal daemon doesn't seem to be
>             running on either of the nodes.
>
>             To reinstall the gluster server (the original brick data
>             was intact but the OS had to be reinstalled)
>             - Reinstalled gluster
>             - Copied over the old uuid from backup
>             - gluster peer probe
>             - gluster volume sync $othernode all
>             - mount -t glusterfs localhost:STORAGE /mnt
>             - find /mnt -noleaf -print0 | xargs --null stat >/dev/null
>             2>/var/log/glusterfs/mnt-selfheal.log
>
>             I let it resync and it was working fine, atleast so I
>             thought. I just came back a few days later to see there's
>             a miss match in the brick volumes. One is 50GB ahead of
>             the other.
>
>             # gluster volume heal STORAGE info
>             Status: self-heal-daemon is not running on
>             966456a1-b8a6-4ca8-9da7-d0eb96997cbe
>
>             /var/log/gluster/glustershd.log doesn't seem to have any
>             recent logs, only those from when the two original gluster
>             servers were running.
>
>             # gluster volume status
>
>             Self-heal Daemon on localhostN/ANN/A
>
>             Any suggestions would be much appreciated!
>
>             Cheers
>             Andrew.
>
>
>
>         _______________________________________________
>         Gluster-users mailing list
>         Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>         http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130925/29ad61b9/attachment.html>