[Gluster-users] how to reboot all bricks safely and seamlessly

[Ravishankar N]

On 08/12/2015 05:20 PM, Kingsley wrote:
> On Wed, 2015-08-12 at 17:09 +0530, Ravishankar N wrote:
>> On 08/11/2015 10:06 PM, Kingsley wrote:
>>> Hi,
>>>
>>> If you need to reboot all bricks in a volume, what's the best way to do
>>> this seamlessly?
>>>
>>> I did this a few days ago by rebooting one, then waiting for "gluster
>>> volume info" on another brick to show it back online before doing the
>>> next, and so on. However, it went a bit wrong and I ended up with a
>>> corruption. It fixed itself after a while, but because the system then
>>> had a backlog of stuff to catch up with, it didn't fix itself for about
>>> a day.
>>>
>>> I wonder that I didn't leave enough time for the freshly rebooted brick
>>> to apply all of the updates to itself that happened while it was
>>> rebooting. So, what's the best way to find out whether a brick has fully
>>> synced itself with the other bricks, before rebooting the next one?
>> 'gluster volume heal <volname> info` must show zero entries. If it
>> doesn't, you can manually launch self heal by 'gluster volume heal
>> <volname>`
> Thanks for that, Ravi, that's very useful to know. I did try the volume
> heal, but I couldn't see how to figure out how far it had got in
> percentage terms. Is there a command that I can issue that gives an
> indication of how far through the heal process has got, eg "healed 370
> or 1022 objects"?
The statistics option for the heal command should give you the number of 
files healed per crawl. You can refer to 
https://github.com/gluster/glusterfs/blob/release-3.7/doc/features/afr-statistics.md 
for an explanation. We do not have an indication in terms of percentage, 
but 'gluster volume heal <volname> info` always lists the files that 
need healing. If that is non-zero, it means self-heal is not yet complete.

Hope this helps.
-Ravi