[Gluster-users] How to re-sync

[Chad]

 > When the server dies suddenly it should tell people it dies suddenly??
 > Hmm, this needs some more thought...

Not the server that went down, the first client that notices (times out on the server) should send out the notifications.
I thought this note made that clear, my bad.
 >> (in this case to all the clients notifying them not to use the downed server [I hope this removes the small hang delay for clients 2-N], as well as via 
e-mail to the sysadmin).

^C



Ed W wrote:
> Hi
> 
>> 1. I think when a server goes down it should be "flagged as faulty" 
>> and send out notifications (in this case to all the clients notifying 
>> them not to use the downed server [I hope this removes the small hang 
>> delay for clients 2-N], as well as via e-mail to the sysadmin).
> 
> When the server dies suddenly it should tell people it dies suddenly?? 
> Hmm, this needs some more thought...
> 
> However, I guess it could be useful to have a way to take servers out of 
> service for scheduled reasons?  Perhaps this is what you meant?
> 
>> 3. Then when the down server comes back and starts glusterfsd it 
>> remains "faulty" and no client can use it.
> 
> I agree with where you are going, but if the autoheal works as 
> advertised then there is no reason to stop any client using it - it will 
> simply self heal as soon as someone requests a file which is stale (this 
> is at  least what it's claimed to do...)
> 
>> 5. A sysadmin changes the "faulty flag" to a "resync flag" ("resync 
>> flag" tells the clients to write to the machine, but not read from it 
>> while it recovers).
>> 6. A sysadmin then runs a re-sync (ls -alR).
>> 7. Once the re-sync completes a sysadmin runs a "re-add" command 
>> removing the "faulty flag" and the clients can begin using the server 
>> again.
> 
> I do agree that it would be very helpful to have an idea of whether 
> servers are properly in sync or not though.
> 
> Consider the scenario of upgrading a cluster, ie take down S1, upgrade 
> it, then bring it up again, take down S2, upgrade it, then bring it up 
> again.  If you don't fully sync S1 and S2 in the middle then you have a 
> split brain situation which must lead to data loss...
> 
> Perhaps the ls -alR is 100% sufficient to guarantee the entire 
> filesystem is synced and hence is completely sufficient, but split brain 
> IS the major fear with clustered systems and it would be nice to have 
> even stronger guarantees of consistency...
> 
> 
>> I feel that this method removes the chance that a server goes down, 
>> gets out of sync, recovers on its own (or though automated tools), and 
>> starts providing services with some old data.
>> In the middle of the night if the server goes down, and nagios trips a 
>> reboot, then the server comes up, no sysadmin is logged in to run the 
>> "ls -alR" to get the server to re-sync.
> 
> Yeah, I agree that this scenario is scary.  Actually you missed out an 
> implied step which is if the *other* server dies before the resync 
> happens then you have a risk of split brain.
> 
> Arguably it's not necessary to fence the recovering server during the 
> recovery, but you definitely want to fence it if cannot completely 
> resync for some reason...
> 
> 
> Ed W
> 
>