[Gluster-devel] solutions for split brain situation

[Stephan von Krawczynski]

On Mon, 14 Sep 2009 21:28:17 +0530
Anand Avati <avati at gluster.com> wrote:

> >> Now, if your system is going down - no removal of the volume - and it
> >> comes back up with the behaviour you describe, then I am very concerned
> >> as well.
> >
> > I can tell you for sure that this is the case, we fell into this hole already
> > twice, shot down around 14 days of logs (half of course because of the
> > distributing file access).
> > This is really a hotspot that should be dealt with as soon as possible. The
> > only current solution is to delete everything before re-adding volumes.
> 
> Can you please describe the exact steps you followed to reproduce this
> situation? If just one server goes down and comes back, it should not
> be resulting in a split brain situation. And if it indeed resulted in
> a split brain situation, it would not heal unless you specified the
> favorite child option (which comes with an elaborate warning). Is it
> that in your situation you indeed specified a favorite child and it
> got used in a wrongly diagnosed split brain?
> 
> A bug report with the debug logs and steps to reproduce will be
> greatly appreciated.
> 
> Avati

Hello,

the situation was really very simple. We drove a simple replicate setup with
one client and two servers with one server down for testing.
This went for about 10 days. In the background we rsync'ed the second server
from an old backup (some TB) hoping that self-heal will go a lot faster if
only the few new files have to be replicated.
Then we switched on glusterfsd and read this in the client logs:

[2009-09-12 16:59:48] N [client-protocol.c:5559:client_setvolume_cbk] remote2:
Connected to 192.168.82.2:6996, attached to remote volume 'p3user'.
[2009-09-12 16:59:48] N [client-protocol.c:5559:client_setvolume_cbk] remote2:
Connected to 192.168.82.2:6996, attached to remote volume 'p3user'.
[2009-09-12 17:00:03] E [afr-self-heal-data.c:858:afr_sh_data_fix] replicate:
Unable to self-heal contents of 'XXX' (possible split-brain). Please delete
the file from all but the preferred subvolume.
[2009-09-12 17:00:03] E [afr-self-heal-data.c:858:afr_sh_data_fix] replicate:
Unable to self-heal contents of 'YYY' (possible split-brain). Please delete
the file from all but the preferred subvolume.
[2009-09-12 17:00:03] E [afr-self-heal-data.c:858:afr_sh_data_fix] replicate:
Unable to self-heal contents of 'ZZZ' (possible split-brain). Please delete
the file from all but the preferred subvolume.

And so on...

These were files that were written at this point, and their content got
overwritten by older versions of the same file that resided on server remote2.
As told this happened to us before, but we did not fully understand the facts
back then. Now we know, no way round deleting before adding...

-- 
Regards,
Stephan