[Gluster-devel] solutions for split brain situation

[Anand Avati]

> Our "split brain" is no real split brain and looks like this: Logfiles are
> written every 5 mins. If you add a secondary server that has 14 days old
> logfiles on it you notice that about half of your data vanishes while not
> successful self heal is performed, because the old logfiles read from the
> secondary server overwrite the new logfiles on your primary while new data is
> added to them.

Have you been using favorite-child option? Auto resolving of
split-brain is bound to make you lose data of one of the subvolumes.
If you had indeed specified favorite-child option, and the
favorite-child option happens to be the server which had 14day old
logs, what just happened was exactly what was in the elaborate warning
log.

Now what is more interesting for me is, the sequence of taking down
and bringing up the servers you followed to split brain? Was is really
just taking one server (any of them) down and bringing it back up? Did
you face a split brain with just this? Can you please describe the
minimal steps necessary to reproduce your issue?

Avati