[Gluster-devel] solutions for split brain situation
mark at mark.mielke.cc
Fri Sep 18 14:39:20 UTC 2009
On 09/18/2009 06:44 AM, Stephan von Krawczynski wrote:
> Only, we talked about OPEN and not REMOVE. Your example comes up with a broken
> replicate situation for a remove and you correctly say that after the second
> subvolume goes alive again the remove should be completely done.
> You may well ask the journal for that, and it tells you that the file on the
> second should be removed. Now, if you enter the same situation for a local fed
> file on secondary I would simply suggest - since there is no journal telling
> you to remove - that the file should be valid and _not_ removed, but
> replicated to node 1.
> Since this decision can be taken based on the journal, both setups have a
> valid answer. Still there is no race. Open in first setup fails, open in
> second setup succeeds. Nevertheless both open tries need a stat to check for
> the files' existence. The first stat finds the file should be gone, the second
> stat replicates the file to node 1 and open can succeed. And guess what:
> exactly that happens on glusterfs. If you stat a file that is only available
> on secondary node, it gets replicated.
I failed to mention another consideration - the journal doesn't live
forever. In most systems, the journal is either removed or marked as
"done" as the backend storage is successfully updated. In many systems,
such as PostgreSQL, the journal can be thought of as an infinite size
list of instructions that if processed in order, will result in the
correct backing data. With this model, the journal is front and
foremost. If the journal does not say the file was created, then the
file shouldn't be considered to exist even if it is found in the backing
That all said - the GlusterFS representative responded they have chosen
to error on the side of "conservative", where they choose to keep the
file if they cannot find proof that it should be removed, which
unintentionally supports your model. This being the case, it does lead
to supporting your position, as you are also looking for "conservative"
behaviour in the case of an error path during self-heal and backend
>> The point here, is that the journal SHOULD be consulted.
> You omitted the most important word: "too". The journal should be consulted
> too. Nevertheless it cannot be the only reason for decision.
If this was a database system, the journal trumps the data every time,
and nothing should go into backend storage without going through the
journal. But, if the authors want to relax this to support other models
(such as a backend file system restore where the backup process, or the
restore process (fsck) strips the extended attributes, effectively
burning the journal into smoke and ash, then it seems like you have a
valid point. :-)
Mark Mielke<mark at mielke.cc>
More information about the Gluster-devel