[Gluster-devel] solutions for split brain situation

[Mark Mielke]

On 09/18/2009 06:44 AM, Stephan von Krawczynski wrote:
> Only, we talked about OPEN and not REMOVE. Your example comes up with a broken
> replicate situation for a remove and you correctly say that after the second
> subvolume goes alive again the remove should be completely done.
> You may well ask the journal for that, and it tells you that the file on the
> second should be removed. Now, if you enter the same situation for a local fed
> file on secondary I would simply suggest - since there is no journal telling
> you to remove - that the file should be valid and _not_ removed, but
> replicated to node 1.
> Since this decision can be taken based on the journal, both setups have a
> valid answer. Still there is no race. Open in first setup fails, open in
> second setup succeeds. Nevertheless both open tries need a stat to check for
> the files' existence. The first stat finds the file should be gone, the second
> stat replicates the file to node 1 and open can succeed. And guess what:
> exactly that happens on glusterfs. If you stat a file that is only available
> on secondary node, it gets replicated.
>    

I failed to mention another consideration - the journal doesn't live 
forever. In most systems, the journal is either removed or marked as 
"done" as the backend storage is successfully updated. In many systems, 
such as PostgreSQL, the journal can be thought of as an infinite size 
list of instructions that if processed in order, will result in the 
correct backing data. With this model, the journal is front and 
foremost. If the journal does not say the file was created, then the 
file shouldn't be considered to exist even if it is found in the backing 
data.

That all said - the GlusterFS representative responded they have chosen 
to error on the side of "conservative", where they choose to keep the 
file if they cannot find proof that it should be removed, which 
unintentionally supports your model. This being the case, it does lead 
to supporting your position, as you are also looking for "conservative" 
behaviour in the case of an error path during self-heal and backend 
consistency checks.


>> The point here, is that the journal SHOULD be consulted.
>>      
> You omitted the most important word: "too". The journal should be consulted
> too. Nevertheless it cannot be the only reason for decision.
>    

If this was a database system, the journal trumps the data every time, 
and nothing should go into backend storage without going through the 
journal. But, if the authors want to relax this to support other models 
(such as a backend file system restore where the backup process, or the 
restore process (fsck) strips the extended attributes, effectively 
burning the journal into smoke and ash, then it  seems like you have a 
valid point. :-)

Cheers,
mark

-- 
Mark Mielke<mark at mielke.cc>