[Gluster-devel] Client side AFR race conditions?

[Gordan Bobic]

Kevan Benson wrote:
> Gordan Bobic wrote:
>> Derek Price wrote:
>>
>>>> I'm not saying I don't want to see a more robust solution for client 
>>>> side AFR, just that each configuration has it's place, and client 
>>>> side AFR isn't currently (and may never be) capable of serving a 
>>>> share that requires high data integrity.
>>
>> As far as I can see, there is no practical difference in this regard 
>> between client and server-side AFR. Throw multiple clients at multiple 
>> servers, and you have the exact same problem.
> 
> Exactly my point.  The difference is that it's much easier to put a 
> constraint on the clients as to which servers they are allowed to talk 
> to.  Put a fail-over IP on the servers, and force all clients to address 
> one server.  It's one step in the direction of eliminating race 
> conditions.  It's not perfect, but it's far simpler (in my opinion) than 
> GFS with it's complexity, and the trade-off may be worth it in certain 
> circumstances.

I think the distributed "equal peers" approach is generally superior to 
single-master, multiple-slaves approaches. For a start, the 
master-failure condition can be dealt with much more gracefully when 
there is no discreet master. This also makes it more scalable and more 
redundant. Graceful degradation is an important aspect.

>>>> If you think fixing this current issue will solve your problems, 
>>>> maybe you haven't considered the implications of connectivity 
>>>> problems between some clients and some (not all) servers...  Add in 
>>>> some clients with slightly off timestamps and you might have some 
>>>> major problems WITHOUT any reboots.
>>
>> Exactly what I'm thinking. But then we're back to a tightly coupled 
>> cluster FS design like GFS or OCFS2: implicit write locking, 
>> journalled metadata for replay, and quorum requirements. Just about 
>> the only thing that can be sanely avoided is mandatory fencing, and 
>> that only because there is no shared storage, so one node going nuts 
>> cannot trash the entire FS after the other nodes boot it out.
>>
>> If anybody can come up with an idea of how to achieve guaranteed 
>> consistency across all nodes without the above, I'd love to hear about 
>> it.
>>
>>> Am I getting this straight?  Even with server-side AFR, you get 
>>> mirrors, but if all the clients aren't talking to the same server 
>>> then there is no forced synchronization going on?  How hard would it 
>>> be to implement some sort of synchronization/locking layer over AFR 
>>> such that reads and writes could still go to the nearest (read: 
>>> fastest) possible server yet still be guaranteed to be in sync?
>>
>> You'd need global implicit write locks.
>>
>>> In other words, the majority of servers would know of new version 
>>> numbers being written anywhere and yet reads would always serve local 
>>> copies (potentially after waiting for synchronization).
>>
>> Unlocked files can always be read. Read-locked files can always be 
>> read. Write-locked files can, in theory, be neither read nor written 
>> until unlocked, because there is no guaranteed consistency. The write 
>> lock also cannot be released until the metadata is journalled and all 
>> the nodes have acknowledged the write back.
>>
>> The problem with this is that you still have to verify on every read 
>> that there are no current write-locks in place. With strong quorum 
>> requirements and write-lock synchronisation, this could potentially be 
>> done away with, though, if you can guarantee that all connected nodes 
>> will always be aware of all write locks, and can acknowledge them 
>> before the lock is granted. This would mean, theoretically, NFS / 
>> local read performance, with an unavoidable write overhead. But at 
>> least this is not too bad because under normal circumstances there 
>> will be some orders of magnitude more reads than writes.
>>
>>> The application I'm thinking of is virtualized read/write storage.  
>>> For example, say you want to share some sort of data repository with 
>>> offices in Europe, India, and the U.S. and you only have slow links 
>>> connecting the various offices.  You would want all client access to 
>>> happen against a local mirror, and you would want to restrict traffic 
>>> between the mirrors to that absolutely required for locking and data 
>>> synchronization.
>>
>> The data transfers are already minimized if you have server-side AFR 
>> set up between sites (one mirror server on each site, with multiple 
>> clients at each site only connecting to the local server).
>>
>>> The only thing I'm not quite sure of in this model is what to do if 
>>> the server processing a write operation crashes before the write 
>>> finishes. I wouldn't want reads against the other mirrors to have to 
>>> wait indefinitely for the crashed server to return, so the best I can 
>>> come up with is that "write locks" for any files that hadn't been 
>>> mirrored to at least one available server before a crash would need 
>>> to be revoked on the first subsequent attempted access of the 
>>> unsynchronized file.  Then when the crashed server came back up and 
>>> tried to synchronize, it would find that its file wasn't the current 
>>> version and sync in the other direction.
>>
>> You heartbeat node status between the nodes. When a node drops out, 
>> the other nodes after a few seconds' grace period, boot it out, and 
>> release all it's locks. Unless a node has a lock, it's journal gets 
>> discarded on writes, so it cannot commit. Note that this means both 
>> file and directory metadata journalling, and they need to be combined 
>> in case of deleting and re-creating the same file name. If a file gets 
>> deleted, it should be safe to just say that file was deleted at 
>> version X in the directory metadata journal. All previous versions in 
>> the journal can be discarded, and the file metadata journal can also 
>> be reset to free up space. Until all connected nodes have acknowledged 
>> the journal commit, the file and directory metadata cannot be discarded.
>>
>> One exception could be where the number of file resync journal entries 
>> exceeds it's storage limit, and we mark the file for full resync. The 
>> only tricky part then is dynamically adding and removing nodes from 
>> the cluster, but that can be solved in a reasonably straightforward 
>> way, by on-line quorum adjustments.
>>
>> But the important point is that this will likely require a lot of 
>> time, effort and thought to implement. :-(
> 
> Thus the complexity of the other cluster file systems out there.

Indeed - it isn't a simple problem, but GlusterFS is interesting by the 
fact that it is distributed and more scaleable than similar solutions. 
It just seems to be in need of inheriting some of the features from the 
other, more complex cluster file systems to give it the ability to 
ensure consistency across the distributed data store. It turns out that 
consistency and incremental updates for fast syncing are not issues as 
separate as they may have originally appeared - both can be solved by 
journaling.

> Thanks for the fairly in-depth assessment of the problems.  That should 
> clear the air a bit.

Glad I can help. In case you couldn't tell, I've been up to my eyeballs 
in various cluster file systems for quite a while now.

Gordan