[Gluster-devel] Client side AFR race conditions?

[Kevan Benson]

Derek Price wrote:
> Kevan Benson wrote:
> 
>> I'm not saying I don't want to see a more robust solution for client 
>> side AFR, just that each configuration has it's place, and client side 
>> AFR isn't currently (and may never be) capable of serving a share that 
>> requires high data integrity.
>>
>> If you think fixing this current issue will solve your problems, maybe 
>> you haven't considered the implications of connectivity problems 
>> between some clients and some (not all) servers...  Add in some 
>> clients with slightly off timestamps and you might have some major 
>> problems WITHOUT any reboots.
> 
> Am I getting this straight?  Even with server-side AFR, you get mirrors, 
> but if all the clients aren't talking to the same server then there is 
> no forced synchronization going on?  How hard would it be to implement 
> some sort of synchronization/locking layer over AFR such that reads and 
> writes could still go to the nearest (read: fastest) possible server yet 
> still be guaranteed to be in sync?

Server side AFR should be susceptible to the same problems as client 
side AFR in clients can use arbitrary servers.  E.g. Client A writes to 
Server A for file X at the same time Client B writes to Server B for 
file X.  Server A and B are essentially "clients" to the AFR, so the 
same race condition should exist.  Possibly even exacerbated due to the 
speed difference in local verses remote AFR sub-volumes.

> In other words, the majority of servers would know of new version 
> numbers being written anywhere and yet reads would always serve local 
> copies (potentially after waiting for synchronization).  The application 
> I'm thinking of is virtualized read/write storage.  For example, say you 
> want to share some sort of data repository with offices in Europe, 
> India, and the U.S. and you only have slow links connecting the various 
> offices.  You would want all client access to happen against a local 
> mirror, and you would want to restrict traffic between the mirrors to 
> that absolutely required for locking and data synchronization.
> 
> The only thing I'm not quite sure of in this model is what to do if the 
> server processing a write operation crashes before the write finishes. I 
> wouldn't want reads against the other mirrors to have to wait 
> indefinitely for the crashed server to return, so the best I can come up 
> with is that "write locks" for any files that hadn't been mirrored to at 
> least one available server before a crash would need to be revoked on 
> the first subsequent attempted access of the unsynchronized file.  Then 
> when the crashed server came back up and tried to synchronize, it would 
> find that its file wasn't the current version and sync in the other 
> direction.

I would think a specialized translator would work great for this. 
Something optimized for the server, where it intercepts writes and 
creates binary diffs for syncing instead of copying the whole file.  In 
essence, trade computing power for bandwidth.  That doesn't help right 
now though, and it doesn't address locking.

The only way I see to ensure data integrity is to have some arbiter vet 
all writes.  You can try to make that arbiter redundant, but good luck 
making it actually distributed.


-- 

-Kevan Benson
-A-1 Networks