[Gluster-users] How does replication work?

[Mark Mielke]

On 09/08/2009 04:14 AM, Daniel Maher wrote:
> Alan Ivey wrote:
>> Like the subject implies, how does replication work exactly?
>>
>> If a client is the only one that has the IP addresses defined for the 
>> servers, does that mean that only a client writing a file ensures 
>> that it goes to both servers? That would tell me that the servers 
>> don't directly communicate with each other for replication.
>>
>> If so, how does healing work? Since the client is the only 
>> configuration with the multiple server IP addresses, is it the 
>> client's "task" to make sure the server heals itself once it's back 
>> online?
>>
>> If not, how do they servers know each other exist if not for the 
>> client config file?
>
> You've answered your own question. :)  AFAIK, in the recommended 
> simple replication scenario, the client is actually responsible for 
> replication, as each server is functionally independant.
> (This seems crazy to me, but yes, that's how it works.)

For Alan: Active healing should only be necessary if the system is not 
working properly. Healing should only be required after a system crash 
or bug, a GlusterFS server or client crash or bug, or somebody messing 
around with the backing store file system underneath. For systems that 
are up and running without problems, healing should be completely 
unnecessary.

For Daniel: For the seems crazy, compared to what? Every time I look at 
other solutions such as Lustre and see how they rely on a single 
metadata server, that itself is supposed to be highly available using 
other means, I have to ask, are they really solving the highly 
availability problem, or are they just narrowing the scope? If the whole 
cluster of 2 to 1000 nodes is relying on a single server to being up, 
this is the weakest link. Sure, having one weakest link to deal with is 
easier to solve using traditional means that having 1000 weakest links, 
but it seems clear that Lustre has not SOLVED the problem. They've just 
reduced it to something that might be more manageable. Even the 
"traditional means" of shared disk storage such as GFS and OCFS rely on 
a single piece of hardware - the shared storage. As a result, they make 
the shared storage really expensive - dual interfaces, dual power 
supplies, dual disks, ... but it's still one piece of hardware that 
everything else is reliant on.

For "shared nothing", each node really does need to be fully independent 
and able to make its own decisions. I think the GlusterFS folk have the 
model right in this regard.

The remaining question is whether they have the *implementation* right. :-)

Right now they seem to be in a compromised position between simplicity, 
performance, and correctness. It seems it is a difficult problem to have 
all three no matter which model is selected (shared disk, shared 
metadata only, shared nothing). The self-healing is a good feature, but 
they seem to be leaning on it to provide correctness, so that they can 
provide performance with some amount of simplicity. An example here is 
how directory listings come from "the first up server". In theory, we 
could have correctness through self-healing if directory listing always 
queried all servers. The combined directory listing would be shown, and 
self healing would kick off in the back ground. But, this would cost 
performance - as all servers in the cluster would be involved in 
directory listing. This is just one example.

I think GlusterFS has a lot of potential to close off on holes such as 
these. I don't think it would be difficult to add in things like an 
automatic election model for defining which machines are considered 
stable and the safest masters to use (simplest might be 'the one with 
the highest glusterfsd uptime'?), and having clients choose to pull 
things like directory listings only from the first stable / safest 
master, and having the non-stable / non-safe machines go into automatic 
full self-heal until they are back up-to-date with the master. In such a 
model, I'd like to see the locks being held against the stable/safe 
masters used for reads. Just throwing stuff out there...

For me, I'm looking at this as - I have a problem to solve, and very few 
solutions seem to meet my requirements. GlusterFS looks very close. Do I 
write my own, which would probably start out only solving my 
requirements, and since my requirements will probably grow, this would 
mean eventually writing something the size of GlusterFS? Or do I start 
looking in to this GlusterFS thing - point out the problems, and see if 
I can help?

I'm leaning towards the latter - try it out, point out the problems, see 
if I can help.

As it is, I think GlusterFS is very stable with sufficient performance 
for the requirements of most potential users. It's the people who are 
really trying to push it to its limits that are causing the majority of 
the breakage being reported here. For these people, which includes me, 
I've looked around - and the solutions out there that are competitive 
are either very expensive, or insufficient.

Cheers,
mark

-- 
Mark Mielke<mark at mielke.cc>