[Gluster-users] How does replication work?

[Mark Mielke]

On 09/08/2009 01:01 PM, Alan Ivey wrote:
> This is for running some "cloud" servers where we want all files available on each machine locally so all servers have the same files but can still get local performance. I don't think I'll need to run a cron like that but it's not my network so I'm trying to figure out how to get all of the gears working together.
>    

Note that unless you are doing mostly read - or unless you have GigE or 
better connections to each of these servers - you are not going to 
"still get local performance". As stat() calls are distributed to 
multiple machines, and write() operations would require replication to 
multiple machines, these will be significantly slower than local disk, 
and significantly slower than NFS (which does not do replication). The 
more machines you have, the worse it will get.

For example, in a test I just did, where we only have 100 Mbit/s between 
the nodes right now, with a 3-node replication cluster, I was only 
getting 5 Mbyte/s writes, but 70 - 130 Mbyte/s reads. Why? Because my 
write to the one server needed to be replicated to 2 other nodes, and if 
we divide 100 Mbit/s transmit speed by 2, we get 50 Mbit/s or 5 Mbyte/s 
to each. If you are going to have a "cloud" of 10 servers instead of 3 - 
this means that every write needs to be sent to all 10, or 9 other nodes 
(depending on if the client is on the server), which would divide your 
network upload capacity by 10, or 9. Go up to 100, and it gets even worse.

The machines themselves - every time I do a write, every node in the 
replication cluster would do a write. They're all working in unison. So, 
if 10 machines in the cluster are all issuing writes, then 10X as many 
writes are all happening to local disk, which means 10X as many seeks, 
and 10X fewer I/O throughput to the disk.

I want to make sure you understand that clustering in this way is not 
really giving you the ability to have each machine with local disk 
speeds. For most uses, clustering is really providing fail over / 
redundancy capabilities. However, if your workload is entirely dominated 
by reads, and very few writes, than this would also provide effective 
load balancing capabilities.


> I did just discover that changing permissions does not seem to heal. I had two servers up, killed glusterfsd on server2, on client I changed the permissions of a file, brought server2 back online, ran ls on client, and server2 still had the old permissions. It's not really a big issue since I don't see myself changing permissions often, esp with any servers down, but interesting nonetheless.

I think those settings are all configurable - I haven't played with them 
myself. More checks at run time = more expensive at run time.

Cheers,
mark

-- 
Mark Mielke<mark at mielke.cc>