[Gluster-users] is glusterfs DHT really distributed?

[Wei Dong]

Your reply makes all sense to me.  I remember that auto-heal happens at 
file reading; doest that mean opening a file for read is also a global 
operation?  Do you mean that there's no other way of copying 30 million 
files to our 66-node glusterfs cluster for parallel processing other 
than waiting for half a month?  Can I somehow disable self-heal and get 
a seedup?

Things turn out to be too bad for me.

- Wei


Mark Mielke wrote:
> On 09/28/2009 10:35 AM, Wei Dong wrote:
>> Hi All,
>>
>> I noticed a very weird phenomenon when I'm copying data (200KB image 
>> files) to our glusterfs storage.  When I run only run client, it 
>> copies roughly 20 files per second and as soon as I start a second 
>> client on another machine, the copy rate of the first client 
>> immediately degrade to 5 files per second.   When I stop the second 
>> client, the first client will immediately speed up to the original 20 
>> files per second.  When I run 15 clients, the aggregate throughput is 
>> about 8 files per second, much worse than running only one client.  
>> Neither CPU nor network is saturated.  My volume file is attached.  
>> The servers are running on a 66 node cluster and the clients are a 
>> 15-node cluster.
>>
>> We have 33x2 servers and at most 15 separate machines, with each 
>> server serving < 0.5 clients on average.  I cannot think of a reason 
>> for a distributed system to behave like this.  There must be some 
>> kind of central access point.
>
> Although there is probably room for the GlusterFS folk to optimize...
>
> You should consider directory write operations to involve the whole 
> cluster. Creating a file is a directory write operation. Think of how 
> it might have to do self-heal across the cluster, make sure the name 
> is right and not already in use across the cluster, and such things.
>
> Once you get to reads and writes for a particular file, it should be 
> distributed.
>
> Cheers,
> mark
>