[Gluster-users] Configuration suggestions (aka poor/slow performance on new hardware)

[Ed W]

On 26/03/2010 18:22, Ramiro Magallanes wrote:
>
> You coud run the genfiles script simultaneuosly (my english is really
> poor, we can change the subject of this mail for something like "poor
> performance and poor english" xDDD) but its not like a thread aplication
> (iozone rulez).
>
> If I run 3 process of genfiles.sh i get 440, 441, and 450 files. (1300
> files aprox.) but if you add some more procees you're not going to
> obtain any big number :)
>
> With 6 genfiles at the same time i have :
>
> PID 12832 : 249 files created in 60 seconds.
> PID 12830 : 249 files created in 60 seconds.
> PID 12829 : 248 files created in 60 seconds.
> PID 12827 : 262 files created in 60 seconds.
> PID 12828 : 252 files created in 60 seconds.
> PID 12831 : 255 files created in 60 seconds.
>
> 1515 files .
>    

Just speaking theoretically, but I believe that without a writebehind 
cache on the client side then gluster is required to effectively "sync" 
after each file operation (well it's probably only half a sync, but some 
variation of this problem), this is safe, but of course decreases 
writespeed to be something which is a function of the network latency.

So in your case if you had say around 1ms of latency then you would be 
limited to around 1,000 operations per second simply due to the wait 
until the far side acks the operation.  This seems to correlate with the 
figures you are seeing (show your ping speed and correlate it with IOs 
per sec?)

I don't see this as a gluster issue - it's a fundamental limitation of 
whether you want an ack for network based operations?  Many people 
switch to fiberchannel or similar for the io for exactly this reason.  
If you can drop the latency by a factor of 10 then you are increasing 
your IOs by a factor of 10.

Untested, but at least theoretically switching on writeback caching on 
the client should mean that it ploughs on without waiting for network 
latency to give you your ack.  Lots of potential issues, but if this is 
ok for your requirements then give it a try?


Note, just an idea for the gluster guys, but I think I saw in AFS (or 
was it something else?) a kind of hybrid server side writeback cache.  
The idea was that the server could ack the write if a certain number of 
storage nodes at least had the pending IO in memory, even if it hadn't 
hit the disk yet.  This is subtly different to server side writeback, 
but seems like a very neat idea.  Note it's probably not relevant to 
small file creation tests like above, but for other situations

I do think some of the benchmarks here might not be really addressing 
network latency as the limiting bottleneck?

Good luck

Ed W