[Gluster-users] Concurrency limitation?

[Arnold Krille]

Hi,

On Tuesday 07 February 2012 14:09:33 Brian Candler wrote:
> I appear to be hitting a limitation in either the glusterfs FUSE client or
> the glusterfsd daemon, and I wonder if there are some knobs I can tweak.
> 
> I have a 12-disk RAID10 array. If I access it locally I get the following
> figures (#p = number of concurrent reader processes)
> 
>  #p  files/sec
>   1      35.52
>   2      66.13
>   5     137.73
>  10     215.51
>  20     291.45
>  30     337.01
> 
> If I access it as a single-brick distributed glusterfs volume over 10GE I
> get the following figures:
> 
>  #p  files/sec
>   1      39.09
>   2      70.44
>   5     135.79
>  10     157.48
>  20     179.75
>  30     206.34
> 
> The performance tracks very closely the raw RAID10 performance at 1, 2 and 5
> concurrent readers.  However at 10+ concurrent readers it falls well below
> what the RAID10 volume is capable of.

I did some similar test but with a slower machine and slower disk.
The "problem" with distributed filesystems is the distributed locking. And even 
though your test volume is on one system only, access locking is not only done 
by the fs in kernel but additionally by the fuse-client and/or the glusterd. 
That imposes a limit. And when the volume stretches across several machines, 
even though the reading might be done from the local disk, the locking has to 
be synchronized across all brick-machines. Another limit.

And with gluster, its the client that does all the synchronization, thats why 
there is one fuse-thread on the client that will max out your cpu when running 
dbench and the likes no matter whether the volume is local or 
distributed/replicated or remote.

My conclusion from my tests so far is that the most rewarding target for 
optimizations is the fuse-client of glusterfs. And maybe the way it talks to 
its companions and to the bricks.

But I am not yet finished with my tests and I still hope that glusterfs proves 
usable for distributed vm-image storage.

Have fun,

Arnold
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120207/14ff75da/attachment.sig>