[Gluster-devel] Mirrored GlusterFS -- very poor read performance

[Joe Landman]

On 01/02/2014 06:26 PM, Mikhail T. wrote:
> We are building a new web-serving farm here. Believing, like most
> people, that the choice of the technology does not affect performance in
> read-dominated work-loads (such as ours), we picked GlusterFS for its
> rich feature set.

[...]

> As mentioned above, four test-files were used for the benchmark:
>
>  1. Small static file - 429 bytes
>  2. Larger static file - 93347 bytes
>  3. Small PHP file (a single php call in it – to |phpinfo()| function).
>     Although the file is small, its output was over 64Kb.
>  4. Large PHP file (apc.php). Although the file is larger, its output
>     was only about 12Kb.

One of the questions I routinely ask our customers is what their 
definitions of "small" and "large" are, as their definitions might not 
match what I use for these terms.

This is directly relevant in your case.  What you call "larger" is 
considered small for GlusterFS.  You want MB sized IOs to amortize the 
cost of the fuse system calls.  kB sized IOs are what GlusterFS does not 
do well on.  If you do many of those small (in Gluster's conception of 
scale) IOs, you are going to wind up losing performance to the context 
switches.  You did note that CPU usage was very high.  Did you look at 
the CSW number?  Is it also high?

Basically, NFS and the local file system are quite effective at local 
write caching as your quote noted.  They also often have good read-ahead 
properties.  Small (from Gluster's view) IOs will suffer, be they reads 
or writes.  Large (from Gluster's view) IOs will be "reasonable" for a 
suitable definition of reasonable.



-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
twtr : @scalableinfo
phone: +1 734 786 8423 x121
cell : +1 734 612 4615