[Gluster-devel] Mirrored GlusterFS -- very poor read performance

[Joe Landman]

On 01/03/2014 01:59 AM, Mikhail T. wrote:
> [Please, CC replies to me directly as I am not subscribed to the list.
> Thank you.]
>
> Joe Landman wrote:
>>
>>     As mentioned above, four test-files were used for the benchmark:
>>
>>       1. Small static file - 429 bytes
>>       2. Larger static file - 93347 bytes
>>       3. Small PHP file (a single php call in it – to phpinfo() function).
>>          Although the file is small, its output was over 64Kb.
>>       4. Large PHP file (apc.php). Although the file is larger, its output
>>          was only about 12Kb.
>>
>> One of the questions I routinely ask our customers is what their
>> definitions of "small" and "large" are, as their definitions might not
>> match what I use for these terms. This is directly relevant in your
>> case. What you call "larger" is considered small for GlusterFS. You
>> want MB sized IOs to amortize the cost of the fuse system calls.
> But these files are representative of what our web-servers will be
> serving -- primarily. Though there may be an occasional video, mostly it
> is static HTML and "web-size" images, plus PHP-scripts...
>
> Are you saying, GlusterFS is not really for us? We expected to pay
> /some/ performance penalty for the features, but the actual numbers are
> causing a sticker-shock...

Possibly ... It has a range of use cases that make sense for users.  But 
your files fall mostly into what is considered the "small" to "tiny" 
scale for the system.  That means you are paying relatively huge 
overhead costs for relatively small data transport/storage.

> (Writing to GlusterFS is even more horrid -- extracting
> thunderbird-24.0.tar.bz2, for example, on a GlusterFS share takes almost
> 30 minutes, instead of seconds on local FS, but we have very few writes,
> and so were willing to ignore that.)

Hmmm .... this sounds like something else might be wrong.  I didn't 
comment on your use of LVM or ext4, but it is not unlikely that one of 
those layers are problematic.

>> Did you look at the CSW number? Is it also high?
> Around 8K per second, if I'm reading the output of vmstat correctly.

This is not what I expected ... you are not being context switched out 
of performance  This is about 250 microseconds per context switch ... a 
moderate but not heavy load.

Something else in your stack is problematic, though your files are 
definitely on the small side.  You have a number of layers, including 
the LVM.  LVM usually does its IO in terms of 4MB physical extents 
(though I believe this size to be tunable).  We might suggest looking at 
the LVM stats from the kernel.


>
>     -mi
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> https://lists.nongnu.org/mailman/listinfo/gluster-devel
>


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
twtr : @scalableinfo
phone: +1 734 786 8423 x121
cell : +1 734 612 4615