[Gluster-users] Need help with optimizing GlusterFS for Apache

[Joe Landman]

On 10/18/2011 06:14 AM, Robert Krig wrote:

> I think I'm going to have to abandon GlusterFS for our Image files. The
> performance is abysmal. I've tried all sorts of settings, but at some
> point the http process just keeps spawning more and more processess
> because clients are waiting because the directory can't be read, since
> glusterfs is busy.
> We're not even reaching 500 apache requests per second and already
> apache locks up.
>
> I'm pretty sure it can't be the hardware, since we're talking about a 12
> Core Hyperthreading Xeon CPU, with 48GB of ram and 30TB of storage in a
> hardware Raid.

 From our experience, and please don't take this incorrectly, the vast 
majority of storage users (and for that matter, storage companies) don't 
know how to design their RAIDs to their needs.  A "fast" CPU (12 core 
Xeon would be X5650 or higher) won't impact small file read speed all 
that much.  48 GB of ram could, if you can cache enough of your small files.

What you need, for your small random file read, is an SSD or Flash 
cache.  It has to be large enough that its relevant for your use case. 
I am not sure what your working set size is for your images, but you can 
buy them from small 300GB units through several 10s of TB.  Small random 
file performance is extremely good, and you can put gluster atop it as a 
file system if you wish to run the images off the cache ... or you can 
use it as a block level cache, which you then need to warm up prior to 
inital use (and then adjust after changes).

> I realise that GlusterFS is not ideal for many small files, but this is
> beyond ridiculous. It certainly doesn't help that the documentation
> doesn't even properly explain how to activate different translators, or
> where exactly to edit them by hand in the config files.
>
> If anyone has any suggestions, I'd be happy to hear them.

See above.  As noted, most people (and companies) do anywhere from a bad 
to terrible job on storage system design.  No one should be using a 
large RAID5 or RAID6 for small random file reads.  Its simply the wrong 
design.  I am guessing its unlikely that you have a RAID10, but even 
with that, you are going to be rate limited by the number of drives you 
have and their about 100 IOP rates.

This particular problem isn't likely Gluster's fault.  It is likely your 
storage design.  I'd suggest doing a quick test using fio to ascertain 
how many random read IOPs you can get out of your file system.  If you 
want to handle 500 apache requests per second, how many IOPs does this 
imply (how many files does each request require to fulfill)?  Chances 
are that you exceed the IOP capacity of your storage by several times.

Your best bet is either a caching system, or putting the small randomly 
accessed image files on SSD or Flash, and using that.  Try that before 
you abandon Gluster.



-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615