[Gluster-users] Need help with optimizing GlusterFS for Apache
landman at scalableinformatics.com
Tue Oct 18 11:14:57 UTC 2011
On 10/18/2011 06:14 AM, Robert Krig wrote:
> I think I'm going to have to abandon GlusterFS for our Image files. The
> performance is abysmal. I've tried all sorts of settings, but at some
> point the http process just keeps spawning more and more processess
> because clients are waiting because the directory can't be read, since
> glusterfs is busy.
> We're not even reaching 500 apache requests per second and already
> apache locks up.
> I'm pretty sure it can't be the hardware, since we're talking about a 12
> Core Hyperthreading Xeon CPU, with 48GB of ram and 30TB of storage in a
> hardware Raid.
From our experience, and please don't take this incorrectly, the vast
majority of storage users (and for that matter, storage companies) don't
know how to design their RAIDs to their needs. A "fast" CPU (12 core
Xeon would be X5650 or higher) won't impact small file read speed all
that much. 48 GB of ram could, if you can cache enough of your small files.
What you need, for your small random file read, is an SSD or Flash
cache. It has to be large enough that its relevant for your use case.
I am not sure what your working set size is for your images, but you can
buy them from small 300GB units through several 10s of TB. Small random
file performance is extremely good, and you can put gluster atop it as a
file system if you wish to run the images off the cache ... or you can
use it as a block level cache, which you then need to warm up prior to
inital use (and then adjust after changes).
> I realise that GlusterFS is not ideal for many small files, but this is
> beyond ridiculous. It certainly doesn't help that the documentation
> doesn't even properly explain how to activate different translators, or
> where exactly to edit them by hand in the config files.
> If anyone has any suggestions, I'd be happy to hear them.
See above. As noted, most people (and companies) do anywhere from a bad
to terrible job on storage system design. No one should be using a
large RAID5 or RAID6 for small random file reads. Its simply the wrong
design. I am guessing its unlikely that you have a RAID10, but even
with that, you are going to be rate limited by the number of drives you
have and their about 100 IOP rates.
This particular problem isn't likely Gluster's fault. It is likely your
storage design. I'd suggest doing a quick test using fio to ascertain
how many random read IOPs you can get out of your file system. If you
want to handle 500 apache requests per second, how many IOPs does this
imply (how many files does each request require to fulfill)? Chances
are that you exceed the IOP capacity of your storage by several times.
Your best bet is either a caching system, or putting the small randomly
accessed image files on SSD or Flash, and using that. Try that before
you abandon Gluster.
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web : http://scalableinformatics.com
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
More information about the Gluster-users