[Gluster-users] Gluster 2.0.3 + Apache on CentOS5 performance issue

[Liam Slusser]

I haven't tried an apples to apples comparison with Apache+mod_gluster vs
Apache+fuse+gluster however i do run both setups.  I load tested both setups
so to verified it could handle 4x our normal daily load and left it at that.
 I didn't actually compare the two (although that might be cool to do
someday).
I really like the idea of Apache+mod_gluster as I don't have to deal with
the whole fuse and mounting the filesystem.  It always scares me having a
public facing webserver with your whole backend fileshare mounted locally.
 Its very slick for serving content such as media files.  We serve audio
content to our CDN with a pair of Apache/mod_gluster servers - pushing
200-300mbit on average daily and everything works very well.

We run an apache+fuse+gluster setup because we need to run some mod_perl
before serving the actual content.  However performance is still very good.
 We do around 50-100 requests (all jpeg images) per second off of a fuse
mount and everything works great.  We also have a java tomcat+fuse+gluster
service which does image manipulation on the fly off of a gluster mount.

We have two backend gluster servers using replication which serve all this
content.

If you would like more information on our setup id be happy to share
offline.  Just email me privately.

thanks,
liam

On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul
<somsaks at gmail.com>wrote:

> Oh thank you, thought noone will reply me :)
>
> Have you tried Apache + Fuse over GlusterFS? How is the performance?
>
> Also, anyone in this mailing-list have tried Apache with booster? I tried
> it but Apache refuse to start (just hang and freeze).
>
> 2009/7/23 Liam Slusser <lslusser at gmail.com>
>
>
>> We use mod_gluster and Apache
>> 2.2 with good results.  We also ran into the same issue as you that we ran out of memory past 150 threads even on a 8gig machine.  We got around this by compiling Apache using mpm-worker
>> (threads) instead of prefork - it uses 1/4 as much ram with the same number
>> of connections (150-200) and everything has been running smoothly.  I cannot
>> see any performance difference except it using way less memory.
>> liam
>>
>>
>> On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul <
>> somsaks at gmail.com> wrote:
>>
>>> Hello,
>>>
>>> We have been evaluating the choice for the new platform for a webboard
>>> system.
>>> The webboard is PHP scripts that generate/modify HTML page when user
>>> posting/add comment to the page, resulting topic is actually stored as a
>>> HTML file with all related file (file attach to the topic, etc.. )stored in
>>> its own directory for each topic. In general, the web site mostly serve a
>>> lot of small static files using Apache while using PHP to do other dynamic
>>> contents. This system has been working very well in the past, with the
>>> increasing page view rate, it is very likely that we will need some kind of
>>> Cluster file system as backend very soon.
>>>
>>> We have set up a test system using Grinder as stress test tool. The test
>>> system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock Apache
>>> (prefork, since the goal is to use this with PHP), linked together with
>>> Gigabit Ethernet. We try to compare the performance of either using single
>>> NFS server in sync mode against using 4 Gluster nodes (distribute of 2
>>> replicated nodes) through Fuse. However, the transaction per second (TPS)
>>> result is not good.
>>>
>>> NFS (single server, sync mode)
>>>  - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, mean
>>> response time = 61.63 ms
>>>  - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33 ms
>>>  - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms
>>>  - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt =
>>> 287.33ms
>>>
>>> 4 nodes Gluster (2 distribute of replicated 2 node)
>>> - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt = 207.33ms
>>> - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt = 672.67ms
>>> - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = 931.33
>>> (no 400-600 threads since we run out of client machine, sorry).
>>>
>>> gfsd is configured to use 32 thread of iothread as brick. gfs-client is
>>> configured to use io-cache->write-behind->readahead->distribute->replicate.
>>> io-cache cache-size is 256MB. I used patched Fuse downloaded from Gluster
>>> web-site (build through DKMS).
>>>
>>> As the result yield, it seems that Gluster performance worse with
>>> increasing no. of client. One observation is that the glusterfs process on
>>> client is taking about 100% of CPU during all the tests. glusterfsd is
>>> utilizing only 70-80% of CPUs during the test time. Note that system is Dual
>>> core.
>>>
>>> I also tried using modglusterfs and not using fuse at all to serve all
>>> the static files and conduct another test with Grinder. The result is about
>>> the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in this test
>>> that each Apache prefork process used more about twice more memory and we
>>> need to lower number of httpd processes by about half.
>>>
>>> I tried disable EnableMMAP and it didn't help much. Adjusting readahead,
>>> write behind according to GlusterOptimization page didn't help much either.
>>>
>>> My question is, there seems to be bottleneck in this setup, but how can I
>>> track this? Note that, I didn't do any other optimization other than what
>>> said above. Are there any best practice configuration for using Apache to
>>> serve a bunch of small static files like this around?
>>>
>>> Regards,
>>>
>>> Somsak
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>>>
>>>
>>
>