[Gluster-users] Gluster 2.0.3 + Apache on CentOS5 performance issue
somsaks at gmail.com
Mon Jul 13 04:04:53 UTC 2009
Sorry, I forgot to include informatino about my testbed.
4 machines - Gluster (as described below)
3 machines - WWW server, load balance using LVS
1 machine - LVS direct routing and weighted lease connection
3 machines - clients, 100 threads per client.
2009/7/12 Somsak Sriprayoonsakul <somsaks at gmail.com>
> We have been evaluating the choice for the new platform for a webboard
> The webboard is PHP scripts that generate/modify HTML page when user
> posting/add comment to the page, resulting topic is actually stored as a
> HTML file with all related file (file attach to the topic, etc.. )stored in
> its own directory for each topic. In general, the web site mostly serve a
> lot of small static files using Apache while using PHP to do other dynamic
> contents. This system has been working very well in the past, with the
> increasing page view rate, it is very likely that we will need some kind of
> Cluster file system as backend very soon.
> We have set up a test system using Grinder as stress test tool. The test
> system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock Apache
> (prefork, since the goal is to use this with PHP), linked together with
> Gigabit Ethernet. We try to compare the performance of either using single
> NFS server in sync mode against using 4 Gluster nodes (distribute of 2
> replicated nodes) through Fuse. However, the transaction per second (TPS)
> result is not good.
> NFS (single server, sync mode)
> - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, mean
> response time = 61.63 ms
> - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33 ms
> - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms
> - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt = 287.33ms
> 4 nodes Gluster (2 distribute of replicated 2 node)
> - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt = 207.33ms
> - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt = 672.67ms
> - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = 931.33
> (no 400-600 threads since we run out of client machine, sorry).
> gfsd is configured to use 32 thread of iothread as brick. gfs-client is
> configured to use io-cache->write-behind->readahead->distribute->replicate.
> io-cache cache-size is 256MB. I used patched Fuse downloaded from Gluster
> web-site (build through DKMS).
> As the result yield, it seems that Gluster performance worse with
> increasing no. of client. One observation is that the glusterfs process on
> client is taking about 100% of CPU during all the tests. glusterfsd is
> utilizing only 70-80% of CPUs during the test time. Note that system is Dual
> I also tried using modglusterfs and not using fuse at all to serve all the
> static files and conduct another test with Grinder. The result is about the
> same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in this test that
> each Apache prefork process used more about twice more memory and we need to
> lower number of httpd processes by about half.
> I tried disable EnableMMAP and it didn't help much. Adjusting readahead,
> write behind according to GlusterOptimization page didn't help much either.
> My question is, there seems to be bottleneck in this setup, but how can I
> track this? Note that, I didn't do any other optimization other than what
> said above. Are there any best practice configuration for using Apache to
> serve a bunch of small static files like this around?
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Gluster-users