[Gluster-users] Gluster 2.0.3 + Apache on CentOS5 performance issue

Liam Slusser lslusser at gmail.com
Thu Jul 30 11:23:43 UTC 2009


You might want to wait until 2.0.5 as there is a ton of bug fixes to  
booster in that release.

Either way please let us know how it goes.

ls

On Jul 30, 2009, at 12:39 AM, Somsak Sriprayoonsakul  
<somsaks at gmail.com> wrote:

> Thank you very much for you reply
>
> At the time we used 2.0.3, and yes we used stock Apache from CentOS.  
> I will try 2.0.4 very soon to see if it's work.
>
> For Booster, it seems not working correctly for me. Booster  
> complains a lots of error with plain 'ls' command (but giving the  
> correct output). Also, with booster, Apache process refuse to start.  
> I will try 2.0.4 to see if it improves. If not, I will attach error  
> log next time.
>
>
> 2009/7/30 Raghavendra G <raghavendra at gluster.com>
> Hi Somsak,
>
> Sorry for the delayed reply. Below you've mentioned that you've  
> problems with apache and booster. Going forward, Apache over booster  
> will be the preferred approach. Can you tell us what version of  
> glusterfs you are using? And as I can understand you are using  
> apache 2.2, am I correct?
>
> regards,
> ----- Original Message -----
> From: "Liam Slusser" <lslusser at gmail.com>
> To: "Somsak Sriprayoonsakul" <somsaks at gmail.com>
> Cc: gluster-users at gluster.org
> Sent: Saturday, July 25, 2009 3:46:14 AM GMT +04:00 Abu Dhabi / Muscat
> Subject: Re: [Gluster-users] Gluster 2.0.3 + Apache on CentOS5  
> performance  issue
>
> I haven't tried an apples to apples comparison with Apache 
> +mod_gluster vs
> Apache+fuse+gluster however i do run both setups.  I load tested  
> both setups
> so to verified it could handle 4x our normal daily load and left it  
> at that.
>  I didn't actually compare the two (although that might be cool to do
> someday).
> I really like the idea of Apache+mod_gluster as I don't have to deal  
> with
> the whole fuse and mounting the filesystem.  It always scares me  
> having a
> public facing webserver with your whole backend fileshare mounted  
> locally.
>  Its very slick for serving content such as media files.  We serve  
> audio
> content to our CDN with a pair of Apache/mod_gluster servers - pushing
> 200-300mbit on average daily and everything works very well.
>
> We run an apache+fuse+gluster setup because we need to run some  
> mod_perl
> before serving the actual content.  However performance is still  
> very good.
>  We do around 50-100 requests (all jpeg images) per second off of a  
> fuse
> mount and everything works great.  We also have a java tomcat+fuse 
> +gluster
> service which does image manipulation on the fly off of a gluster  
> mount.
>
> We have two backend gluster servers using replication which serve  
> all this
> content.
>
> If you would like more information on our setup id be happy to share
> offline.  Just email me privately.
>
> thanks,
> liam
>
> On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul
> <somsaks at gmail.com>wrote:
>
> > Oh thank you, thought noone will reply me :)
> >
> > Have you tried Apache + Fuse over GlusterFS? How is the performance?
> >
> > Also, anyone in this mailing-list have tried Apache with booster?  
> I tried
> > it but Apache refuse to start (just hang and freeze).
> >
> > 2009/7/23 Liam Slusser <lslusser at gmail.com>
> >
> >
> >> We use mod_gluster and Apache
> >> 2.2 with good results.  We also ran into the same issue as you  
> that we ran out of memory past 150 threads even on a 8gig machine.   
> We got around this by compiling Apache using mpm-worker
> >> (threads) instead of prefork - it uses 1/4 as much ram with the  
> same number
> >> of connections (150-200) and everything has been running  
> smoothly.  I cannot
> >> see any performance difference except it using way less memory.
> >> liam
> >>
> >>
> >> On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul <
> >> somsaks at gmail.com> wrote:
> >>
> >>> Hello,
> >>>
> >>> We have been evaluating the choice for the new platform for a  
> webboard
> >>> system.
> >>> The webboard is PHP scripts that generate/modify HTML page when  
> user
> >>> posting/add comment to the page, resulting topic is actually  
> stored as a
> >>> HTML file with all related file (file attach to the topic,  
> etc.. )stored in
> >>> its own directory for each topic. In general, the web site  
> mostly serve a
> >>> lot of small static files using Apache while using PHP to do  
> other dynamic
> >>> contents. This system has been working very well in the past,  
> with the
> >>> increasing page view rate, it is very likely that we will need  
> some kind of
> >>> Cluster file system as backend very soon.
> >>>
> >>> We have set up a test system using Grinder as stress test tool.  
> The test
> >>> system is 11 machines of Intel Dual Core x86_64 CentOS5 with  
> stock Apache
> >>> (prefork, since the goal is to use this with PHP), linked  
> together with
> >>> Gigabit Ethernet. We try to compare the performance of either  
> using single
> >>> NFS server in sync mode against using 4 Gluster nodes  
> (distribute of 2
> >>> replicated nodes) through Fuse. However, the transaction per  
> second (TPS)
> >>> result is not good.
> >>>
> >>> NFS (single server, sync mode)
> >>>  - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066,  
> mean
> >>> response time = 61.63 ms
> >>>  - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt =  
> 87.33 ms
> >>>  - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms
> >>>  - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt =
> >>> 287.33ms
> >>>
> >>> 4 nodes Gluster (2 distribute of replicated 2 node)
> >>> - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt =  
> 207.33ms
> >>> - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt =  
> 672.67ms
> >>> - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt =  
> 931.33
> >>> (no 400-600 threads since we run out of client machine, sorry).
> >>>
> >>> gfsd is configured to use 32 thread of iothread as brick. gfs- 
> client is
> >>> configured to use io-cache->write-behind->readahead->distribute- 
> >replicate.
> >>> io-cache cache-size is 256MB. I used patched Fuse downloaded  
> from Gluster
> >>> web-site (build through DKMS).
> >>>
> >>> As the result yield, it seems that Gluster performance worse with
> >>> increasing no. of client. One observation is that the glusterfs  
> process on
> >>> client is taking about 100% of CPU during all the tests.  
> glusterfsd is
> >>> utilizing only 70-80% of CPUs during the test time. Note that  
> system is Dual
> >>> core.
> >>>
> >>> I also tried using modglusterfs and not using fuse at all to  
> serve all
> >>> the static files and conduct another test with Grinder. The  
> result is about
> >>> the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in  
> this test
> >>> that each Apache prefork process used more about twice more  
> memory and we
> >>> need to lower number of httpd processes by about half.
> >>>
> >>> I tried disable EnableMMAP and it didn't help much. Adjusting  
> readahead,
> >>> write behind according to GlusterOptimization page didn't help  
> much either.
> >>>
> >>> My question is, there seems to be bottleneck in this setup, but  
> how can I
> >>> track this? Note that, I didn't do any other optimization other  
> than what
> >>> said above. Are there any best practice configuration for using  
> Apache to
> >>> serve a bunch of small static files like this around?
> >>>
> >>> Regards,
> >>>
> >>> Somsak
> >>>
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Gluster-users mailing list
> >>> Gluster-users at gluster.org
> >>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
> >>>
> >>>
> >>
> >
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>


More information about the Gluster-users mailing list