[Gluster-users] Gluster 2.0.3 + Apache on CentOS5 performance issue

Fri Jul 31 04:34:32 UTC 2009

Its not released yet, but it is in QA.  You can download it here:
http://ftp.gluster.com/pub/gluster/glusterfs/qa-releases/glusterfs-2.0.5.tar.gzor
grab the newest git which has all the changes in it.
liam

On Thu, Jul 30, 2009 at 8:45 PM, Somsak Sriprayoonsakul
<somsaks at gmail.com>wrote:

> Could you let me know when will this be (estimately). I can wait until
> 2.0.5 and test it out again.
>
> 2009/7/30 Liam Slusser <lslusser at gmail.com>
>
> You might want to wait until 2.0.5 as there is a ton of bug fixes to
>> booster in that release.
>>
>> Either way please let us know how it goes.
>>
>> ls
>>
>> On Jul 30, 2009, at 12:39 AM, Somsak Sriprayoonsakul <somsaks at gmail.com>
>> wrote:
>>
>> Thank you very much for you reply
>>
>> At the time we used 2.0.3, and yes we used stock Apache from CentOS. I
>> will try 2.0.4 very soon to see if it's work.
>>
>> For Booster, it seems not working correctly for me. Booster complains a
>> lots of error with plain 'ls' command (but giving the correct output). Also,
>> with booster, Apache process refuse to start. I will try 2.0.4 to see if it
>> improves. If not, I will attach error log next time.
>>
>>
>> 2009/7/30 Raghavendra G < <raghavendra at gluster.com>
>> raghavendra at gluster.com>
>>
>>> Hi Somsak,
>>>
>>> Sorry for the delayed reply. Below you've mentioned that you've problems
>>> with apache and booster. Going forward, Apache over booster will be the
>>> preferred approach. Can you tell us what version of glusterfs you are using?
>>> And as I can understand you are using apache 2.2, am I correct?
>>>
>>> regards,
>>> ----- Original Message -----
>>> From: "Liam Slusser" < <lslusser at gmail.com>lslusser at gmail.com>
>>> To: "Somsak Sriprayoonsakul" < <somsaks at gmail.com>somsaks at gmail.com>
>>> Cc: <gluster-users at gluster.org>gluster-users at gluster.org
>>> Sent: Saturday, July 25, 2009 3:46:14 AM GMT +04:00 Abu Dhabi / Muscat
>>> Subject: Re: [Gluster-users] Gluster 2.0.3 + Apache on CentOS5
>>> performance  issue
>>>
>>> I haven't tried an apples to apples comparison with Apache+mod_gluster vs
>>> Apache+fuse+gluster however i do run both setups.  I load tested both
>>> setups
>>> so to verified it could handle 4x our normal daily load and left it at
>>> that.
>>>  I didn't actually compare the two (although that might be cool to do
>>> someday).
>>> I really like the idea of Apache+mod_gluster as I don't have to deal with
>>> the whole fuse and mounting the filesystem.  It always scares me having a
>>> public facing webserver with your whole backend fileshare mounted
>>> locally.
>>>  Its very slick for serving content such as media files.  We serve audio
>>> content to our CDN with a pair of Apache/mod_gluster servers - pushing
>>> 200-300mbit on average daily and everything works very well.
>>>
>>> We run an apache+fuse+gluster setup because we need to run some mod_perl
>>> before serving the actual content.  However performance is still very
>>> good.
>>>  We do around 50-100 requests (all jpeg images) per second off of a fuse
>>> mount and everything works great.  We also have a java
>>> tomcat+fuse+gluster
>>> service which does image manipulation on the fly off of a gluster mount.
>>>
>>> We have two backend gluster servers using replication which serve all
>>> this
>>> content.
>>>
>>> If you would like more information on our setup id be happy to share
>>> offline.  Just email me privately.
>>>
>>> thanks,
>>> liam
>>>
>>> On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul
>>> < <somsaks at gmail.com>somsaks at gmail.com>wrote:
>>>
>>> > Oh thank you, thought noone will reply me :)
>>> >
>>> > Have you tried Apache + Fuse over GlusterFS? How is the performance?
>>> >
>>> > Also, anyone in this mailing-list have tried Apache with booster? I
>>> tried
>>> > it but Apache refuse to start (just hang and freeze).
>>> >
>>> > 2009/7/23 Liam Slusser < <lslusser at gmail.com>lslusser at gmail.com>
>>> >
>>> >
>>> >> We use mod_gluster and Apache
>>> >> 2.2 with good results.  We also ran into the same issue as you that we
>>> ran out of memory past 150 threads even on a 8gig machine.  We got around
>>> this by compiling Apache using mpm-worker
>>> >> (threads) instead of prefork - it uses 1/4 as much ram with the same
>>> number
>>> >> of connections (150-200) and everything has been running smoothly.  I
>>> cannot
>>> >> see any performance difference except it using way less memory.
>>> >> liam
>>> >>
>>> >>
>>> >> On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul <
>>> >> <somsaks at gmail.com>somsaks at gmail.com> wrote:
>>> >>
>>> >>> Hello,
>>> >>>
>>> >>> We have been evaluating the choice for the new platform for a
>>> webboard
>>> >>> system.
>>> >>> The webboard is PHP scripts that generate/modify HTML page when user
>>> >>> posting/add comment to the page, resulting topic is actually stored
>>> as a
>>> >>> HTML file with all related file (file attach to the topic, etc..
>>> )stored in
>>> >>> its own directory for each topic. In general, the web site mostly
>>> serve a
>>> >>> lot of small static files using Apache while using PHP to do other
>>> dynamic
>>> >>> contents. This system has been working very well in the past, with
>>> the
>>> >>> increasing page view rate, it is very likely that we will need some
>>> kind of
>>> >>> Cluster file system as backend very soon.
>>> >>>
>>> >>> We have set up a test system using Grinder as stress test tool. The
>>> test
>>> >>> system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock
>>> Apache
>>> >>> (prefork, since the goal is to use this with PHP), linked together
>>> with
>>> >>> Gigabit Ethernet. We try to compare the performance of either using
>>> single
>>> >>> NFS server in sync mode against using 4 Gluster nodes (distribute of
>>> 2
>>> >>> replicated nodes) through Fuse. However, the transaction per second
>>> (TPS)
>>> >>> result is not good.
>>> >>>
>>> >>> NFS (single server, sync mode)
>>> >>>  - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, mean
>>> >>> response time = 61.63 ms
>>> >>>  - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33 ms
>>> >>>  - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms
>>> >>>  - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt =
>>> >>> 287.33ms
>>> >>>
>>> >>> 4 nodes Gluster (2 distribute of replicated 2 node)
>>> >>> - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt = 207.33ms
>>> >>> - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt =
>>> 672.67ms
>>> >>> - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = 931.33
>>> >>> (no 400-600 threads since we run out of client machine, sorry).
>>> >>>
>>> >>> gfsd is configured to use 32 thread of iothread as brick. gfs-client
>>> is
>>> >>> configured to use
>>> io-cache->write-behind->readahead->distribute->replicate.
>>> >>> io-cache cache-size is 256MB. I used patched Fuse downloaded from
>>> Gluster
>>> >>> web-site (build through DKMS).
>>> >>>
>>> >>> As the result yield, it seems that Gluster performance worse with
>>> >>> increasing no. of client. One observation is that the glusterfs
>>> process on
>>> >>> client is taking about 100% of CPU during all the tests. glusterfsd
>>> is
>>> >>> utilizing only 70-80% of CPUs during the test time. Note that system
>>> is Dual
>>> >>> core.
>>> >>>
>>> >>> I also tried using modglusterfs and not using fuse at all to serve
>>> all
>>> >>> the static files and conduct another test with Grinder. The result is
>>> about
>>> >>> the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in this
>>> test
>>> >>> that each Apache prefork process used more about twice more memory
>>> and we
>>> >>> need to lower number of httpd processes by about half.
>>> >>>
>>> >>> I tried disable EnableMMAP and it didn't help much. Adjusting
>>> readahead,
>>> >>> write behind according to GlusterOptimization page didn't help much
>>> either.
>>> >>>
>>> >>> My question is, there seems to be bottleneck in this setup, but how
>>> can I
>>> >>> track this? Note that, I didn't do any other optimization other than
>>> what
>>> >>> said above. Are there any best practice configuration for using
>>> Apache to
>>> >>> serve a bunch of small static files like this around?
>>> >>>
>>> >>> Regards,
>>> >>>
>>> >>> Somsak
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> _______________________________________________
>>> >>> Gluster-users mailing list
>>> >>> <Gluster-users at gluster.org>Gluster-users at gluster.org
>>> >>> <http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users>
>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>>> >>>
>>> >>>
>>> >>
>>> >
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>>  <Gluster-users at gluster.org>Gluster-users at gluster.org
>>>  <http://gluster.org/cgi-bin/mailman/listinfo/gluster-users>
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>
>>
>>
>