[Gluster-users] Gluster 2.0.3 + Apache on CentOS5 performance issue
Shehjar Tikoo
shehjart at gluster.com
Tue Aug 4 06:16:52 UTC 2009
Somsak Sriprayoonsakul wrote:
> I have run html read benchmark test using exactly the same old testset.
>
> - Runing normal httpd (prefork) over NFS-sync option - Average TPS ~ 1500,
> Peak TPS ~ 3500
>
> - Running normal httpd (prefork) using booster - Average TPS < 500, Peak TPS
> ~ 1200, slight improvements but each HTTPD still eat up about 50MB+ (keep
> increasing) memory instead of about 20MB without booster. I have to cut the
> number of httpd process by half, but still the process used up some swap
> space.
>
> - Running httpd.worker using booster - The benchmark result is very low and
> error rate is very high. There seems to be some trouble in this mode.
> - Here's what I did
> - Changed from httpd to httpd.worker in /etc/sysconfig/httpd
> - Disable PHP4
> - LD_PRELOAD in /etc/init.d/httpd then start httpd
> - The HTTPD start correctly and seems to work ok, but not all the time. Two
> consecutive wget's on exactly the same URL yield different results
Hi
I can see a few reasons why this could happen with recent
releases. If possible, could you please try the same test
with the release below. In this release, I've fixed a few bugs that
could result in behaviour seen in your tests.
http://ftp.gluster.com/pub/gluster/glusterfs/qa-releases/glusterfs-2.0.6rc2.tar.gz
Thanks
Shehjar
>
> [root at compute-0-9 ~]# wget -v --header "Host: www.myhost.local" -O /dev/null
> http://c0-3/cafe/siam/topic/F7800428/F7800428.html
> --15:52:12-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html
> Resolving c0-3... 10.1.255.251
> Connecting to c0-3|10.1.255.251|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 264373 (258K) [text/html]
> Saving to: `/dev/null'
>
> 100%[====================================================================================================================>]
> 264,373 --.-K/s in 0.003s
>
> 15:52:12 (83.3 MB/s) - `/dev/null' saved [264373/264373]
>
> [root at compute-0-9 ~]# wget -v --header "Host: www.myhost.local" -O /dev/null
> http://c0-3/cafe/siam/topic/F7800428/F7800428.html
> --15:52:15-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html
> Resolving c0-3... 10.1.255.251
> Connecting to c0-3|10.1.255.251|:80... connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 0 [text/html]
> Saving to: `/dev/null'
>
> [
> <=>
> ] 0 --.-K/s in 0s
>
> 15:52:15 (0.00 B/s) - `/dev/null' saved [0/0]
>
> [root at compute-0-9 ~]#
>
> (Note the returned content-length. Returned contents, when corrected, is
> ok)
>
> Here's the error log in booster log file
>
> (First wget)
>
> [2009-08-03 15:36:21] D [libglusterfsclient.c:1340:libgf_vmp_search_entry]
> libglusterfsclient: VMP Entry found: /usr/home/www/
> www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html: /usr/home/
> [2009-08-03 15:36:21] D
> [libglusterfsclient-dentry.c:381:libgf_client_path_lookup]
> libglusterfsclient: resolved path(/www/
> www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html) to
> 2626532/2650924
> [2009-08-03 15:36:21] D [libglusterfsclient.c:1340:libgf_vmp_search_entry]
> libglusterfsclient: VMP Entry found: /usr/home/www/
> www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html: /usr/home/
> [2009-08-03 15:36:21] D
> [libglusterfsclient-dentry.c:381:libgf_client_path_lookup]
> libglusterfsclient: resolved path(/www/
> www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html) to
> 2626532/2650924
>
> (second wget)
>
> [2009-08-03 15:36:26] D [libglusterfsclient.c:1340:libgf_vmp_search_entry]
> libglusterfsclient: VMP Entry found: /usr/home/www/
> www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html: /usr/home/
> [2009-08-03 15:36:26] D
> [libglusterfsclient-dentry.c:381:libgf_client_path_lookup]
> libglusterfsclient: resolved path(/www/
> www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html) to
> 2626532/2650924
> [2009-08-03 15:36:26] D [libglusterfsclient.c:1340:libgf_vmp_search_entry]
> libglusterfsclient: VMP Entry found: /usr/home/www/
> www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html: /usr/home/
> [2009-08-03 15:36:26] D
> [libglusterfsclient-dentry.c:381:libgf_client_path_lookup]
> libglusterfsclient: resolved path(/www/
> www.myhost.com/webdoc/cafe/siam/topic/F7800428/F7800428.html) to
> 2626532/2650924
>
> In server logs, there are just connect-destroy-connection messages in the
> log. Nothing particularily wrong to me.
>
> No such failure occur with Apache+Prefork.
>
> 2009/8/2 Somsak Sriprayoonsakul <somsaks at gmail.com>
>
>> After moving the embed configuration file out from glusterfs, now httpd
>> boot up ok and the web seems to work now.
>>
>> However, comment posting is not working. It seems that the code that do the
>> html modification is not working in GlusterFS context. I found that the code
>> modify web page locally instead. So I think workaround for my case is to
>> mount glusterfs with fuse at the same path as booster.
>>
>> I will give it another benchmark and see how it goes.
>>
>> 2009/8/2 Somsak Sriprayoonsakul <somsaks at gmail.com>
>>
>> Ok, I have a chance to run booster over 2.0.4
>>> Please find the attach file for my configuration
>>>
>>> I did configure boost and try simple ls over my Gluster file system.
>>> Here's the output of ls -al
>>>
>>> [root at compute-0-3 ~]#
>>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so ls -l /gluster/www/
>>> ls: /gluster/www/: Invalid argument
>>> ls: /gluster/www/members.pantip.com: Invalid argument
>>> ls: /gluster/www/cafe.pantip.com: Invalid argument
>>> ls: /gluster/www/admin.pantip.com: Invalid argument
>>> ls: /gluster/www/www.pantip.com: Invalid argument
>>> ls: /gluster/www/passwd3.sql: Invalid argument
>>> ls: /gluster/www/passwd.sql: Invalid argument
>>> ls: closing directory /gluster/www/: File descriptor in bad state
>>> total 129972
>>> drwxr-xr-x 3 root root 8192 May 11 16:13 admin.pantip.com
>>> drwxr-xr-x 5 root root 8192 May 18 11:11 cafe.pantip.com
>>> drwxr-xr-x 3 root root 8192 May 11 18:48 members.pantip.com
>>> -rw-r--r-- 1 root root 66654820 May 18 10:50 passwd3.sql
>>> -rw-r--r-- 1 root root 66225769 May 18 10:33 passwd.sql
>>> drwxr-xr-x 11 apache apache 8192 May 18 09:47 www.pantip.com
>>> [root at compute-0-3 ~]#
>>>
>>> [root at compute-0-3 ~]#
>>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so cp /etc/issue /gluster/
>>> [root at compute-0-3 ~]#
>>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so ls -l /gluster/issue
>>> ls: /gluster/issue: Invalid argument
>>> -rw-r--r-- 1 root root 47 Aug 2 14:57 /gluster/issue
>>> [root at compute-0-3 ~]#
>>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so cat /gluster/issue
>>> CentOS release 5.3 (Final)
>>> Kernel \r on an \m
>>>
>>> [root at compute-0-3 ~]#
>>>
>>>
>>> Despite all those errors, output seems to be fine
>>>
>>> And this is what inside my booster.log
>>>
>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>>> libglusterfsclient: invalid argument: value
>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>>> libglusterfsclient: invalid argument: value
>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>>> libglusterfsclient: invalid argument: value
>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>>> libglusterfsclient: invalid argument: value
>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>>> libglusterfsclient: invalid argument: value
>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>>> libglusterfsclient: invalid argument: value
>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>>> libglusterfsclient: invalid argument: value
>>> [2009-08-02 14:56:27] E [libglusterfsclient.c:4194:__glusterfs_stat]
>>> libglusterfsclient: path lookup failed for (/hosts)
>>> [2009-08-02 14:56:37] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>>> libglusterfsclient: invalid argument: value
>>> [2009-08-02 14:57:00] E [libglusterfsclient.c:4194:__glusterfs_stat]
>>> libglusterfsclient: path lookup failed for (/issue)
>>> [2009-08-02 14:57:07] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>>> libglusterfsclient: invalid argument: value
>>>
>>> Then, I try to LD_PRELOAD apache (prefork). I change the target from
>>> /gluster to /usr/home instead (the web application needs it). Then I tried
>>> to strace the httpd process and found that httpd crash at the points where
>>> httpd tried to read configuration file stored on Gluster volume (bad file
>>> descriptor). I will try to move this configuration file some other places
>>> and test again.
>>>
>>> 2009/7/31 Raghavendra G <raghavendra at gluster.com>
>>>
>>> Hi,
>>>> On Thu, Jul 30, 2009 at 11:39 AM, Somsak
>>>> Sriprayoonsakul<somsaks at gmail.com> wrote:
>>>>> Thank you very much for you reply
>>>>>
>>>>> At the time we used 2.0.3, and yes we used stock Apache from CentOS. I
>>>> will
>>>>> try 2.0.4 very soon to see if it's work.
>>>>>
>>>>> For Booster, it seems not working correctly for me. Booster complains a
>>>> lots
>>>>> of error with plain 'ls' command (but giving the correct output). Also,
>>>> with
>>>>
>>>> Can you mail those errors?
>>>>
>>>>> booster, Apache process refuse to start. I will try 2.0.4 to see if it
>>>>> improves. If not, I will attach error log next time.
>>>> logs are very much appreciated.
>>>>
>>>>>
>>>>> 2009/7/30 Raghavendra G <raghavendra at gluster.com>
>>>>>> Hi Somsak,
>>>>>>
>>>>>> Sorry for the delayed reply. Below you've mentioned that you've
>>>> problems
>>>>>> with apache and booster. Going forward, Apache over booster will be
>>>> the
>>>>>> preferred approach. Can you tell us what version of glusterfs you are
>>>> using?
>>>>>> And as I can understand you are using apache 2.2, am I correct?
>>>>>>
>>>>>> regards,
>>>>>> ----- Original Message -----
>>>>>> From: "Liam Slusser" <lslusser at gmail.com>
>>>>>> To: "Somsak Sriprayoonsakul" <somsaks at gmail.com>
>>>>>> Cc: gluster-users at gluster.org
>>>>>> Sent: Saturday, July 25, 2009 3:46:14 AM GMT +04:00 Abu Dhabi / Muscat
>>>>>> Subject: Re: [Gluster-users] Gluster 2.0.3 + Apache on CentOS5
>>>> performance
>>>>>> issue
>>>>>>
>>>>>> I haven't tried an apples to apples comparison with Apache+mod_gluster
>>>> vs
>>>>>> Apache+fuse+gluster however i do run both setups. I load tested both
>>>>>> setups
>>>>>> so to verified it could handle 4x our normal daily load and left it at
>>>>>> that.
>>>>>> I didn't actually compare the two (although that might be cool to do
>>>>>> someday).
>>>>>> I really like the idea of Apache+mod_gluster as I don't have to deal
>>>> with
>>>>>> the whole fuse and mounting the filesystem. It always scares me
>>>> having a
>>>>>> public facing webserver with your whole backend fileshare mounted
>>>> locally.
>>>>>> Its very slick for serving content such as media files. We serve
>>>> audio
>>>>>> content to our CDN with a pair of Apache/mod_gluster servers - pushing
>>>>>> 200-300mbit on average daily and everything works very well.
>>>>>>
>>>>>> We run an apache+fuse+gluster setup because we need to run some
>>>> mod_perl
>>>>>> before serving the actual content. However performance is still very
>>>>>> good.
>>>>>> We do around 50-100 requests (all jpeg images) per second off of a
>>>> fuse
>>>>>> mount and everything works great. We also have a java
>>>> tomcat+fuse+gluster
>>>>>> service which does image manipulation on the fly off of a gluster
>>>> mount.
>>>>>> We have two backend gluster servers using replication which serve all
>>>> this
>>>>>> content.
>>>>>>
>>>>>> If you would like more information on our setup id be happy to share
>>>>>> offline. Just email me privately.
>>>>>>
>>>>>> thanks,
>>>>>> liam
>>>>>>
>>>>>> On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul
>>>>>> <somsaks at gmail.com>wrote:
>>>>>>
>>>>>>> Oh thank you, thought noone will reply me :)
>>>>>>>
>>>>>>> Have you tried Apache + Fuse over GlusterFS? How is the performance?
>>>>>>>
>>>>>>> Also, anyone in this mailing-list have tried Apache with booster? I
>>>>>>> tried
>>>>>>> it but Apache refuse to start (just hang and freeze).
>>>>>>>
>>>>>>> 2009/7/23 Liam Slusser <lslusser at gmail.com>
>>>>>>>
>>>>>>>
>>>>>>>> We use mod_gluster and Apache
>>>>>>>> 2.2 with good results. We also ran into the same issue as you that
>>>> we
>>>>>>>> ran out of memory past 150 threads even on a 8gig machine. We got
>>>> around
>>>>>>>> this by compiling Apache using mpm-worker
>>>>>>>> (threads) instead of prefork - it uses 1/4 as much ram with the
>>>> same
>>>>>>>> number
>>>>>>>> of connections (150-200) and everything has been running smoothly.
>>>> I
>>>>>>>> cannot
>>>>>>>> see any performance difference except it using way less memory.
>>>>>>>> liam
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul <
>>>>>>>> somsaks at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> We have been evaluating the choice for the new platform for a
>>>> webboard
>>>>>>>>> system.
>>>>>>>>> The webboard is PHP scripts that generate/modify HTML page when
>>>> user
>>>>>>>>> posting/add comment to the page, resulting topic is actually
>>>> stored as
>>>>>>>>> a
>>>>>>>>> HTML file with all related file (file attach to the topic, etc..
>>>>>>>>> )stored in
>>>>>>>>> its own directory for each topic. In general, the web site mostly
>>>>>>>>> serve a
>>>>>>>>> lot of small static files using Apache while using PHP to do other
>>>>>>>>> dynamic
>>>>>>>>> contents. This system has been working very well in the past, with
>>>> the
>>>>>>>>> increasing page view rate, it is very likely that we will need
>>>> some
>>>>>>>>> kind of
>>>>>>>>> Cluster file system as backend very soon.
>>>>>>>>>
>>>>>>>>> We have set up a test system using Grinder as stress test tool.
>>>> The
>>>>>>>>> test
>>>>>>>>> system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock
>>>>>>>>> Apache
>>>>>>>>> (prefork, since the goal is to use this with PHP), linked together
>>>>>>>>> with
>>>>>>>>> Gigabit Ethernet. We try to compare the performance of either
>>>> using
>>>>>>>>> single
>>>>>>>>> NFS server in sync mode against using 4 Gluster nodes (distribute
>>>> of 2
>>>>>>>>> replicated nodes) through Fuse. However, the transaction per
>>>> second
>>>>>>>>> (TPS)
>>>>>>>>> result is not good.
>>>>>>>>>
>>>>>>>>> NFS (single server, sync mode)
>>>>>>>>> - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066,
>>>> mean
>>>>>>>>> response time = 61.63 ms
>>>>>>>>> - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33
>>>> ms
>>>>>>>>> - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms
>>>>>>>>> - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt =
>>>>>>>>> 287.33ms
>>>>>>>>>
>>>>>>>>> 4 nodes Gluster (2 distribute of replicated 2 node)
>>>>>>>>> - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt =
>>>> 207.33ms
>>>>>>>>> - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt =
>>>>>>>>> 672.67ms
>>>>>>>>> - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt =
>>>> 931.33
>>>>>>>>> (no 400-600 threads since we run out of client machine, sorry).
>>>>>>>>>
>>>>>>>>> gfsd is configured to use 32 thread of iothread as brick.
>>>> gfs-client
>>>>>>>>> is
>>>>>>>>> configured to use
>>>>>>>>> io-cache->write-behind->readahead->distribute->replicate.
>>>>>>>>> io-cache cache-size is 256MB. I used patched Fuse downloaded from
>>>>>>>>> Gluster
>>>>>>>>> web-site (build through DKMS).
>>>>>>>>>
>>>>>>>>> As the result yield, it seems that Gluster performance worse with
>>>>>>>>> increasing no. of client. One observation is that the glusterfs
>>>>>>>>> process on
>>>>>>>>> client is taking about 100% of CPU during all the tests.
>>>> glusterfsd is
>>>>>>>>> utilizing only 70-80% of CPUs during the test time. Note that
>>>> system
>>>>>>>>> is Dual
>>>>>>>>> core.
>>>>>>>>>
>>>>>>>>> I also tried using modglusterfs and not using fuse at all to serve
>>>> all
>>>>>>>>> the static files and conduct another test with Grinder. The result
>>>> is
>>>>>>>>> about
>>>>>>>>> the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in
>>>> this
>>>>>>>>> test
>>>>>>>>> that each Apache prefork process used more about twice more memory
>>>> and
>>>>>>>>> we
>>>>>>>>> need to lower number of httpd processes by about half.
>>>>>>>>>
>>>>>>>>> I tried disable EnableMMAP and it didn't help much. Adjusting
>>>>>>>>> readahead,
>>>>>>>>> write behind according to GlusterOptimization page didn't help
>>>> much
>>>>>>>>> either.
>>>>>>>>>
>>>>>>>>> My question is, there seems to be bottleneck in this setup, but
>>>> how
>>>>>>>>> can I
>>>>>>>>> track this? Note that, I didn't do any other optimization other
>>>> than
>>>>>>>>> what
>>>>>>>>> said above. Are there any best practice configuration for using
>>>> Apache
>>>>>>>>> to
>>>>>>>>> serve a bunch of small static files like this around?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Somsak
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Gluster-users mailing list
>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>>>
>>>>>>>>>
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>
>>>>
>>>> regards,
>>>> --
>>>> Raghavendra G
>>>>
>>>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
More information about the Gluster-users
mailing list