[Gluster-users] Gluster 2.0.3 + Apache on CentOS5 performance issue
Somsak Sriprayoonsakul
somsaks at gmail.com
Mon Aug 17 08:17:55 UTC 2009
I may have send the wrong file. Here is what exactly put in booster.conf in
3 servers
/etc/glusterfs/glusterfs-client.vol /usr/home glusterfs
subvolume=cache,logfile=/var/log/glusterfs/usrhome.log
BTW, the problem occur randomly on the same node. Sometime I get the file,
sometimes it's not.
2009/8/14 Raghavendra G <raghavendra at gluster.com>
> Hi Somsak,
>
> Sorry for the delayed reply.
>
> I've tried to reproduce the issue you mentioned with 2.0.6 release but
> without success. Both siege and wget report no inconsistencies in file size.
> My configuration is httpd-2.2.11 (with mpm-worker), glusterfs 2.0.6.
>
> I also noticed that the virtual mount point (vmp) in your booster config is
> /gluster. I am curious to know your what is DOCUMENT_ROOT configured to in
> your httpd.conf. Because with DOCUMENT_ROOT configured to /usr/htdocs, the
> vmp should be /usr/htdocs/gluster. For your configuration to work, the
> DOCUMENT_ROOT should be '/'.
>
> Also, I am not finding the vmp specified in the url specified to wget. With
> DOCUMENT_ROOT '/usr/htdocs' and vmp '/usr/htdocs/gluster', a wget of the
> form,
>
> wget http:host/gluster/file
>
> will fetch the file from glusterfs mount point. On 'ls' on the gluster
> mount point (mounted using same volume specification file as in
> booster-config) you should be able to see 'file' listed.
>
> regards,
>
>
> On Thu, Aug 6, 2009 at 9:24 AM, Somsak Sriprayoonsakul <somsaks at gmail.com>wrote:
>
>> The behavior is still the same
>>
>> [root at compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local"
>> http://c0-3/cafe/siam/topic/F7800428/F7800428.html
>> --12:20:33-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html
>> Resolving c0-3... 10.1.255.251
>> Connecting to c0-3|10.1.255.251|:80... connected.
>> HTTP request sent, awaiting response... 200 OK
>> Length: 264373 (258K) [text/html]
>> Saving to: `/dev/null'
>>
>> 100%[====================================================================================================================>]
>> 264,373 --.-K/s in 0.003s
>>
>> 12:20:33 (81.8 MB/s) - `/dev/null' saved [264373/264373]
>>
>> [root at compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local"
>> http://c0-3/cafe/siam/topic/F7800428/F7800428.html
>> --12:20:35-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html
>> Resolving c0-3... 10.1.255.251
>> Connecting to c0-3|10.1.255.251|:80... connected.
>> HTTP request sent, awaiting response... 200 OK
>> Length: 0 [text/html]
>> Saving to: `/dev/null'
>>
>> [
>> <=>
>> ] 0 --.-K/s in 0s
>>
>> 12:20:35 (0.00 B/s) - `/dev/null' saved [0/0]
>>
>> [root at compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local"
>> http://c0-3/cafe/siam/topic/F7800428/F7800428.html
>> --12:20:38-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html
>> Resolving c0-3... 10.1.255.251
>> Connecting to c0-3|10.1.255.251|:80... connected.
>> HTTP request sent, awaiting response... 200 OK
>> Length: 0 [text/html]
>> Saving to: `/dev/null'
>>
>> [
>> <=>
>> ] 0 --.-K/s in 0s
>>
>> 12:20:38 (0.00 B/s) - `/dev/null' saved [0/0]
>>
>> [root at compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local"
>> http://c0-3/cafe/siam/topic/F7800428/F7800428.html
>> --12:20:39-- http://c0-3/cafe/siam/topic/F7800428/F7800428.html
>> Resolving c0-3... 10.1.255.251
>> Connecting to c0-3|10.1.255.251|:80... connected.
>> HTTP request sent, awaiting response... 200 OK
>> Length: 264373 (258K) [text/html]
>> Saving to: `/dev/null'
>>
>> 100%[====================================================================================================================>]
>> 264,373 --.-K/s in 0.003s
>>
>> 12:20:39 (82.3 MB/s) - `/dev/null' saved [264373/264373]
>>
>> [root at compute-0-0 ~]#
>>
>>
>> 4 consequtive wget yield different content length. Note that, this only
>> happened with worker + booster, prefork has no problem.
>>
>> And the web server seems to use correct version of glusterfs
>>
>> [root at compute-0-3 ~]# glusterfs -V
>> glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03
>> Repository revision: v2.0.5-13-g5e3ca25
>> Copyright (c) 2006-2009 Z RESEARCH Inc. <http://www.zresearch.com>
>> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>> You may redistribute copies of GlusterFS under the terms of the GNU
>> General Public License.
>> [root at compute-0-3 ~]#
>>
>> And all glusterfsd too
>>
>> [root at fe1 ~]# pexec -p storage glusterfs -V \| grep 2\.0\.6
>> compute-0-6: glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03
>> compute-0-8: glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03
>> compute-0-9: glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03
>> compute-0-7: glusterfs 2.0.6rc2 built on Aug 6 2009 12:02:03
>> [root at fe1 ~]#
>>
>> Don't sure if there is anyway to query version from running glusterfsd,
>> but I did /etc/init.d/glusterfsd restart already.
>>
>> 2009/8/3 Shehjar Tikoo <shehjart at gluster.com>
>>
>>> Somsak Sriprayoonsakul wrote:
>>>
>>>> Ok, I have a chance to run booster over 2.0.4
>>>>
>>>
>>>
>>> Have you tried configuring booster with the help doc available at:
>>> http://www.gluster.org/docs/index.php/BoosterConfiguration
>>>
>>> -Shehjar
>>>
>>>
>>>> Please find the attach file for my configuration
>>>>
>>>> I did configure boost and try simple ls over my Gluster file system.
>>>> Here's
>>>> the output of ls -al
>>>>
>>>> [root at compute-0-3 ~]#
>>>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
>>>> ls -l /gluster/www/
>>>> ls: /gluster/www/: Invalid argument
>>>> ls: /gluster/www/members.pantip.com: Invalid argument
>>>> ls: /gluster/www/cafe.pantip.com: Invalid argument
>>>> ls: /gluster/www/admin.pantip.com: Invalid argument
>>>> ls: /gluster/www/www.pantip.com: Invalid argument
>>>> ls: /gluster/www/passwd3.sql: Invalid argument
>>>> ls: /gluster/www/passwd.sql: Invalid argument
>>>> ls: closing directory /gluster/www/: File descriptor in bad state
>>>> total 129972
>>>> drwxr-xr-x 3 root root 8192 May 11 16:13 admin.pantip.com
>>>> drwxr-xr-x 5 root root 8192 May 18 11:11 cafe.pantip.com
>>>> drwxr-xr-x 3 root root 8192 May 11 18:48 members.pantip.com
>>>> -rw-r--r-- 1 root root 66654820 May 18 10:50 passwd3.sql
>>>> -rw-r--r-- 1 root root 66225769 May 18 10:33 passwd.sql
>>>> drwxr-xr-x 11 apache apache 8192 May 18 09:47 www.pantip.com
>>>> [root at compute-0-3 ~]#
>>>>
>>>> [root at compute-0-3 ~]#
>>>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
>>>> cp /etc/issue /gluster/
>>>> [root at compute-0-3 ~]#
>>>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
>>>> ls -l /gluster/issue
>>>> ls: /gluster/issue: Invalid argument
>>>> -rw-r--r-- 1 root root 47 Aug 2 14:57 /gluster/issue
>>>> [root at compute-0-3 ~]#
>>>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
>>>> cat /gluster/issue
>>>> CentOS release 5.3 (Final)
>>>> Kernel \r on an \m
>>>>
>>>> [root at compute-0-3 ~]#
>>>>
>>>>
>>>> Despite all those errors, output seems to be fine
>>>>
>>>> And this is what inside my booster.log
>>>>
>>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>>>> libglusterfsclient: invalid argument: value
>>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>>>> libglusterfsclient: invalid argument: value
>>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>>>> libglusterfsclient: invalid argument: value
>>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>>>> libglusterfsclient: invalid argument: value
>>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>>>> libglusterfsclient: invalid argument: value
>>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>>>> libglusterfsclient: invalid argument: value
>>>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>>>> libglusterfsclient: invalid argument: value
>>>> [2009-08-02 14:56:27] E [libglusterfsclient.c:4194:__glusterfs_stat]
>>>> libglusterfsclient: path lookup failed for (/hosts)
>>>> [2009-08-02 14:56:37] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>>>> libglusterfsclient: invalid argument: value
>>>> [2009-08-02 14:57:00] E [libglusterfsclient.c:4194:__glusterfs_stat]
>>>> libglusterfsclient: path lookup failed for (/issue)
>>>> [2009-08-02 14:57:07] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>>>> libglusterfsclient: invalid argument: value
>>>>
>>>> Then, I try to LD_PRELOAD apache (prefork). I change the target from
>>>> /gluster to /usr/home instead (the web application needs it). Then I
>>>> tried
>>>> to strace the httpd process and found that httpd crash at the points
>>>> where
>>>> httpd tried to read configuration file stored on Gluster volume (bad
>>>> file
>>>> descriptor). I will try to move this configuration file some other
>>>> places
>>>> and test again.
>>>>
>>>> 2009/7/31 Raghavendra G <raghavendra at gluster.com>
>>>>
>>>> Hi,
>>>>>
>>>>> On Thu, Jul 30, 2009 at 11:39 AM, Somsak
>>>>> Sriprayoonsakul<somsaks at gmail.com> wrote:
>>>>>
>>>>>> Thank you very much for you reply
>>>>>>
>>>>>> At the time we used 2.0.3, and yes we used stock Apache from CentOS. I
>>>>>>
>>>>> will
>>>>>
>>>>>> try 2.0.4 very soon to see if it's work.
>>>>>>
>>>>>> For Booster, it seems not working correctly for me. Booster complains
>>>>>> a
>>>>>>
>>>>> lots
>>>>>
>>>>>> of error with plain 'ls' command (but giving the correct output).
>>>>>> Also,
>>>>>>
>>>>> with
>>>>>
>>>>> Can you mail those errors?
>>>>>
>>>>> booster, Apache process refuse to start. I will try 2.0.4 to see if it
>>>>>> improves. If not, I will attach error log next time.
>>>>>>
>>>>> logs are very much appreciated.
>>>>>
>>>>>
>>>>>> 2009/7/30 Raghavendra G <raghavendra at gluster.com>
>>>>>>
>>>>>>> Hi Somsak,
>>>>>>>
>>>>>>> Sorry for the delayed reply. Below you've mentioned that you've
>>>>>>> problems
>>>>>>> with apache and booster. Going forward, Apache over booster will be
>>>>>>> the
>>>>>>> preferred approach. Can you tell us what version of glusterfs you are
>>>>>>>
>>>>>> using?
>>>>>
>>>>>> And as I can understand you are using apache 2.2, am I correct?
>>>>>>>
>>>>>>> regards,
>>>>>>> ----- Original Message -----
>>>>>>> From: "Liam Slusser" <lslusser at gmail.com>
>>>>>>> To: "Somsak Sriprayoonsakul" <somsaks at gmail.com>
>>>>>>> Cc: gluster-users at gluster.org
>>>>>>> Sent: Saturday, July 25, 2009 3:46:14 AM GMT +04:00 Abu Dhabi /
>>>>>>> Muscat
>>>>>>> Subject: Re: [Gluster-users] Gluster 2.0.3 + Apache on CentOS5
>>>>>>>
>>>>>> performance
>>>>>
>>>>>> issue
>>>>>>>
>>>>>>> I haven't tried an apples to apples comparison with
>>>>>>> Apache+mod_gluster
>>>>>>>
>>>>>> vs
>>>>>
>>>>>> Apache+fuse+gluster however i do run both setups. I load tested both
>>>>>>> setups
>>>>>>> so to verified it could handle 4x our normal daily load and left it
>>>>>>> at
>>>>>>> that.
>>>>>>> I didn't actually compare the two (although that might be cool to do
>>>>>>> someday).
>>>>>>> I really like the idea of Apache+mod_gluster as I don't have to deal
>>>>>>>
>>>>>> with
>>>>>
>>>>>> the whole fuse and mounting the filesystem. It always scares me
>>>>>>> having
>>>>>>>
>>>>>> a
>>>>>
>>>>>> public facing webserver with your whole backend fileshare mounted
>>>>>>>
>>>>>> locally.
>>>>>
>>>>>> Its very slick for serving content such as media files. We serve
>>>>>>> audio
>>>>>>> content to our CDN with a pair of Apache/mod_gluster servers -
>>>>>>> pushing
>>>>>>> 200-300mbit on average daily and everything works very well.
>>>>>>>
>>>>>>> We run an apache+fuse+gluster setup because we need to run some
>>>>>>> mod_perl
>>>>>>> before serving the actual content. However performance is still very
>>>>>>> good.
>>>>>>> We do around 50-100 requests (all jpeg images) per second off of a
>>>>>>> fuse
>>>>>>> mount and everything works great. We also have a java
>>>>>>>
>>>>>> tomcat+fuse+gluster
>>>>>
>>>>>> service which does image manipulation on the fly off of a gluster
>>>>>>> mount.
>>>>>>>
>>>>>>> We have two backend gluster servers using replication which serve all
>>>>>>>
>>>>>> this
>>>>>
>>>>>> content.
>>>>>>>
>>>>>>> If you would like more information on our setup id be happy to share
>>>>>>> offline. Just email me privately.
>>>>>>>
>>>>>>> thanks,
>>>>>>> liam
>>>>>>>
>>>>>>> On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul
>>>>>>> <somsaks at gmail.com>wrote:
>>>>>>>
>>>>>>> Oh thank you, thought noone will reply me :)
>>>>>>>>
>>>>>>>> Have you tried Apache + Fuse over GlusterFS? How is the performance?
>>>>>>>>
>>>>>>>> Also, anyone in this mailing-list have tried Apache with booster? I
>>>>>>>> tried
>>>>>>>> it but Apache refuse to start (just hang and freeze).
>>>>>>>>
>>>>>>>> 2009/7/23 Liam Slusser <lslusser at gmail.com>
>>>>>>>>
>>>>>>>>
>>>>>>>> We use mod_gluster and Apache
>>>>>>>>> 2.2 with good results. We also ran into the same issue as you that
>>>>>>>>>
>>>>>>>> we
>>>>>
>>>>>> ran out of memory past 150 threads even on a 8gig machine. We got
>>>>>>>>>
>>>>>>>> around
>>>>>
>>>>>> this by compiling Apache using mpm-worker
>>>>>>>>> (threads) instead of prefork - it uses 1/4 as much ram with the
>>>>>>>>> same
>>>>>>>>> number
>>>>>>>>> of connections (150-200) and everything has been running smoothly.
>>>>>>>>> I
>>>>>>>>> cannot
>>>>>>>>> see any performance difference except it using way less memory.
>>>>>>>>> liam
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul <
>>>>>>>>> somsaks at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> We have been evaluating the choice for the new platform for a
>>>>>>>>>>
>>>>>>>>> webboard
>>>>>
>>>>>> system.
>>>>>>>>>> The webboard is PHP scripts that generate/modify HTML page when
>>>>>>>>>> user
>>>>>>>>>> posting/add comment to the page, resulting topic is actually
>>>>>>>>>> stored
>>>>>>>>>>
>>>>>>>>> as
>>>>>
>>>>>> a
>>>>>>>>>> HTML file with all related file (file attach to the topic, etc..
>>>>>>>>>> )stored in
>>>>>>>>>> its own directory for each topic. In general, the web site mostly
>>>>>>>>>> serve a
>>>>>>>>>> lot of small static files using Apache while using PHP to do other
>>>>>>>>>> dynamic
>>>>>>>>>> contents. This system has been working very well in the past, with
>>>>>>>>>>
>>>>>>>>> the
>>>>>
>>>>>> increasing page view rate, it is very likely that we will need some
>>>>>>>>>> kind of
>>>>>>>>>> Cluster file system as backend very soon.
>>>>>>>>>>
>>>>>>>>>> We have set up a test system using Grinder as stress test tool.
>>>>>>>>>> The
>>>>>>>>>> test
>>>>>>>>>> system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock
>>>>>>>>>> Apache
>>>>>>>>>> (prefork, since the goal is to use this with PHP), linked together
>>>>>>>>>> with
>>>>>>>>>> Gigabit Ethernet. We try to compare the performance of either
>>>>>>>>>> using
>>>>>>>>>> single
>>>>>>>>>> NFS server in sync mode against using 4 Gluster nodes (distribute
>>>>>>>>>> of
>>>>>>>>>>
>>>>>>>>> 2
>>>>>
>>>>>> replicated nodes) through Fuse. However, the transaction per second
>>>>>>>>>> (TPS)
>>>>>>>>>> result is not good.
>>>>>>>>>>
>>>>>>>>>> NFS (single server, sync mode)
>>>>>>>>>> - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066,
>>>>>>>>>> mean
>>>>>>>>>> response time = 61.63 ms
>>>>>>>>>> - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33
>>>>>>>>>>
>>>>>>>>> ms
>>>>>
>>>>>> - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms
>>>>>>>>>> - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt =
>>>>>>>>>> 287.33ms
>>>>>>>>>>
>>>>>>>>>> 4 nodes Gluster (2 distribute of replicated 2 node)
>>>>>>>>>> - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt =
>>>>>>>>>>
>>>>>>>>> 207.33ms
>>>>>
>>>>>> - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt =
>>>>>>>>>> 672.67ms
>>>>>>>>>> - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt =
>>>>>>>>>> 931.33
>>>>>>>>>> (no 400-600 threads since we run out of client machine, sorry).
>>>>>>>>>>
>>>>>>>>>> gfsd is configured to use 32 thread of iothread as brick.
>>>>>>>>>> gfs-client
>>>>>>>>>> is
>>>>>>>>>> configured to use
>>>>>>>>>> io-cache->write-behind->readahead->distribute->replicate.
>>>>>>>>>> io-cache cache-size is 256MB. I used patched Fuse downloaded from
>>>>>>>>>> Gluster
>>>>>>>>>> web-site (build through DKMS).
>>>>>>>>>>
>>>>>>>>>> As the result yield, it seems that Gluster performance worse with
>>>>>>>>>> increasing no. of client. One observation is that the glusterfs
>>>>>>>>>> process on
>>>>>>>>>> client is taking about 100% of CPU during all the tests.
>>>>>>>>>> glusterfsd
>>>>>>>>>>
>>>>>>>>> is
>>>>>
>>>>>> utilizing only 70-80% of CPUs during the test time. Note that system
>>>>>>>>>> is Dual
>>>>>>>>>> core.
>>>>>>>>>>
>>>>>>>>>> I also tried using modglusterfs and not using fuse at all to serve
>>>>>>>>>>
>>>>>>>>> all
>>>>>
>>>>>> the static files and conduct another test with Grinder. The result
>>>>>>>>>>
>>>>>>>>> is
>>>>>
>>>>>> about
>>>>>>>>>> the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in
>>>>>>>>>>
>>>>>>>>> this
>>>>>
>>>>>> test
>>>>>>>>>> that each Apache prefork process used more about twice more memory
>>>>>>>>>>
>>>>>>>>> and
>>>>>
>>>>>> we
>>>>>>>>>> need to lower number of httpd processes by about half.
>>>>>>>>>>
>>>>>>>>>> I tried disable EnableMMAP and it didn't help much. Adjusting
>>>>>>>>>> readahead,
>>>>>>>>>> write behind according to GlusterOptimization page didn't help
>>>>>>>>>> much
>>>>>>>>>> either.
>>>>>>>>>>
>>>>>>>>>> My question is, there seems to be bottleneck in this setup, but
>>>>>>>>>> how
>>>>>>>>>> can I
>>>>>>>>>> track this? Note that, I didn't do any other optimization other
>>>>>>>>>> than
>>>>>>>>>> what
>>>>>>>>>> said above. Are there any best practice configuration for using
>>>>>>>>>>
>>>>>>>>> Apache
>>>>>
>>>>>> to
>>>>>>>>>> serve a bunch of small static files like this around?
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Somsak
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>
>>>>>>
>>>>>>
>>>>> regards,
>>>>> --
>>>>> Raghavendra G
>>>>>
>>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>
>>>
>>>
>>
>
>
> --
> Raghavendra G
>
>
More information about the Gluster-users
mailing list