[Gluster-users] Gluster 2.0.3 + Apache on CentOS5 performance issue

Somsak Sriprayoonsakul somsaks at gmail.com
Thu Aug 6 05:24:30 UTC 2009


The behavior is still the same

[root at compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local"
http://c0-3/cafe/siam/topic/F7800428/F7800428.html
--12:20:33--  http://c0-3/cafe/siam/topic/F7800428/F7800428.html
Resolving c0-3... 10.1.255.251
Connecting to c0-3|10.1.255.251|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 264373 (258K) [text/html]
Saving to: `/dev/null'

100%[====================================================================================================================>]
264,373     --.-K/s   in 0.003s

12:20:33 (81.8 MB/s) - `/dev/null' saved [264373/264373]

[root at compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local"
http://c0-3/cafe/siam/topic/F7800428/F7800428.html
--12:20:35--  http://c0-3/cafe/siam/topic/F7800428/F7800428.html
Resolving c0-3... 10.1.255.251
Connecting to c0-3|10.1.255.251|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/html]
Saving to: `/dev/null'

    [
<=>
] 0           --.-K/s   in 0s

12:20:35 (0.00 B/s) - `/dev/null' saved [0/0]

[root at compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local"
http://c0-3/cafe/siam/topic/F7800428/F7800428.html
--12:20:38--  http://c0-3/cafe/siam/topic/F7800428/F7800428.html
Resolving c0-3... 10.1.255.251
Connecting to c0-3|10.1.255.251|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/html]
Saving to: `/dev/null'

    [
<=>
] 0           --.-K/s   in 0s

12:20:38 (0.00 B/s) - `/dev/null' saved [0/0]

[root at compute-0-0 ~]# wget -O /dev/null --header "Host: www.myhost.local"
http://c0-3/cafe/siam/topic/F7800428/F7800428.html
--12:20:39--  http://c0-3/cafe/siam/topic/F7800428/F7800428.html
Resolving c0-3... 10.1.255.251
Connecting to c0-3|10.1.255.251|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 264373 (258K) [text/html]
Saving to: `/dev/null'

100%[====================================================================================================================>]
264,373     --.-K/s   in 0.003s

12:20:39 (82.3 MB/s) - `/dev/null' saved [264373/264373]

[root at compute-0-0 ~]#


4 consequtive wget yield different content length. Note that, this only
happened with worker + booster, prefork has no problem.

And the web server seems to use correct version of glusterfs

[root at compute-0-3 ~]# glusterfs -V
glusterfs 2.0.6rc2 built on Aug  6 2009 12:02:03
Repository revision: v2.0.5-13-g5e3ca25
Copyright (c) 2006-2009 Z RESEARCH Inc. <http://www.zresearch.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General
Public License.
[root at compute-0-3 ~]#

And all glusterfsd too

[root at fe1 ~]# pexec -p storage glusterfs -V \| grep 2\.0\.6
compute-0-6:    glusterfs 2.0.6rc2 built on Aug  6 2009 12:02:03
compute-0-8:    glusterfs 2.0.6rc2 built on Aug  6 2009 12:02:03
compute-0-9:    glusterfs 2.0.6rc2 built on Aug  6 2009 12:02:03
compute-0-7:    glusterfs 2.0.6rc2 built on Aug  6 2009 12:02:03
[root at fe1 ~]#

Don't sure if there is anyway to query version from running glusterfsd, but
I did /etc/init.d/glusterfsd restart already.

2009/8/3 Shehjar Tikoo <shehjart at gluster.com>

> Somsak Sriprayoonsakul wrote:
>
>> Ok, I have a chance to run booster over 2.0.4
>>
>
>
> Have you tried configuring booster with the help doc available at:
> http://www.gluster.org/docs/index.php/BoosterConfiguration
>
> -Shehjar
>
>
>> Please find the attach file for my configuration
>>
>> I did configure boost and try simple ls over my Gluster file system.
>> Here's
>> the output of ls -al
>>
>> [root at compute-0-3 ~]#
>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
>> ls -l /gluster/www/
>> ls: /gluster/www/: Invalid argument
>> ls: /gluster/www/members.pantip.com: Invalid argument
>> ls: /gluster/www/cafe.pantip.com: Invalid argument
>> ls: /gluster/www/admin.pantip.com: Invalid argument
>> ls: /gluster/www/www.pantip.com: Invalid argument
>> ls: /gluster/www/passwd3.sql: Invalid argument
>> ls: /gluster/www/passwd.sql: Invalid argument
>> ls: closing directory /gluster/www/: File descriptor in bad state
>> total 129972
>> drwxr-xr-x  3 root   root       8192 May 11 16:13 admin.pantip.com
>> drwxr-xr-x  5 root   root       8192 May 18 11:11 cafe.pantip.com
>> drwxr-xr-x  3 root   root       8192 May 11 18:48 members.pantip.com
>> -rw-r--r--  1 root   root   66654820 May 18 10:50 passwd3.sql
>> -rw-r--r--  1 root   root   66225769 May 18 10:33 passwd.sql
>> drwxr-xr-x 11 apache apache     8192 May 18 09:47 www.pantip.com
>> [root at compute-0-3 ~]#
>>
>> [root at compute-0-3 ~]#
>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
>> cp /etc/issue /gluster/
>> [root at compute-0-3 ~]#
>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
>> ls -l /gluster/issue
>> ls: /gluster/issue: Invalid argument
>> -rw-r--r-- 1 root root 47 Aug  2 14:57 /gluster/issue
>> [root at compute-0-3 ~]#
>> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
>> cat /gluster/issue
>> CentOS release 5.3 (Final)
>> Kernel \r on an \m
>>
>> [root at compute-0-3 ~]#
>>
>>
>> Despite all those errors, output seems to be fine
>>
>> And this is what inside my booster.log
>>
>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>> libglusterfsclient: invalid argument: value
>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>> libglusterfsclient: invalid argument: value
>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>> libglusterfsclient: invalid argument: value
>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>> libglusterfsclient: invalid argument: value
>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>> libglusterfsclient: invalid argument: value
>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>> libglusterfsclient: invalid argument: value
>> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>> libglusterfsclient: invalid argument: value
>> [2009-08-02 14:56:27] E [libglusterfsclient.c:4194:__glusterfs_stat]
>> libglusterfsclient: path lookup failed for (/hosts)
>> [2009-08-02 14:56:37] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>> libglusterfsclient: invalid argument: value
>> [2009-08-02 14:57:00] E [libglusterfsclient.c:4194:__glusterfs_stat]
>> libglusterfsclient: path lookup failed for (/issue)
>> [2009-08-02 14:57:07] E [libglusterfsclient.c:2244:glusterfs_getxattr]
>> libglusterfsclient: invalid argument: value
>>
>> Then, I try to LD_PRELOAD apache (prefork). I change the target from
>> /gluster to /usr/home instead (the web application needs it). Then I tried
>> to strace the httpd process and found that httpd crash at the points where
>> httpd tried to read configuration file stored on Gluster volume (bad file
>> descriptor). I will try to move this configuration file some other places
>> and test again.
>>
>> 2009/7/31 Raghavendra G <raghavendra at gluster.com>
>>
>>  Hi,
>>>
>>> On Thu, Jul 30, 2009 at 11:39 AM, Somsak
>>> Sriprayoonsakul<somsaks at gmail.com> wrote:
>>>
>>>> Thank you very much for you reply
>>>>
>>>> At the time we used 2.0.3, and yes we used stock Apache from CentOS. I
>>>>
>>> will
>>>
>>>> try 2.0.4 very soon to see if it's work.
>>>>
>>>> For Booster, it seems not working correctly for me. Booster complains a
>>>>
>>> lots
>>>
>>>> of error with plain 'ls' command (but giving the correct output). Also,
>>>>
>>> with
>>>
>>> Can you mail those errors?
>>>
>>>  booster, Apache process refuse to start. I will try 2.0.4 to see if it
>>>> improves. If not, I will attach error log next time.
>>>>
>>> logs are very much appreciated.
>>>
>>>
>>>> 2009/7/30 Raghavendra G <raghavendra at gluster.com>
>>>>
>>>>> Hi Somsak,
>>>>>
>>>>> Sorry for the delayed reply. Below you've mentioned that you've
>>>>> problems
>>>>> with apache and booster. Going forward, Apache over booster will be the
>>>>> preferred approach. Can you tell us what version of glusterfs you are
>>>>>
>>>> using?
>>>
>>>> And as I can understand you are using apache 2.2, am I correct?
>>>>>
>>>>> regards,
>>>>> ----- Original Message -----
>>>>> From: "Liam Slusser" <lslusser at gmail.com>
>>>>> To: "Somsak Sriprayoonsakul" <somsaks at gmail.com>
>>>>> Cc: gluster-users at gluster.org
>>>>> Sent: Saturday, July 25, 2009 3:46:14 AM GMT +04:00 Abu Dhabi / Muscat
>>>>> Subject: Re: [Gluster-users] Gluster 2.0.3 + Apache on CentOS5
>>>>>
>>>> performance
>>>
>>>>  issue
>>>>>
>>>>> I haven't tried an apples to apples comparison with Apache+mod_gluster
>>>>>
>>>> vs
>>>
>>>> Apache+fuse+gluster however i do run both setups.  I load tested both
>>>>> setups
>>>>> so to verified it could handle 4x our normal daily load and left it at
>>>>> that.
>>>>>  I didn't actually compare the two (although that might be cool to do
>>>>> someday).
>>>>> I really like the idea of Apache+mod_gluster as I don't have to deal
>>>>>
>>>> with
>>>
>>>> the whole fuse and mounting the filesystem.  It always scares me having
>>>>>
>>>> a
>>>
>>>> public facing webserver with your whole backend fileshare mounted
>>>>>
>>>> locally.
>>>
>>>>  Its very slick for serving content such as media files.  We serve audio
>>>>> content to our CDN with a pair of Apache/mod_gluster servers - pushing
>>>>> 200-300mbit on average daily and everything works very well.
>>>>>
>>>>> We run an apache+fuse+gluster setup because we need to run some
>>>>> mod_perl
>>>>> before serving the actual content.  However performance is still very
>>>>> good.
>>>>>  We do around 50-100 requests (all jpeg images) per second off of a
>>>>> fuse
>>>>> mount and everything works great.  We also have a java
>>>>>
>>>> tomcat+fuse+gluster
>>>
>>>> service which does image manipulation on the fly off of a gluster mount.
>>>>>
>>>>> We have two backend gluster servers using replication which serve all
>>>>>
>>>> this
>>>
>>>> content.
>>>>>
>>>>> If you would like more information on our setup id be happy to share
>>>>> offline.  Just email me privately.
>>>>>
>>>>> thanks,
>>>>> liam
>>>>>
>>>>> On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul
>>>>> <somsaks at gmail.com>wrote:
>>>>>
>>>>>  Oh thank you, thought noone will reply me :)
>>>>>>
>>>>>> Have you tried Apache + Fuse over GlusterFS? How is the performance?
>>>>>>
>>>>>> Also, anyone in this mailing-list have tried Apache with booster? I
>>>>>> tried
>>>>>> it but Apache refuse to start (just hang and freeze).
>>>>>>
>>>>>> 2009/7/23 Liam Slusser <lslusser at gmail.com>
>>>>>>
>>>>>>
>>>>>>  We use mod_gluster and Apache
>>>>>>> 2.2 with good results.  We also ran into the same issue as you that
>>>>>>>
>>>>>> we
>>>
>>>>  ran out of memory past 150 threads even on a 8gig machine.  We got
>>>>>>>
>>>>>> around
>>>
>>>>  this by compiling Apache using mpm-worker
>>>>>>> (threads) instead of prefork - it uses 1/4 as much ram with the same
>>>>>>> number
>>>>>>> of connections (150-200) and everything has been running smoothly.  I
>>>>>>> cannot
>>>>>>> see any performance difference except it using way less memory.
>>>>>>> liam
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul <
>>>>>>> somsaks at gmail.com> wrote:
>>>>>>>
>>>>>>>  Hello,
>>>>>>>>
>>>>>>>> We have been evaluating the choice for the new platform for a
>>>>>>>>
>>>>>>> webboard
>>>
>>>>  system.
>>>>>>>> The webboard is PHP scripts that generate/modify HTML page when user
>>>>>>>> posting/add comment to the page, resulting topic is actually stored
>>>>>>>>
>>>>>>> as
>>>
>>>>  a
>>>>>>>> HTML file with all related file (file attach to the topic, etc..
>>>>>>>> )stored in
>>>>>>>> its own directory for each topic. In general, the web site mostly
>>>>>>>> serve a
>>>>>>>> lot of small static files using Apache while using PHP to do other
>>>>>>>> dynamic
>>>>>>>> contents. This system has been working very well in the past, with
>>>>>>>>
>>>>>>> the
>>>
>>>>  increasing page view rate, it is very likely that we will need some
>>>>>>>> kind of
>>>>>>>> Cluster file system as backend very soon.
>>>>>>>>
>>>>>>>> We have set up a test system using Grinder as stress test tool. The
>>>>>>>> test
>>>>>>>> system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock
>>>>>>>> Apache
>>>>>>>> (prefork, since the goal is to use this with PHP), linked together
>>>>>>>> with
>>>>>>>> Gigabit Ethernet. We try to compare the performance of either using
>>>>>>>> single
>>>>>>>> NFS server in sync mode against using 4 Gluster nodes (distribute of
>>>>>>>>
>>>>>>> 2
>>>
>>>>  replicated nodes) through Fuse. However, the transaction per second
>>>>>>>> (TPS)
>>>>>>>> result is not good.
>>>>>>>>
>>>>>>>> NFS (single server, sync mode)
>>>>>>>>  - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, mean
>>>>>>>> response time = 61.63 ms
>>>>>>>>  - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33
>>>>>>>>
>>>>>>> ms
>>>
>>>>   - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms
>>>>>>>>  - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt =
>>>>>>>> 287.33ms
>>>>>>>>
>>>>>>>> 4 nodes Gluster (2 distribute of replicated 2 node)
>>>>>>>> - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt =
>>>>>>>>
>>>>>>> 207.33ms
>>>
>>>>  - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt =
>>>>>>>> 672.67ms
>>>>>>>> - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = 931.33
>>>>>>>> (no 400-600 threads since we run out of client machine, sorry).
>>>>>>>>
>>>>>>>> gfsd is configured to use 32 thread of iothread as brick. gfs-client
>>>>>>>> is
>>>>>>>> configured to use
>>>>>>>> io-cache->write-behind->readahead->distribute->replicate.
>>>>>>>> io-cache cache-size is 256MB. I used patched Fuse downloaded from
>>>>>>>> Gluster
>>>>>>>> web-site (build through DKMS).
>>>>>>>>
>>>>>>>> As the result yield, it seems that Gluster performance worse with
>>>>>>>> increasing no. of client. One observation is that the glusterfs
>>>>>>>> process on
>>>>>>>> client is taking about 100% of CPU during all the tests. glusterfsd
>>>>>>>>
>>>>>>> is
>>>
>>>>  utilizing only 70-80% of CPUs during the test time. Note that system
>>>>>>>> is Dual
>>>>>>>> core.
>>>>>>>>
>>>>>>>> I also tried using modglusterfs and not using fuse at all to serve
>>>>>>>>
>>>>>>> all
>>>
>>>>  the static files and conduct another test with Grinder. The result
>>>>>>>>
>>>>>>> is
>>>
>>>>  about
>>>>>>>> the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in
>>>>>>>>
>>>>>>> this
>>>
>>>>  test
>>>>>>>> that each Apache prefork process used more about twice more memory
>>>>>>>>
>>>>>>> and
>>>
>>>>  we
>>>>>>>> need to lower number of httpd processes by about half.
>>>>>>>>
>>>>>>>> I tried disable EnableMMAP and it didn't help much. Adjusting
>>>>>>>> readahead,
>>>>>>>> write behind according to GlusterOptimization page didn't help much
>>>>>>>> either.
>>>>>>>>
>>>>>>>> My question is, there seems to be bottleneck in this setup, but how
>>>>>>>> can I
>>>>>>>> track this? Note that, I didn't do any other optimization other than
>>>>>>>> what
>>>>>>>> said above. Are there any best practice configuration for using
>>>>>>>>
>>>>>>> Apache
>>>
>>>>  to
>>>>>>>> serve a bunch of small static files like this around?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Somsak
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Gluster-users mailing list
>>>>>>>> Gluster-users at gluster.org
>>>>>>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>>
>>>>>>>>
>>>>>>>>  _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>
>>>>
>>>>
>>> regards,
>>> --
>>> Raghavendra G
>>>
>>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>
>


More information about the Gluster-users mailing list