[Gluster-users] Gluster 2.0.3 + Apache on CentOS5 performance issue

Somsak Sriprayoonsakul somsaks at gmail.com
Sun Aug 2 13:19:09 UTC 2009


After moving the embed configuration file out from glusterfs, now httpd boot
up ok and the web seems to work now.

However, comment posting is not working. It seems that the code that do the
html modification is not working in GlusterFS context. I found that the code
modify web page locally instead. So I think workaround for my case is to
mount glusterfs with fuse at the same path as booster.

I will give it another benchmark and see how it goes.

2009/8/2 Somsak Sriprayoonsakul <somsaks at gmail.com>

> Ok, I have a chance to run booster over 2.0.4
>
> Please find the attach file for my configuration
>
> I did configure boost and try simple ls over my Gluster file system. Here's
> the output of ls -al
>
> [root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
> ls -l /gluster/www/
> ls: /gluster/www/: Invalid argument
> ls: /gluster/www/members.pantip.com: Invalid argument
> ls: /gluster/www/cafe.pantip.com: Invalid argument
> ls: /gluster/www/admin.pantip.com: Invalid argument
> ls: /gluster/www/www.pantip.com: Invalid argument
> ls: /gluster/www/passwd3.sql: Invalid argument
> ls: /gluster/www/passwd.sql: Invalid argument
> ls: closing directory /gluster/www/: File descriptor in bad state
> total 129972
> drwxr-xr-x  3 root   root       8192 May 11 16:13 admin.pantip.com
> drwxr-xr-x  5 root   root       8192 May 18 11:11 cafe.pantip.com
> drwxr-xr-x  3 root   root       8192 May 11 18:48 members.pantip.com
> -rw-r--r--  1 root   root   66654820 May 18 10:50 passwd3.sql
> -rw-r--r--  1 root   root   66225769 May 18 10:33 passwd.sql
> drwxr-xr-x 11 apache apache     8192 May 18 09:47 www.pantip.com
> [root at compute-0-3 ~]#
>
>  [root at compute-0-3 ~]#
> LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so cp /etc/issue /gluster/
> [root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
> ls -l /gluster/issue
> ls: /gluster/issue: Invalid argument
> -rw-r--r-- 1 root root 47 Aug  2 14:57 /gluster/issue
> [root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
> cat /gluster/issue
> CentOS release 5.3 (Final)
> Kernel \r on an \m
>
> [root at compute-0-3 ~]#
>
>
> Despite all those errors, output seems to be fine
>
> And this is what inside my booster.log
>
> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
> libglusterfsclient: invalid argument: value
> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
> libglusterfsclient: invalid argument: value
> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
> libglusterfsclient: invalid argument: value
> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
> libglusterfsclient: invalid argument: value
> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
> libglusterfsclient: invalid argument: value
> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
> libglusterfsclient: invalid argument: value
> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
> libglusterfsclient: invalid argument: value
> [2009-08-02 14:56:27] E [libglusterfsclient.c:4194:__glusterfs_stat]
> libglusterfsclient: path lookup failed for (/hosts)
> [2009-08-02 14:56:37] E [libglusterfsclient.c:2244:glusterfs_getxattr]
> libglusterfsclient: invalid argument: value
> [2009-08-02 14:57:00] E [libglusterfsclient.c:4194:__glusterfs_stat]
> libglusterfsclient: path lookup failed for (/issue)
> [2009-08-02 14:57:07] E [libglusterfsclient.c:2244:glusterfs_getxattr]
> libglusterfsclient: invalid argument: value
>
> Then, I try to LD_PRELOAD apache (prefork). I change the target from
> /gluster to /usr/home instead (the web application needs it). Then I tried
> to strace the httpd process and found that httpd crash at the points where
> httpd tried to read configuration file stored on Gluster volume (bad file
> descriptor). I will try to move this configuration file some other places
> and test again.
>
> 2009/7/31 Raghavendra G <raghavendra at gluster.com>
>
> Hi,
>>
>> On Thu, Jul 30, 2009 at 11:39 AM, Somsak
>> Sriprayoonsakul<somsaks at gmail.com> wrote:
>> > Thank you very much for you reply
>> >
>> > At the time we used 2.0.3, and yes we used stock Apache from CentOS. I
>> will
>> > try 2.0.4 very soon to see if it's work.
>> >
>> > For Booster, it seems not working correctly for me. Booster complains a
>> lots
>> > of error with plain 'ls' command (but giving the correct output). Also,
>> with
>>
>> Can you mail those errors?
>>
>> > booster, Apache process refuse to start. I will try 2.0.4 to see if it
>> > improves. If not, I will attach error log next time.
>>
>> logs are very much appreciated.
>>
>> >
>> >
>> > 2009/7/30 Raghavendra G <raghavendra at gluster.com>
>> >>
>> >> Hi Somsak,
>> >>
>> >> Sorry for the delayed reply. Below you've mentioned that you've
>> problems
>> >> with apache and booster. Going forward, Apache over booster will be the
>> >> preferred approach. Can you tell us what version of glusterfs you are
>> using?
>> >> And as I can understand you are using apache 2.2, am I correct?
>> >>
>> >> regards,
>> >> ----- Original Message -----
>> >> From: "Liam Slusser" <lslusser at gmail.com>
>> >> To: "Somsak Sriprayoonsakul" <somsaks at gmail.com>
>> >> Cc: gluster-users at gluster.org
>> >> Sent: Saturday, July 25, 2009 3:46:14 AM GMT +04:00 Abu Dhabi / Muscat
>> >> Subject: Re: [Gluster-users] Gluster 2.0.3 + Apache on CentOS5
>> performance
>> >>  issue
>> >>
>> >> I haven't tried an apples to apples comparison with Apache+mod_gluster
>> vs
>> >> Apache+fuse+gluster however i do run both setups.  I load tested both
>> >> setups
>> >> so to verified it could handle 4x our normal daily load and left it at
>> >> that.
>> >>  I didn't actually compare the two (although that might be cool to do
>> >> someday).
>> >> I really like the idea of Apache+mod_gluster as I don't have to deal
>> with
>> >> the whole fuse and mounting the filesystem.  It always scares me having
>> a
>> >> public facing webserver with your whole backend fileshare mounted
>> locally.
>> >>  Its very slick for serving content such as media files.  We serve
>> audio
>> >> content to our CDN with a pair of Apache/mod_gluster servers - pushing
>> >> 200-300mbit on average daily and everything works very well.
>> >>
>> >> We run an apache+fuse+gluster setup because we need to run some
>> mod_perl
>> >> before serving the actual content.  However performance is still very
>> >> good.
>> >>  We do around 50-100 requests (all jpeg images) per second off of a
>> fuse
>> >> mount and everything works great.  We also have a java
>> tomcat+fuse+gluster
>> >> service which does image manipulation on the fly off of a gluster
>> mount.
>> >>
>> >> We have two backend gluster servers using replication which serve all
>> this
>> >> content.
>> >>
>> >> If you would like more information on our setup id be happy to share
>> >> offline.  Just email me privately.
>> >>
>> >> thanks,
>> >> liam
>> >>
>> >> On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul
>> >> <somsaks at gmail.com>wrote:
>> >>
>> >> > Oh thank you, thought noone will reply me :)
>> >> >
>> >> > Have you tried Apache + Fuse over GlusterFS? How is the performance?
>> >> >
>> >> > Also, anyone in this mailing-list have tried Apache with booster? I
>> >> > tried
>> >> > it but Apache refuse to start (just hang and freeze).
>> >> >
>> >> > 2009/7/23 Liam Slusser <lslusser at gmail.com>
>> >> >
>> >> >
>> >> >> We use mod_gluster and Apache
>> >> >> 2.2 with good results.  We also ran into the same issue as you that
>> we
>> >> >> ran out of memory past 150 threads even on a 8gig machine.  We got
>> around
>> >> >> this by compiling Apache using mpm-worker
>> >> >> (threads) instead of prefork - it uses 1/4 as much ram with the same
>> >> >> number
>> >> >> of connections (150-200) and everything has been running smoothly.
>>  I
>> >> >> cannot
>> >> >> see any performance difference except it using way less memory.
>> >> >> liam
>> >> >>
>> >> >>
>> >> >> On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul <
>> >> >> somsaks at gmail.com> wrote:
>> >> >>
>> >> >>> Hello,
>> >> >>>
>> >> >>> We have been evaluating the choice for the new platform for a
>> webboard
>> >> >>> system.
>> >> >>> The webboard is PHP scripts that generate/modify HTML page when
>> user
>> >> >>> posting/add comment to the page, resulting topic is actually stored
>> as
>> >> >>> a
>> >> >>> HTML file with all related file (file attach to the topic, etc..
>> >> >>> )stored in
>> >> >>> its own directory for each topic. In general, the web site mostly
>> >> >>> serve a
>> >> >>> lot of small static files using Apache while using PHP to do other
>> >> >>> dynamic
>> >> >>> contents. This system has been working very well in the past, with
>> the
>> >> >>> increasing page view rate, it is very likely that we will need some
>> >> >>> kind of
>> >> >>> Cluster file system as backend very soon.
>> >> >>>
>> >> >>> We have set up a test system using Grinder as stress test tool. The
>> >> >>> test
>> >> >>> system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock
>> >> >>> Apache
>> >> >>> (prefork, since the goal is to use this with PHP), linked together
>> >> >>> with
>> >> >>> Gigabit Ethernet. We try to compare the performance of either using
>> >> >>> single
>> >> >>> NFS server in sync mode against using 4 Gluster nodes (distribute
>> of 2
>> >> >>> replicated nodes) through Fuse. However, the transaction per second
>> >> >>> (TPS)
>> >> >>> result is not good.
>> >> >>>
>> >> >>> NFS (single server, sync mode)
>> >> >>>  - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, mean
>> >> >>> response time = 61.63 ms
>> >> >>>  - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33
>> ms
>> >> >>>  - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms
>> >> >>>  - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt =
>> >> >>> 287.33ms
>> >> >>>
>> >> >>> 4 nodes Gluster (2 distribute of replicated 2 node)
>> >> >>> - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt =
>> 207.33ms
>> >> >>> - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt =
>> >> >>> 672.67ms
>> >> >>> - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = 931.33
>> >> >>> (no 400-600 threads since we run out of client machine, sorry).
>> >> >>>
>> >> >>> gfsd is configured to use 32 thread of iothread as brick.
>> gfs-client
>> >> >>> is
>> >> >>> configured to use
>> >> >>> io-cache->write-behind->readahead->distribute->replicate.
>> >> >>> io-cache cache-size is 256MB. I used patched Fuse downloaded from
>> >> >>> Gluster
>> >> >>> web-site (build through DKMS).
>> >> >>>
>> >> >>> As the result yield, it seems that Gluster performance worse with
>> >> >>> increasing no. of client. One observation is that the glusterfs
>> >> >>> process on
>> >> >>> client is taking about 100% of CPU during all the tests. glusterfsd
>> is
>> >> >>> utilizing only 70-80% of CPUs during the test time. Note that
>> system
>> >> >>> is Dual
>> >> >>> core.
>> >> >>>
>> >> >>> I also tried using modglusterfs and not using fuse at all to serve
>> all
>> >> >>> the static files and conduct another test with Grinder. The result
>> is
>> >> >>> about
>> >> >>> the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in
>> this
>> >> >>> test
>> >> >>> that each Apache prefork process used more about twice more memory
>> and
>> >> >>> we
>> >> >>> need to lower number of httpd processes by about half.
>> >> >>>
>> >> >>> I tried disable EnableMMAP and it didn't help much. Adjusting
>> >> >>> readahead,
>> >> >>> write behind according to GlusterOptimization page didn't help much
>> >> >>> either.
>> >> >>>
>> >> >>> My question is, there seems to be bottleneck in this setup, but how
>> >> >>> can I
>> >> >>> track this? Note that, I didn't do any other optimization other
>> than
>> >> >>> what
>> >> >>> said above. Are there any best practice configuration for using
>> Apache
>> >> >>> to
>> >> >>> serve a bunch of small static files like this around?
>> >> >>>
>> >> >>> Regards,
>> >> >>>
>> >> >>> Somsak
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> _______________________________________________
>> >> >>> Gluster-users mailing list
>> >> >>> Gluster-users at gluster.org
>> >> >>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>> >> >>>
>> >> >>>
>> >> >>
>> >> >
>> >>
>> >> _______________________________________________
>> >> Gluster-users mailing list
>> >> Gluster-users at gluster.org
>> >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>> >
>> >
>>
>>
>> regards,
>> --
>> Raghavendra G
>>
>
>


More information about the Gluster-users mailing list