[Gluster-users] Gluster 2.0.3 + Apache on CentOS5 performance issue

Mon Aug 3 05:49:57 UTC 2009

Somsak Sriprayoonsakul wrote:
> Ok, I have a chance to run booster over 2.0.4


Have you tried configuring booster with the help doc available at:
http://www.gluster.org/docs/index.php/BoosterConfiguration

-Shehjar

> 
> Please find the attach file for my configuration
> 
> I did configure boost and try simple ls over my Gluster file system. Here's
> the output of ls -al
> 
> [root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
> ls -l /gluster/www/
> ls: /gluster/www/: Invalid argument
> ls: /gluster/www/members.pantip.com: Invalid argument
> ls: /gluster/www/cafe.pantip.com: Invalid argument
> ls: /gluster/www/admin.pantip.com: Invalid argument
> ls: /gluster/www/www.pantip.com: Invalid argument
> ls: /gluster/www/passwd3.sql: Invalid argument
> ls: /gluster/www/passwd.sql: Invalid argument
> ls: closing directory /gluster/www/: File descriptor in bad state
> total 129972
> drwxr-xr-x  3 root   root       8192 May 11 16:13 admin.pantip.com
> drwxr-xr-x  5 root   root       8192 May 18 11:11 cafe.pantip.com
> drwxr-xr-x  3 root   root       8192 May 11 18:48 members.pantip.com
> -rw-r--r--  1 root   root   66654820 May 18 10:50 passwd3.sql
> -rw-r--r--  1 root   root   66225769 May 18 10:33 passwd.sql
> drwxr-xr-x 11 apache apache     8192 May 18 09:47 www.pantip.com
> [root at compute-0-3 ~]#
> 
> [root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
> cp /etc/issue /gluster/
> [root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
> ls -l /gluster/issue
> ls: /gluster/issue: Invalid argument
> -rw-r--r-- 1 root root 47 Aug  2 14:57 /gluster/issue
> [root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
> cat /gluster/issue
> CentOS release 5.3 (Final)
> Kernel \r on an \m
> 
> [root at compute-0-3 ~]#
> 
> 
> Despite all those errors, output seems to be fine
> 
> And this is what inside my booster.log
> 
> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
> libglusterfsclient: invalid argument: value
> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
> libglusterfsclient: invalid argument: value
> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
> libglusterfsclient: invalid argument: value
> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
> libglusterfsclient: invalid argument: value
> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
> libglusterfsclient: invalid argument: value
> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
> libglusterfsclient: invalid argument: value
> [2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
> libglusterfsclient: invalid argument: value
> [2009-08-02 14:56:27] E [libglusterfsclient.c:4194:__glusterfs_stat]
> libglusterfsclient: path lookup failed for (/hosts)
> [2009-08-02 14:56:37] E [libglusterfsclient.c:2244:glusterfs_getxattr]
> libglusterfsclient: invalid argument: value
> [2009-08-02 14:57:00] E [libglusterfsclient.c:4194:__glusterfs_stat]
> libglusterfsclient: path lookup failed for (/issue)
> [2009-08-02 14:57:07] E [libglusterfsclient.c:2244:glusterfs_getxattr]
> libglusterfsclient: invalid argument: value
> 
> Then, I try to LD_PRELOAD apache (prefork). I change the target from
> /gluster to /usr/home instead (the web application needs it). Then I tried
> to strace the httpd process and found that httpd crash at the points where
> httpd tried to read configuration file stored on Gluster volume (bad file
> descriptor). I will try to move this configuration file some other places
> and test again.
> 
> 2009/7/31 Raghavendra G <raghavendra at gluster.com>
> 
>> Hi,
>>
>> On Thu, Jul 30, 2009 at 11:39 AM, Somsak
>> Sriprayoonsakul<somsaks at gmail.com> wrote:
>>> Thank you very much for you reply
>>>
>>> At the time we used 2.0.3, and yes we used stock Apache from CentOS. I
>> will
>>> try 2.0.4 very soon to see if it's work.
>>>
>>> For Booster, it seems not working correctly for me. Booster complains a
>> lots
>>> of error with plain 'ls' command (but giving the correct output). Also,
>> with
>>
>> Can you mail those errors?
>>
>>> booster, Apache process refuse to start. I will try 2.0.4 to see if it
>>> improves. If not, I will attach error log next time.
>> logs are very much appreciated.
>>
>>>
>>> 2009/7/30 Raghavendra G <raghavendra at gluster.com>
>>>> Hi Somsak,
>>>>
>>>> Sorry for the delayed reply. Below you've mentioned that you've problems
>>>> with apache and booster. Going forward, Apache over booster will be the
>>>> preferred approach. Can you tell us what version of glusterfs you are
>> using?
>>>> And as I can understand you are using apache 2.2, am I correct?
>>>>
>>>> regards,
>>>> ----- Original Message -----
>>>> From: "Liam Slusser" <lslusser at gmail.com>
>>>> To: "Somsak Sriprayoonsakul" <somsaks at gmail.com>
>>>> Cc: gluster-users at gluster.org
>>>> Sent: Saturday, July 25, 2009 3:46:14 AM GMT +04:00 Abu Dhabi / Muscat
>>>> Subject: Re: [Gluster-users] Gluster 2.0.3 + Apache on CentOS5
>> performance
>>>>  issue
>>>>
>>>> I haven't tried an apples to apples comparison with Apache+mod_gluster
>> vs
>>>> Apache+fuse+gluster however i do run both setups.  I load tested both
>>>> setups
>>>> so to verified it could handle 4x our normal daily load and left it at
>>>> that.
>>>>  I didn't actually compare the two (although that might be cool to do
>>>> someday).
>>>> I really like the idea of Apache+mod_gluster as I don't have to deal
>> with
>>>> the whole fuse and mounting the filesystem.  It always scares me having
>> a
>>>> public facing webserver with your whole backend fileshare mounted
>> locally.
>>>>  Its very slick for serving content such as media files.  We serve audio
>>>> content to our CDN with a pair of Apache/mod_gluster servers - pushing
>>>> 200-300mbit on average daily and everything works very well.
>>>>
>>>> We run an apache+fuse+gluster setup because we need to run some mod_perl
>>>> before serving the actual content.  However performance is still very
>>>> good.
>>>>  We do around 50-100 requests (all jpeg images) per second off of a fuse
>>>> mount and everything works great.  We also have a java
>> tomcat+fuse+gluster
>>>> service which does image manipulation on the fly off of a gluster mount.
>>>>
>>>> We have two backend gluster servers using replication which serve all
>> this
>>>> content.
>>>>
>>>> If you would like more information on our setup id be happy to share
>>>> offline.  Just email me privately.
>>>>
>>>> thanks,
>>>> liam
>>>>
>>>> On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul
>>>> <somsaks at gmail.com>wrote:
>>>>
>>>>> Oh thank you, thought noone will reply me :)
>>>>>
>>>>> Have you tried Apache + Fuse over GlusterFS? How is the performance?
>>>>>
>>>>> Also, anyone in this mailing-list have tried Apache with booster? I
>>>>> tried
>>>>> it but Apache refuse to start (just hang and freeze).
>>>>>
>>>>> 2009/7/23 Liam Slusser <lslusser at gmail.com>
>>>>>
>>>>>
>>>>>> We use mod_gluster and Apache
>>>>>> 2.2 with good results.  We also ran into the same issue as you that
>> we
>>>>>> ran out of memory past 150 threads even on a 8gig machine.  We got
>> around
>>>>>> this by compiling Apache using mpm-worker
>>>>>> (threads) instead of prefork - it uses 1/4 as much ram with the same
>>>>>> number
>>>>>> of connections (150-200) and everything has been running smoothly.  I
>>>>>> cannot
>>>>>> see any performance difference except it using way less memory.
>>>>>> liam
>>>>>>
>>>>>>
>>>>>> On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul <
>>>>>> somsaks at gmail.com> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> We have been evaluating the choice for the new platform for a
>> webboard
>>>>>>> system.
>>>>>>> The webboard is PHP scripts that generate/modify HTML page when user
>>>>>>> posting/add comment to the page, resulting topic is actually stored
>> as
>>>>>>> a
>>>>>>> HTML file with all related file (file attach to the topic, etc..
>>>>>>> )stored in
>>>>>>> its own directory for each topic. In general, the web site mostly
>>>>>>> serve a
>>>>>>> lot of small static files using Apache while using PHP to do other
>>>>>>> dynamic
>>>>>>> contents. This system has been working very well in the past, with
>> the
>>>>>>> increasing page view rate, it is very likely that we will need some
>>>>>>> kind of
>>>>>>> Cluster file system as backend very soon.
>>>>>>>
>>>>>>> We have set up a test system using Grinder as stress test tool. The
>>>>>>> test
>>>>>>> system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock
>>>>>>> Apache
>>>>>>> (prefork, since the goal is to use this with PHP), linked together
>>>>>>> with
>>>>>>> Gigabit Ethernet. We try to compare the performance of either using
>>>>>>> single
>>>>>>> NFS server in sync mode against using 4 Gluster nodes (distribute of
>> 2
>>>>>>> replicated nodes) through Fuse. However, the transaction per second
>>>>>>> (TPS)
>>>>>>> result is not good.
>>>>>>>
>>>>>>> NFS (single server, sync mode)
>>>>>>>  - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, mean
>>>>>>> response time = 61.63 ms
>>>>>>>  - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33
>> ms
>>>>>>>  - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms
>>>>>>>  - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt =
>>>>>>> 287.33ms
>>>>>>>
>>>>>>> 4 nodes Gluster (2 distribute of replicated 2 node)
>>>>>>> - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt =
>> 207.33ms
>>>>>>> - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt =
>>>>>>> 672.67ms
>>>>>>> - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = 931.33
>>>>>>> (no 400-600 threads since we run out of client machine, sorry).
>>>>>>>
>>>>>>> gfsd is configured to use 32 thread of iothread as brick. gfs-client
>>>>>>> is
>>>>>>> configured to use
>>>>>>> io-cache->write-behind->readahead->distribute->replicate.
>>>>>>> io-cache cache-size is 256MB. I used patched Fuse downloaded from
>>>>>>> Gluster
>>>>>>> web-site (build through DKMS).
>>>>>>>
>>>>>>> As the result yield, it seems that Gluster performance worse with
>>>>>>> increasing no. of client. One observation is that the glusterfs
>>>>>>> process on
>>>>>>> client is taking about 100% of CPU during all the tests. glusterfsd
>> is
>>>>>>> utilizing only 70-80% of CPUs during the test time. Note that system
>>>>>>> is Dual
>>>>>>> core.
>>>>>>>
>>>>>>> I also tried using modglusterfs and not using fuse at all to serve
>> all
>>>>>>> the static files and conduct another test with Grinder. The result
>> is
>>>>>>> about
>>>>>>> the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in
>> this
>>>>>>> test
>>>>>>> that each Apache prefork process used more about twice more memory
>> and
>>>>>>> we
>>>>>>> need to lower number of httpd processes by about half.
>>>>>>>
>>>>>>> I tried disable EnableMMAP and it didn't help much. Adjusting
>>>>>>> readahead,
>>>>>>> write behind according to GlusterOptimization page didn't help much
>>>>>>> either.
>>>>>>>
>>>>>>> My question is, there seems to be bottleneck in this setup, but how
>>>>>>> can I
>>>>>>> track this? Note that, I didn't do any other optimization other than
>>>>>>> what
>>>>>>> said above. Are there any best practice configuration for using
>> Apache
>>>>>>> to
>>>>>>> serve a bunch of small static files like this around?
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Somsak
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>
>>>>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>
>>
>> regards,
>> --
>> Raghavendra G
>>
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users