[Gluster-users] Gluster 2.0.3 + Apache on CentOS5 performance issue
Somsak Sriprayoonsakul
somsaks at gmail.com
Sun Aug 2 08:16:24 UTC 2009
Ok, I have a chance to run booster over 2.0.4
Please find the attach file for my configuration
I did configure boost and try simple ls over my Gluster file system. Here's
the output of ls -al
[root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
ls -l /gluster/www/
ls: /gluster/www/: Invalid argument
ls: /gluster/www/members.pantip.com: Invalid argument
ls: /gluster/www/cafe.pantip.com: Invalid argument
ls: /gluster/www/admin.pantip.com: Invalid argument
ls: /gluster/www/www.pantip.com: Invalid argument
ls: /gluster/www/passwd3.sql: Invalid argument
ls: /gluster/www/passwd.sql: Invalid argument
ls: closing directory /gluster/www/: File descriptor in bad state
total 129972
drwxr-xr-x 3 root root 8192 May 11 16:13 admin.pantip.com
drwxr-xr-x 5 root root 8192 May 18 11:11 cafe.pantip.com
drwxr-xr-x 3 root root 8192 May 11 18:48 members.pantip.com
-rw-r--r-- 1 root root 66654820 May 18 10:50 passwd3.sql
-rw-r--r-- 1 root root 66225769 May 18 10:33 passwd.sql
drwxr-xr-x 11 apache apache 8192 May 18 09:47 www.pantip.com
[root at compute-0-3 ~]#
[root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
cp /etc/issue /gluster/
[root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
ls -l /gluster/issue
ls: /gluster/issue: Invalid argument
-rw-r--r-- 1 root root 47 Aug 2 14:57 /gluster/issue
[root at compute-0-3 ~]# LD_PRELOAD=/usr/lib64/glusterfs/glusterfs-booster.so
cat /gluster/issue
CentOS release 5.3 (Final)
Kernel \r on an \m
[root at compute-0-3 ~]#
Despite all those errors, output seems to be fine
And this is what inside my booster.log
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:55:05] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:56:27] E [libglusterfsclient.c:4194:__glusterfs_stat]
libglusterfsclient: path lookup failed for (/hosts)
[2009-08-02 14:56:37] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
[2009-08-02 14:57:00] E [libglusterfsclient.c:4194:__glusterfs_stat]
libglusterfsclient: path lookup failed for (/issue)
[2009-08-02 14:57:07] E [libglusterfsclient.c:2244:glusterfs_getxattr]
libglusterfsclient: invalid argument: value
Then, I try to LD_PRELOAD apache (prefork). I change the target from
/gluster to /usr/home instead (the web application needs it). Then I tried
to strace the httpd process and found that httpd crash at the points where
httpd tried to read configuration file stored on Gluster volume (bad file
descriptor). I will try to move this configuration file some other places
and test again.
2009/7/31 Raghavendra G <raghavendra at gluster.com>
> Hi,
>
> On Thu, Jul 30, 2009 at 11:39 AM, Somsak
> Sriprayoonsakul<somsaks at gmail.com> wrote:
> > Thank you very much for you reply
> >
> > At the time we used 2.0.3, and yes we used stock Apache from CentOS. I
> will
> > try 2.0.4 very soon to see if it's work.
> >
> > For Booster, it seems not working correctly for me. Booster complains a
> lots
> > of error with plain 'ls' command (but giving the correct output). Also,
> with
>
> Can you mail those errors?
>
> > booster, Apache process refuse to start. I will try 2.0.4 to see if it
> > improves. If not, I will attach error log next time.
>
> logs are very much appreciated.
>
> >
> >
> > 2009/7/30 Raghavendra G <raghavendra at gluster.com>
> >>
> >> Hi Somsak,
> >>
> >> Sorry for the delayed reply. Below you've mentioned that you've problems
> >> with apache and booster. Going forward, Apache over booster will be the
> >> preferred approach. Can you tell us what version of glusterfs you are
> using?
> >> And as I can understand you are using apache 2.2, am I correct?
> >>
> >> regards,
> >> ----- Original Message -----
> >> From: "Liam Slusser" <lslusser at gmail.com>
> >> To: "Somsak Sriprayoonsakul" <somsaks at gmail.com>
> >> Cc: gluster-users at gluster.org
> >> Sent: Saturday, July 25, 2009 3:46:14 AM GMT +04:00 Abu Dhabi / Muscat
> >> Subject: Re: [Gluster-users] Gluster 2.0.3 + Apache on CentOS5
> performance
> >> issue
> >>
> >> I haven't tried an apples to apples comparison with Apache+mod_gluster
> vs
> >> Apache+fuse+gluster however i do run both setups. I load tested both
> >> setups
> >> so to verified it could handle 4x our normal daily load and left it at
> >> that.
> >> I didn't actually compare the two (although that might be cool to do
> >> someday).
> >> I really like the idea of Apache+mod_gluster as I don't have to deal
> with
> >> the whole fuse and mounting the filesystem. It always scares me having
> a
> >> public facing webserver with your whole backend fileshare mounted
> locally.
> >> Its very slick for serving content such as media files. We serve audio
> >> content to our CDN with a pair of Apache/mod_gluster servers - pushing
> >> 200-300mbit on average daily and everything works very well.
> >>
> >> We run an apache+fuse+gluster setup because we need to run some mod_perl
> >> before serving the actual content. However performance is still very
> >> good.
> >> We do around 50-100 requests (all jpeg images) per second off of a fuse
> >> mount and everything works great. We also have a java
> tomcat+fuse+gluster
> >> service which does image manipulation on the fly off of a gluster mount.
> >>
> >> We have two backend gluster servers using replication which serve all
> this
> >> content.
> >>
> >> If you would like more information on our setup id be happy to share
> >> offline. Just email me privately.
> >>
> >> thanks,
> >> liam
> >>
> >> On Fri, Jul 24, 2009 at 8:08 AM, Somsak Sriprayoonsakul
> >> <somsaks at gmail.com>wrote:
> >>
> >> > Oh thank you, thought noone will reply me :)
> >> >
> >> > Have you tried Apache + Fuse over GlusterFS? How is the performance?
> >> >
> >> > Also, anyone in this mailing-list have tried Apache with booster? I
> >> > tried
> >> > it but Apache refuse to start (just hang and freeze).
> >> >
> >> > 2009/7/23 Liam Slusser <lslusser at gmail.com>
> >> >
> >> >
> >> >> We use mod_gluster and Apache
> >> >> 2.2 with good results. We also ran into the same issue as you that
> we
> >> >> ran out of memory past 150 threads even on a 8gig machine. We got
> around
> >> >> this by compiling Apache using mpm-worker
> >> >> (threads) instead of prefork - it uses 1/4 as much ram with the same
> >> >> number
> >> >> of connections (150-200) and everything has been running smoothly. I
> >> >> cannot
> >> >> see any performance difference except it using way less memory.
> >> >> liam
> >> >>
> >> >>
> >> >> On Sun, Jul 12, 2009 at 5:11 AM, Somsak Sriprayoonsakul <
> >> >> somsaks at gmail.com> wrote:
> >> >>
> >> >>> Hello,
> >> >>>
> >> >>> We have been evaluating the choice for the new platform for a
> webboard
> >> >>> system.
> >> >>> The webboard is PHP scripts that generate/modify HTML page when user
> >> >>> posting/add comment to the page, resulting topic is actually stored
> as
> >> >>> a
> >> >>> HTML file with all related file (file attach to the topic, etc..
> >> >>> )stored in
> >> >>> its own directory for each topic. In general, the web site mostly
> >> >>> serve a
> >> >>> lot of small static files using Apache while using PHP to do other
> >> >>> dynamic
> >> >>> contents. This system has been working very well in the past, with
> the
> >> >>> increasing page view rate, it is very likely that we will need some
> >> >>> kind of
> >> >>> Cluster file system as backend very soon.
> >> >>>
> >> >>> We have set up a test system using Grinder as stress test tool. The
> >> >>> test
> >> >>> system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock
> >> >>> Apache
> >> >>> (prefork, since the goal is to use this with PHP), linked together
> >> >>> with
> >> >>> Gigabit Ethernet. We try to compare the performance of either using
> >> >>> single
> >> >>> NFS server in sync mode against using 4 Gluster nodes (distribute of
> 2
> >> >>> replicated nodes) through Fuse. However, the transaction per second
> >> >>> (TPS)
> >> >>> result is not good.
> >> >>>
> >> >>> NFS (single server, sync mode)
> >> >>> - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, mean
> >> >>> response time = 61.63 ms
> >> >>> - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33
> ms
> >> >>> - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms
> >> >>> - 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt =
> >> >>> 287.33ms
> >> >>>
> >> >>> 4 nodes Gluster (2 distribute of replicated 2 node)
> >> >>> - 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt =
> 207.33ms
> >> >>> - 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt =
> >> >>> 672.67ms
> >> >>> - 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = 931.33
> >> >>> (no 400-600 threads since we run out of client machine, sorry).
> >> >>>
> >> >>> gfsd is configured to use 32 thread of iothread as brick. gfs-client
> >> >>> is
> >> >>> configured to use
> >> >>> io-cache->write-behind->readahead->distribute->replicate.
> >> >>> io-cache cache-size is 256MB. I used patched Fuse downloaded from
> >> >>> Gluster
> >> >>> web-site (build through DKMS).
> >> >>>
> >> >>> As the result yield, it seems that Gluster performance worse with
> >> >>> increasing no. of client. One observation is that the glusterfs
> >> >>> process on
> >> >>> client is taking about 100% of CPU during all the tests. glusterfsd
> is
> >> >>> utilizing only 70-80% of CPUs during the test time. Note that system
> >> >>> is Dual
> >> >>> core.
> >> >>>
> >> >>> I also tried using modglusterfs and not using fuse at all to serve
> all
> >> >>> the static files and conduct another test with Grinder. The result
> is
> >> >>> about
> >> >>> the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in
> this
> >> >>> test
> >> >>> that each Apache prefork process used more about twice more memory
> and
> >> >>> we
> >> >>> need to lower number of httpd processes by about half.
> >> >>>
> >> >>> I tried disable EnableMMAP and it didn't help much. Adjusting
> >> >>> readahead,
> >> >>> write behind according to GlusterOptimization page didn't help much
> >> >>> either.
> >> >>>
> >> >>> My question is, there seems to be bottleneck in this setup, but how
> >> >>> can I
> >> >>> track this? Note that, I didn't do any other optimization other than
> >> >>> what
> >> >>> said above. Are there any best practice configuration for using
> Apache
> >> >>> to
> >> >>> serve a bunch of small static files like this around?
> >> >>>
> >> >>> Regards,
> >> >>>
> >> >>> Somsak
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> _______________________________________________
> >> >>> Gluster-users mailing list
> >> >>> Gluster-users at gluster.org
> >> >>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
> >> >>>
> >> >>>
> >> >>
> >> >
> >>
> >> _______________________________________________
> >> Gluster-users mailing list
> >> Gluster-users at gluster.org
> >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> >
> >
>
>
> regards,
> --
> Raghavendra G
>
More information about the Gluster-users
mailing list