mod_glusterfs (was Re: [Gluster-devel] Hammering start() calls)

[Anand Avati]

> Sorry for the late reply.
>
> The workload is a cluster of web servers with PHP serving from a single
> glusterfs mount point. All web servers mount the same single share from a
> file server connected via GE.
>
> The web servers use internal stat caches, so when serving static files the
> performance is perfect.
>
> The problem is PHP hammering stat() for each file that gets included or
> searched in include paths. PHP is always asking for the same set of about
> 100 files or so.


the entry/attribute timeout should definitely help in these cases. try
setting it to a high value like 600 (10mins).

I hoped that iocache would solve that, but I can push only 1/15 of requests,
> compared to when serving directly from the backend file system.


io-cache does help to a certain extent. you could also try loading
write-behind and set 'option flush-behind on'. This will cut close() times
even if you are not writing data.

File are like 1KB each.
>
> Would booster help me? Is it fuse causing the latency?


booster will not help you here. fuse involves context switches which are
sometimes a bit too expensive for small files.

You might also be interested in mod_glusterfs. This is an apache module
which lets you run the filesystem inside apache address space (in a seperate
thread) which can greatly boost the performance of serving static files.
This module is available if you checkout glusterfs--mainline--3.0. Currently
mod_glusterfs works with apache-1.3.x, but work for getting it ready for
apache2.x and lighttpd is on the way. The way to use it - (in your
httpd.conf)

<Location /prefix>
  GlusterfsVolumeSpecfile "/path/to/client.vol"
  GlusterfsLogfile "/tmp/glusterfs.log"
  GlusterfsLookupCacheTimeout 600
  GlusterfsStatCacheTimeout 600
  GlusterfsAsyncWindowSize 65536
  SetHandler "glusterfs-handler"
</Location>

now http://server/prefix will map to your "mountpoint". it does not yet
support directory listing.

use the same client.vol which you are currently using with fuse. Using
io-cache with a high "option force-revalidate-timeout 600" is a good idea.
Also using read-ahead for serving large static files is suggested.

One important thing to notice is that while using mod_glusterfs as the
client, the server authentication should be set to use 'login' instead of
'ip' (since apache would run as non root and not be able to bind to a port <
1024). see docs/auth.txt for details how to use the auth modules.

mod_glusterfs is still in beta stage and any feedback would be welcome.

avati