mod_glusterfs (was Re: [Gluster-devel] Hammering start() calls)

[Hannes Dorbath]

Anand Avati wrote:
> You might also be interested in mod_glusterfs. This is an apache module
> which lets you run the filesystem inside apache address space (in a seperate
> thread) which can greatly boost the performance of serving static files.
> This module is available if you checkout glusterfs--mainline--3.0. Currently
> mod_glusterfs works with apache-1.3.x, but work for getting it ready for
> apache2.x and lighttpd is on the way. The way to use it - (in your
> httpd.conf)

That sounds great, however I'm currently bound to Lighttpd.

I've setup a clean test case to benchmark the issues I'm seeing. With -e 
600 -a 600 it seems I'm no longer bound to stat(), but to open() now.

Following is a trace of a "Hello World" PHP script (FCGI) as well as my 
current configuration.

I think I messed something with iothreads. Without GlusterFS total 
throughput increases a lot with multiple HTTP clients, with GlusterFS it 
decreases. I could not find any load where using iothreads improved my 
situation, so there still seems to be a single point of serialisation.

If you have any suggestion on how to improve this configuration, I'd 
really appreciative it.

Thanks in advance.


Strace with GlusterFS:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  43.99    0.053319           5     10000           open
  16.22    0.019654           2     10000           accept
  12.95    0.015696           1     20000           close
   7.66    0.009287           0     60000           read
   6.73    0.008154           0     20000           fstat
   3.53    0.004273           0     10000           write
   1.71    0.002071           0     10000           munmap
   1.61    0.001955           0     50000           gettimeofday
   1.33    0.001614           0     40000           fcntl
   1.07    0.001301           0     10000           mmap
   1.01    0.001222           0     10000           chdir
   0.51    0.000620           0     30000           setitimer
   0.45    0.000551           0     20000           recvfrom
   0.36    0.000442           0     10000           shutdown
   0.26    0.000313           0     20000           rt_sigaction
   0.25    0.000302           0     10000           select
   0.22    0.000261           0     20000           rt_sigprocmask
   0.14    0.000164           0     10000           lseek
------ ----------- ----------- --------- --------- ----------------
100.00    0.121199                370000           total


Strace without GlusterFS (XFS noatime, lazy-counters):

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  34.98    0.008714           1     10000           accept
  13.22    0.003293           0     60000           read
   9.41    0.002345           0     10000           write
   7.59    0.001890           0     10000           open
   6.81    0.001696           0     50000           gettimeofday
   5.34    0.001331           0     40000           fcntl
   5.07    0.001264           0     20000           close
   3.11    0.000774           0     10000           munmap
   2.77    0.000690           0     10000           chdir
   1.92    0.000478           0     20000           recvfrom
   1.91    0.000476           0     20000           fstat
   1.79    0.000446           0     30000           setitimer
   1.60    0.000399           0     10000           mmap
   1.09    0.000271           0     20000           rt_sigaction
   1.04    0.000260           0     10000           lseek
   0.93    0.000231           0     20000           rt_sigprocmask
   0.77    0.000191           0     10000           select
   0.65    0.000161           0     10000           shutdown
------ ----------- ----------- --------- --------- ----------------
100.00    0.024910                370000           total


The server config is:

volume webroot
   type storage/posix
   option directory /data/webs/default/webroot_server
end-volume

volume src
   type performance/io-threads
   option thread-count 8
   option cache-size 32MB
   subvolumes webroot
end-volume

volume server
   type protocol/server
   option transport-type tcp/server
   subvolumes webroot
   option auth.ip.webroot.allow *
end-volume


The client config:

volume client
   type protocol/client
   option transport-type tcp/client
   option remote-host localhost
   option transport-timeout 10
   option remote-subvolume webroot
end-volume

volume iot
   type performance/io-threads
   option thread-count 8
   option cache-size 32MB
   subvolumes client
end-volume

volume wb
   type performance/write-behind
   option aggregate-size 1MB
   option flush-behind on
   subvolumes iot
end-volume

volume io-cache
   type performance/io-cache
   option cache-size 64MB
   option page-size 128KB
   option force-revalidate-timeout 600
   subvolumes wb
end-volume


-- 
Best regards,
Hannes Dorbath