[Gluster-devel] webservers vs. glusterfs vs. namespace

Sun Jan 20 23:03:14 UTC 2008

Am Sonntag 20 Januar 2008 14:39:38 schrieb Angel:
> find /glustefs/mntpnt/* -type f -print0 | xargs -0 -l100 -n100 -P N
> md5sum
>
> how the sistem performs as N increases?
> Is this test as slow as apache2 access?

Angel,

thanks a lot for your hints, I will give it a try. When it comes to 
apache2, I think I found a configuration mistake that was part of the 
problem: Each request for a file resulted in ca. a dozen additional 
calls for (non-existing) .htaccess files (the files are always deep 
down a directory tree. After fixing this, apache2 performs almost like 
apache-1.3, and io-cache then does a good job also. However, apache2 
(with worker) puts it's machine under significant higher load than 
apache-1.3.

I already experimented with EnableMMAP off, but did not see any effect. 
That might be cause the files to serve are never written to, and only 
rarely deleted. But I've experienced another strange effect: If I 
enable mod_disk_cache, the "siege" benchmark constantly segfaults, and 
even the glusterfs crashes sometimes. I can't find any logic in this.

I'm still puzzled though, why nginx and lighttpd can't benefit from the 
io-cache, and show relatively poor performance. I suspect that there 
exist a relation to both phenomenons, but I still can't figure what 
that might be :-(

Thanks, Sascha

>
> from: http://httpd.apache.org/docs/2.0/misc/perf-tuning.html
> you can read that
>
> Memory-mapping
>
> In situations where Apache 2.x needs to look at the contents of a
> file being delivered--for example, when doing server-side-include
> processing--it normally memory-maps the file if the OS supports some
> form of mmap(2).
>
> On some platforms, this memory-mapping improves performance.
> However, there are cases where memory-mapping can hurt the
> performance or even the stability of the httpd:
>
> On some operating systems, mmap does not scale as well as read(2)
> when the number of CPUs increases. On multiprocessor Solaris servers,
> for example, Apache 2.x sometimes delivers server-parsed files faster
> when mmap is disabled. If you memory-map a file located on an
> NFS-mounted filesystem and a process on another NFS client machine
> deletes or truncates the file, your process may get a bus error the
> next time it tries to access the mapped file content.
>
> For installations where either of these factors applies, you should
> use EnableMMAP off to disable the memory-mapping of delivered files.
> (Note: This directive can be overridden on a per-directory basis.)
>
> Can you rey EnableMMAP off?
>
> ¿Is mmaping the problem?
>
> El Domingo, 20 de Enero de 2008 Sascha Ottolski escribió:
> > Am Sonntag 20 Januar 2008 01:38:28 schrieb Anand Avati:
> > > Sascha,
> > >  Please run by removing AFR (make AFR's first subvolume a direct
> > > subvolume of unify). AFR can potentially flush io-cache pages
> > > while doing read scheduling across subvolumes (by returning
> > > inconsistent mtimes with the read buffer). we are working on
> > > fixing this.
> > >
> > > avati
> >
> > I will try this, but isn't it still strange that the caching _does_
> > work with the exact same config for apache-1.3? I would hope that
> > this different behaviour could be related to the unexpected
> > slowness of the other werbservers (which also is there with caching
> > disabled!).
> >
> > Thanks, Sascha
> >
> > > 2008/1/20, Sascha Ottolski <ottolski at web.de>:
> > > > Am Sonntag 20 Januar 2008 01:14:06 schrieb Anand Avati:
> > > > > can you please share your spec files with
> > > > > glusterfs.pastebin.com ?
> > > >
> > > > my pleasure: http://glusterfs.pastebin.com/m8820300
> > > >
> > > >
> > > > Thanks for your time,
> > > >
> > > > Sascha
> > > >
> > > > > thanks,
> > > > > avati
> > > > >
> > > > > 2008/1/20, Sascha Ottolski <ottolski at web.de>:
> > > > > > Am Samstag 19 Januar 2008 11:03:43 schrieb Sascha Ottolski:
> > > > > > > Am Freitag 18 Januar 2008 17:49:15 schrieb Anand Avati:
> > > > > > > > Sascha,
> > > > > > > >  the reason why 1.3.0pre4 might be faster would not be
> > > > > > > > because of the missing namespace, but most likely
> > > > > > > > because of missing self-heal. can you try with 'option
> > > > > > > > self-heal off' in the unify section?
> > > > > > >
> > > > > > > may ask again, any idea why the old apache-1.3 performs
> > > > > > > way better on either gluster version than the others? or
> > > > > > > any idea which knobs to tweak to get more out of the
> > > > > > > others?
> > > > > >
> > > > > > now, another astonishing observation: if I enable the
> > > > > > io-cache, it has a good effect for apache1 (almost doubles
> > > > > > the requests/second), but almost none for apache2, nginx
> > > > > > and lighttpd.
> > > > > >
> > > > > > could this help to understand more about the performance
> > > > > > differences?
> > > > > >
> > > > > >
> > > > > > Thanks, Sascha
> > > > > >
> > > > > > > usally, for static files from a local fileseystem, one
> > > > > > > would expect that nginx and lighttpd would outperform the
> > > > > > > apaches remarcably...may be my observations have a common
> > > > > > > cause with those of
> > > > > > > http://lists.nongnu.org/archive/html/gluster-devel/2008-0
> > > > > > >1/ms g001 42.h tml ?
> > > > > > >
> > > > > > >
> > > > > > > Thanks a lot, Sascha
> > > > > > >
> > > > > > > > are the test results same for multiple runs too?
> > > > > > > >
> > > > > > > > avati
> > > > > > > >
> > > > > > > > 2008/1/18, Sascha Ottolski <ottolski at web.de>:
> > > > > > > > > Hi Folks,
> > > > > > > > >
> > > > > > > > > I'm wondering if anyone might have some general
> > > > > > > > > advices if I miss something important in my test
> > > > > > > > > setup. I'm trying to figure out how to tweak the
> > > > > > > > > configs to achieve the best performance, but get
> > > > > > > > > result that feel strange to me. I will post some
> > > > > > > > > numbers at a later point, but up to now what I
> > > > > > > > > discovered is:
> > > > > > > > >
> > > > > > > > > - glusterfs without a namespace (1.3.0pre4) seems to
> > > > > > > > > be significant faster than with namespace (tla
> > > > > > > > > patch-628)
> > > > > > > > >
> > > > > > > > > that seems to logical, at least I would expect some
> > > > > > > > > overhead for the namespace.
> > > > > > > > >
> > > > > > > > > what i absolutely not understand is, how different
> > > > > > > > > the webservers perform. i tested with
> > > > > > > > >
> > > > > > > > >     siege -f /tmp/siege-urls.txt.new -c100 -i -r50 -b
> > > > > > > > >
> > > > > > > > > with up to 3 sessions in parellel, each firing it's
> > > > > > > > > requests to a seperate webserver (on seperate
> > > > > > > > > machines, of course).
> > > > > > > > >
> > > > > > > > > up to now my ranking by means of requests/per second
> > > > > > > > > is something like
> > > > > > > > >
> > > > > > > > > 630 | apache
> > > > > > > > > 430 | apache2 (worker)
> > > > > > > > > 350 | nginx
> > > > > > > > > 250 | lighttpd
> > > > > > > > >
> > > > > > > > > (with 1.3.0pre4 and no namespace, the best I've seen
> > > > > > > > > was apache2 with about 900, apache still 750). I must
> > > > > > > > > admit that up to now I did not compare it to local
> > > > > > > > > filesystem, but from my past experiences with
> > > > > > > > > webservers I would expect nginx and lighttpd way
> > > > > > > > > ahead of the apaches...
> > > > > > > > >
> > > > > > > > > Also, I exprimented a bit with different settings for
> > > > > > > > > io-threads on the server (1, 2, 4, 8, and cache-size
> > > > > > > > > 64 or 128MB), but that didn't seem to make much of a
> > > > > > > > > difference. Same with read-ahead (which seems
> > > > > > > > > logical, as I test with relatively small images).
> > > > > > > > >
> > > > > > > > > So far I did not try the booster. I use
> > > > > > > > > fuse-2.7.0-glfs7. I also did not try the latest tla
> > > > > > > > > nor fuse-2.7.2-glfs8.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks a lot for any pointer,
> > > > > > > > >
> > > > > > > > > Sascha
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > _______________________________________________
> > > > > > > > > Gluster-devel mailing list
> > > > > > > > > Gluster-devel at nongnu.org
> > > > > > > > > http://lists.nongnu.org/mailman/listinfo/gluster-deve
> > > > > > > > >l
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > Gluster-devel mailing list
> > > > > > > Gluster-devel at nongnu.org
> > > > > > > http://lists.nongnu.org/mailman/listinfo/gluster-devel
> > > > > >
> > > > > > _______________________________________________
> > > > > > Gluster-devel mailing list
> > > > > > Gluster-devel at nongnu.org
> > > > > > http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at nongnu.org
> > http://lists.nongnu.org/mailman/listinfo/gluster-devel