[Gluster-devel] glusterfs-1.3.8pre1

[Sascha Ottolski]

Am Freitag 22 Februar 2008 18:52:37 schrieb Dan Podeanu:
> Current configuration:
>
> - 16 clients / 16 servers (one client/server on each machine)
> - servers are dual opteron, some of them quad core, 8 or 12 gb ram
> - kernel 2.6.24-2, linux gentoo (can provide gluster ebuilds)
> - fuse 2.7.2glfs8, glusterfs 1.3.7 - see config files- basicly a
> simple unify with no ra/wt cache


Hi Dan,

sorry, I can't give you very much advice.

However, answering your first question, my experiemnts showed that you 
need to upgrade all clients and servers at once if you want to move 
from 1.3.0pre4 to 1.3.7 or later; the different version seem not to be 
able to communicate with each other. I guess a more informative error 
message in the logs would be most helpful...

I'd like to ask if you could share your experience with using glusterfs 
for serving your images; that is, what's the performance of your setup?

I'm trying to do something similar, serving about 20 mio. image files 
with (currently 7) webservers that read them from a 4 server, afr/unify 
gluster mount. unfortunately, the performance is not really making me 
happy, in the live application it never goes beyond ~40 req/sec per 
werbserver, that is, the accumulated performance ist below 300 req/sec. 
I did experiement with different websevers, with varying and sometimes 
amazing results (see my earlier posts if you may). You say you see 300 
reads/second, is this per server, or for the whole cluster? My goal 
would be to achieve serveral thousands requests/second...

If I understand, correctly, you have no afr in your setup, and you keep 
some of the files local to the webserver. I'm wondering, do you have a 
load balancing in place that tries to schedulde the requests in a way 
that the chance is high that the respective webserver has the file in 
question on it's local share?

I also see the memory leak, but as I'm still running a 1.3.0pre4 the 
reason may not be the same. What might be noteable though: The leaks 
only appear on glusterfs clients that to heavy reading (i.e.: the 
webservers). I also have mounted roughly 20 app servers that do writes 
to the gluster-servers, but there appears not to happen such leakage 
(just checked that on one of these servers the glusterfs process 
footprint is as small as 10 MB).

FWIW, I experimented a lot with read-ahead, write-behind, io-threads, 
but could never see any significant difference performance-wise. For 
io-cache, I've seen a gain by roughly 100% in some setups, but none at 
all in others (that is, apache gains, nginx and lighttpd don't).


Thanks,

Sascha