[Gluster-users] Shared web hosting with GlusterFS and inotify

[phil cryer]

We're interested in this as well, as we will be serving our docroot
from a GlusterFS share. Have you tried nginx? I have not tested this,
but after your benchmarks it sounds like I need to. Your inotifiy
script looks like it would work, but it wouldn't for me; we use
GlusterFS so we can store ~70TB of data, which we can't copy to the
regular filesystem. This is a big risk indeed - can you share your
benchmarking method? Did you simply use ab?

Thanks for the heads up

P

On Wed, Sep 15, 2010 at 9:58 AM, Emile Heitor
<emile.heitor at nbs-system.com> wrote:
> Hi list,
>
> For a couple of weeks, we're experimenting a web hosting system based on
> GlusterFS in order to share customers documentroots between
> more-than-one machine.
>
> Involved hardware and software are :
>
> Two servers composed of 2x Intel 5650 (i.e. 2x12 cores @2,6Ghz), 24GB
> DDR3 RAM, 146GB SAS disks / RAID 1
> Both servers running 64bits Debian Lenny GNU/Linux with GlusterFS 3.0.5
> The web server is Apache 2.2, the application is a huge PHP/MySQL monster.
>
> For our first naive tests were using the glusterfs mountpoint as
> apache's documentroot. In short, performances were catastrophic.
> A single of these servers, without GlusterFS, is capable of handling
> about 170 pages per second with 100 concurrent users.
> The same server, with apache documentroot being a gluster mountpoint,
> drops to 5 PPS for 20 CU and just stops responding for 40+.
>
> We tried a lot of tips (quick-read, io-threads, io-cache, thread-count,
> timeouts...) we read on this very mailing list, various websites, or
> experiences on our own, we never got better than 10 PPS / 20 users.
>
> So we took another approach: instead of declaring gluster mountpoint as
> the documentroot, we declared the local storage, but of course, without
> any modification, this would lead to inconsistencies if by any chance
> apache writes something (.htaccess, tmp file, log...). And so enters
> inotify. Using inotify-tools's "inotifywait", we have this little script
> watching for local documentroot modifications, duplicating them to the
> glusterfs share. The infinite loop is avoided by a md5 comparison. Here
> a very early proof of concept :
>
> #!/bin/sh
>
> [ $# -lt 2  ]&&  echo "usage: $0<source>  <destination>"&&  exit 1
>
> PATH=${PATH}:/bin:/sbin:/usr/bin:/usr/sbin; export PATH
>
> SRC=$1
> DST=$2
>
> cd ${SRC}
>
> # no recursion
> RSYNC='rsync -dlptgoD --delete "${srcdir}" "${dstdir}/"'
>
> inotifywait -mr \
>    --exclude \..*\.sw.* \
>    -e close_write -e create -e delete_self -e delete . | \
>    while read dir action file
>    do
>        srcdir="${SRC}/${dir}"
>        dstdir="${DST}/${dir}"
>
>        [ -d "${srcdir}" ]&&  \
>        [ ! -z "`df -T \"${srcdir}\"|grep tmpfs`" ] \
> &&  continue
>
>        # debug
>        echo ${dir} ${action} ${file}
>
>        case "${action}" in
>        CLOSE_WRITE,CLOSE)
>            [ ! -f "${dstdir}/${file}" ]&&  eval ${RSYNC}&&  continue
>
>            md5src="`md5sum \"${srcdir}/${file}\"|cut -d' ' -f1`"
>            md5dst="`md5sum \"${dstdir}/${file}\"|cut -d' ' -f1`"
>            [ ! $md5src == $md5dst ]&&  eval ${RSYNC}
>            ;;
>        CREATE,ISDIR)
>            [ ! -d "${dstdir}/${file}" ]&&  eval ${RSYNC}
>            ;;
>        DELETE|DELETE,ISDIR)
>            eval ${RSYNC}
>            ;;
>        esac
>    done
>
> As for now a gluster mountpoint is barely unusable as an Apache
> DocumentRoot for us (and yes, with htaccess disabled), i'd like to have
> the list's point of view on this approach. Do you see any terrible glitch ?
>
> Thanks in advance,
>
> --
> Emile Heitor, Responsable d'Exploitation
> ---
> www.nbs-system.com, 140 Bd Haussmann, 75008 Paris
> Tel: 01.58.56.60.80 / Fax: 01.58.56.60.81
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>



-- 
http://philcryer.com