[Gluster-users] Slower than NFS
Anand Avati
avati at gluster.com
Wed Apr 1 17:23:49 UTC 2009
This is due to the NFS read caching. Use io-cache to have the
equivalent of the functionality in glusterfs enabled.
Avati
On Wed, Apr 1, 2009 at 6:31 PM, Greg <greg at easyflirt.com> wrote:
> Hi,
>
> I have a production setup, with clients that are both NFS and GlusterFS
> clients. I've made a _very_ quick benchmark (repeated ten times), to compare
> read performance between local hardrives, a NFS mount and a GlusterFS mount.
> - a 17k file :
> cache_c21f969b5f03d33d43e04f8f136e7682_c57d91c82f2ed2e96c13765e33fcd5ae
> - a 50M file : users0.new.spi
> - /nfs is a NFS mounted point, server with SATA drives
> - /gluster is a GlusterFS mounted point, 2x servers with SAS drives
>
> php-14:~# for f in
> /var/www/www/cache/cache_c/cache_cf/cache_cfb/cache_c21f969b5f03d33d43e04f8f136e7682_c57d91c82f2ed2e96c13765e33fcd5ae
> /nfs/tmp/cache_c21f969b5f03d33d43e04f8f136e7682_c57d91c82f2ed2e96c13765e33fcd5ae
> /tmp/cache_c21f969b5f03d33d43e04f8f136e7682_c57d91c82f2ed2e96c13765e33fcd5ae
> /gluster/sphinx/users0.new.spi /nfs/tmp/users0.new.spi /tmp/users0.new.spi;
> do echo $f; time cat $f >/dev/null; done
> /var/www/www/cache/cache_c/cache_cf/cache_cfb/cache_c21f969b5f03d33d43e04f8f136e7682_c57d91c82f2ed2e96c13765e33fcd5ae
>
> real 0m0.011s
> user 0m0.000s
> sys 0m0.000s
> /nfs/tmp/cache_c21f969b5f03d33d43e04f8f136e7682_c57d91c82f2ed2e96c13765e33fcd5ae
>
> real 0m0.002s
> user 0m0.000s
> sys 0m0.000s
> /tmp/cache_c21f969b5f03d33d43e04f8f136e7682_c57d91c82f2ed2e96c13765e33fcd5ae
>
> real 0m0.001s
> user 0m0.000s
> sys 0m0.000s
> /gluster/sphinx/users0.new.spi
>
> real 0m0.081s
> user 0m0.000s
> sys 0m0.068s
> /nfs/tmp/users0.new.spi
>
> real 0m0.037s
> user 0m0.000s
> sys 0m0.036s
> /tmp/users0.new.spi
>
> real 0m0.028s
> user 0m0.000s
> sys 0m0.028s
>
>
> Configuration:
> # file: /etc/glusterfs/glusterfsd.vol
>
> #
> # Volumes
> #
> volume media-small
> type storage/posix
> option directory /var/local/glusterfs/media_small
> end-volume
>
> volume media-medium
> type storage/posix
> option directory /var/local/glusterfs/media_medium
> end-volume
>
> # Lock posix
> volume media-small-locks
> type features/posix-locks
> option mandatory-locks on
> subvolumes media-small
> # subvolumes trash # enable this if you need trash can support (NOTE: not
> present in 1.3.0-pre5+ releases)
> end-volume
>
> volume media-medium-locks
> type features/posix-locks
> option mandatory-locks on
> subvolumes media-medium
> # subvolumes trash # enable this if you need trash can support (NOTE: not
> present in 1.3.0-pre5+ releases)
> end-volume
>
>
> #
> # Performance
> #
> volume media-small-iot
> type performance/io-threads
> subvolumes media-small-locks
> option thread-count 4 # default value is 1
> end-volume
>
> volume media-small-wb
> type performance/write-behind
> option flush-behind on # default is off
> subvolumes media-small-iot
> end-volume
>
> volume media-medium-iot
> type performance/io-threads
> subvolumes media-medium-locks
> option thread-count 4 # default value is 1
> end-volume
>
> volume media-medium-wb
> type performance/write-behind
> option flush-behind on # default is off
> subvolumes media-medium-iot
> end-volume
>
>
>
> #
> # Serveur
> #
> volume server
> type protocol/server
> option transport-type tcp/server
> option auth.addr.media-small-wb.allow 10.0.*.*
> option auth.addr.media-medium-wb.allow 10.0.*.*
> # Autoconfiguration, e.g. :
> # glusterfs -l /tmp/glusterfs.log --server=filer-04 ./Cache
> option client-volume-filename /etc/glusterfs/glusterfs.vol
> subvolumes media-small-wb media-medium-wb # volumes exportés
> end-volume
>
> # file: /etc/glusterfs/glusterfs.vol
>
>
> #
> # Clients
> #
> volume media-small-filer-04
> type protocol/client
> option transport-type tcp/client
> option remote-host filer-04.local
> option remote-subvolume media-small-wb
> end-volume
>
> volume media-small-filer-05
> type protocol/client
> option transport-type tcp/client
> option remote-host filer-05.local
> option remote-subvolume media-small-wb
> end-volume
>
> volume media-medium-filer-04
> type protocol/client
> option transport-type tcp/client
> option remote-host filer-04.local
> option remote-subvolume media-medium-wb
> end-volume
>
> volume media-medium-filer-05
> type protocol/client
> option transport-type tcp/client
> option remote-host filer-05.local
> option remote-subvolume media-medium-wb
> end-volume
>
>
> #
> # Volume principal
> #
> volume afr-small
> # AFR has been renamed to "Replicate" for simplicity.
> type cluster/replicate
> # Il faut mettre le serveur avec le moins d'espace disque en 1er :
> # "When doing a "df -h" on a client, the AVAILABLE disk space will
> display the maximum disk space of the first AFR sub volume defined in the
> spec file. So if you have two servers with 50 gigs and 100 gigs of free disk
> space, and the server with 100 gigs is listed first, then you will see 100
> gigs available even though one server only has 50 gigs free. "
> subvolumes media-small-filer-04 media-small-filer-05
> end-volume
>
> volume afr-medium
> # AFR has been renamed to "Replicate" for simplicity.
> type cluster/replicate
> subvolumes media-medium-filer-04 media-medium-filer-05
> end-volume
>
> #
> # Performance
> #
> volume iot-small
> type performance/io-threads
> subvolumes afr-small
> end-volume
>
> volume readahead-small
> type performance/read-ahead
> subvolumes iot-small
> option page-size 1MB # default is 256KB
> option page-count 4 # default is 2 - cache per file =
> (page-count x page-size)
> option force-atime-update no # defalut is 'no'
> end-volume
>
> volume iocache-small
> type performance/io-cache
> option cache-size 64MB # default is 32MB
> option page-size 256KB # default is 128KB
> subvolumes readahead-small
> end-volume
>
> volume writeback-small
> type performance/write-behind
> option block-size 1MB # max 4MB
> option flush-behind on # default is off
> subvolumes iocache-small
> end-volume
>
> volume iot-medium
> type performance/io-threads
> subvolumes afr-medium
> end-volume
>
> volume readahead-medium
> type performance/read-ahead
> subvolumes iot-medium
> option page-size 1MB # default is 256KB
> option page-count 4 # default is 2 - cache per file =
> (page-count x page-size)
> option force-atime-update no # defalut is 'no'
> end-volume
>
> volume iocache-medium
> type performance/io-cache
> option cache-size 64MB # default is 32MB
> option page-size 256KB # default is 128KB
> option page-count 2
> subvolumes readahead-medium
> end-volume
>
> volume writeback-medium
> type performance/write-behind
> option block-size 1MB # max 4MB
> option flush-behind on # default is off
> subvolumes iocache-medium
> end-volume
>
> Regards,
>
> --
> Greg
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>
More information about the Gluster-users
mailing list