[Gluster-devel] Speeding up file creation
Brent A Nelson
brent at phys.ufl.edu
Mon Mar 12 18:24:32 UTC 2007
This probably has little to nothing to do with your performance, but I
don't thing cluster/unify is really doing anything in your present
configuration; you would need more subvolumes for it to be useful. I
assume you have it in there for when you do add more nodes?
Thanks,
Brent
On Mon, 12 Mar 2007, T0aD wrote:
> Hello,
>
> By looking at your configuration, my first guess is that you have latency
> issues in your network, which would definitely explain that awful
> performances. So using your configuration files, I've set up the same
> architecture, but locally (one computer running on Sempron 2600+ with 2
> hitachi SATA1 disks in RAID1 software) and here are the results I got:
>
> toad at web1:~/afr$ time for i in {1..100}; do touch ./mnt/test.$i; done
>
> real 0m0.560s
> user 0m0.160s
> sys 0m0.190s
> toad at web1:~/afr$ rm -f ./mnt/*
> toad at web1:~/afr$ time for i in {1..10000}; do touch ./mnt/test.$i; done
>
> real 1m8.060s
> user 0m16.680s
> sys 0m18.180s
> toad at web1:~/afr$ find ./mnt/ -type f | xargs -n100 rm -f
> toad at web1:~/afr$ time for i in {1..1000}; do touch ./mnt/test.$i; done
>
> real 0m5.829s
> user 0m1.670s
> sys 0m1.910s
>
>
> So my advise would be: check your network :)
>
> Hope it helped,
>
> Have a nice day everyone
>
> Julien Perez
>
> On 3/12/07, Erik Osterman <e at osterman.com> wrote:
>>
>> I've configured a cluster with replication that uses most of the
>> advanced features you've implemented including io-threads, afr,
>> readahead, and writebehind. I am very satisfied with the write
>> performance, but the file creation performance leaves much to be
>> desired. What can we do to speed this up?
>>
>> Creating 100 empty files
>>
>> # time for i in {1..100}; do touch test.$i;done
>> real 0m46.913s
>> user 0m0.023s
>> sys 0m0.067s
>>
>> That's about 0.500 seconds just to create an empty file.
>>
>>
>> In general, what do you advise for tuning the performance of
>> reading/writing tons of tiny files. Can the client use io-threads to
>> improve performance? Right now, our application stuffs all the tiny
>> files in a single directory. Eventually, we were planning on hashing
>> them out to directories. Would hashing them out into multiple
>> directories positively and significantly affect the performance of
>> GlusterFS?
>>
>>
>>
>> Best,
>>
>> Erik Osterman
>>
>>
>>
>>
>>
>> For what it's worth, here are my configurations:
>>
>> #
>> # Master
>> #
>> volume posix0
>> type storage/posix # POSIX FS translator
>> option directory /home/glusterfs # Export this directory
>> end-volume
>>
>> volume brick0
>> type performance/io-threads
>> option thread-count 8
>> option queue-limit 1024
>> subvolumes posix0
>> end-volume
>>
>> ### Add network serving capability to above brick.
>> volume server
>> type protocol/server
>> option transport-type tcp/server # For TCP/IP transport
>> # option bind-address 192.168.1.10 # Default is to listen on all
>> interfaces
>> option listen-port 6996 # Default is 6996
>> option client-volume-filename /etc/glusterfs/client.vol
>> subvolumes brick0
>> option auth.ip.brick0.allow * # access to "brick" volume
>> end-volume
>>
>>
>>
>> #
>> # Mirror
>> #
>> volume posix0
>> type storage/posix # POSIX FS translator
>> option directory /home/glusterfs # Export this directory
>> end-volume
>>
>> volume mirror0
>> type performance/io-threads
>> option thread-count 8
>> option queue-limit 1024
>> subvolumes posix0
>> end-volume
>>
>> ### Add network serving capability to above brick.
>> volume server
>> type protocol/server
>> option transport-type tcp/server # For TCP/IP transport
>> # option bind-address 192.168.1.11 # Default is to listen on all
>> interfaces
>> option listen-port 6996 # Default is 6996
>> option client-volume-filename /etc/glusterfs/client.vol
>> subvolumes mirror0
>> option auth.ip.mirror0.allow * # access to "brick" volume
>> end-volume
>>
>>
>> #
>> # Client
>> #
>>
>> ### Add client feature and attach to remote subvolume of server
>> volume brick0
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> option remote-host 216.182.237.155 # IP address of the remote brick
>> server
>> option remote-port 6996 # default server port is 6996
>> option remote-subvolume brick0 # name of the remote volume
>> end-volume
>>
>> ### Add client feature and attach to remote mirror of brick0
>> volume mirror0
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> option remote-host 216.55.170.26 # IP address of the remote
>> mirror server
>> option remote-port 6996 # default server port is 6996
>> option remote-subvolume mirror0 # name of the remote volume
>> end-volume
>>
>> ### Add AFR feature to brick
>> volume afr0
>> type cluster/afr
>> subvolumes brick0 mirror0
>> option replicate *:2 # All files 2 copies (RAID-1)
>> end-volume
>>
>> ### Add unify feature to cluster the servers. Associate an
>> ### appropriate scheduler that matches your I/O demand.
>> volume bricks
>> type cluster/unify
>> subvolumes afr0
>> ### ** Round Robin (RR) Scheduler **
>> option scheduler rr
>> option rr.limits.min-free-disk 2GB
>> end-volume
>>
>> ### Add performance feature
>> volume writebehind
>> type performance/write-behind
>> option aggregate-size 131072 # aggregate block size in bytes
>> subvolumes bricks
>> end-volume
>>
>> ### Add performance feature
>> volume readahead
>> type performance/read-ahead
>> option page-size 131072
>> option page-count 16
>> subvolumes writebehind
>> end-volume
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at nongnu.org
>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
More information about the Gluster-devel
mailing list