[Gluster-devel] Speeding up file creation
Erik Osterman
e at osterman.com
Mon Mar 12 18:49:02 UTC 2007
I do believe I'm using that translator appropriately in the
configuration below. Is there a more effective way of applying the
translators? E.g. order of application. Right now, I have it as the last
translator in the chain. My thinking was this would then apply to the
operation as a whole.
Erik Osterman
Krishna Srinivas wrote:
> Erik,
> check out write behind translator:
> http://www.gluster.org/docs/index.php/GlusterFS_User_Guide#Write_Behind_Translator
>
> Krishna
>
> On 3/13/07, Erik Osterman <e at osterman.com> wrote:
>> Thank you Julien, that's very useful for comparison. I should have tried
>> that myself first. So, it would appear that with AFR enabled, the
>> transaction is not complete until the entire request (replicating the
>> file to the mirror site) is completed. This makes sense from a
>> consistency perspective; however, has there been any talk of adding an
>> option of "lazy" AFR, where writes to the mirror site would not block
>> the transaction? We are not overly concerned with this level of
>> consistency and can periodically rsync the data, or when the f*
>> utilities some out, run some sort of repair.
>>
>> Best,
>>
>> Erik Osterman
>>
>> Julien Perez wrote:
>> > Hello,
>> >
>> > By looking at your configuration, my first guess is that you have
>> > latency issues in your network, which would definitely explain that
>> > awful performances. So using your configuration files, I've set up the
>> > same architecture, but locally (one computer running on Sempron 2600+
>> > with 2 hitachi SATA1 disks in RAID1 software) and here are the results
>> > I got:
>> >
>> > toad at web1:~/afr$ time for i in {1..100}; do touch ./mnt/test.$i; done
>> >
>> > real 0m0.560s
>> > user 0m0.160s
>> > sys 0m0.190s
>> > toad at web1:~/afr$ rm -f ./mnt/*
>> > toad at web1:~/afr$ time for i in {1..10000}; do touch ./mnt/test.$i;
>> done
>> >
>> > real 1m8.060s
>> > user 0m16.680s
>> > sys 0m18.180s
>> > toad at web1:~/afr$ find ./mnt/ -type f | xargs -n100 rm -f
>> > toad at web1:~/afr$ time for i in {1..1000}; do touch ./mnt/test.$i; done
>> >
>> > real 0m5.829s
>> > user 0m1.670s
>> > sys 0m1.910s
>> >
>> >
>> > So my advise would be: check your network :)
>> >
>> > Hope it helped,
>> >
>> > Have a nice day everyone
>> >
>> > Julien Perez
>> >
>> >
>> > On 3/12/07, *Erik Osterman* <e at osterman.com <mailto:e at osterman.com>>
>> > wrote:
>> >
>> > I've configured a cluster with replication that uses most of the
>> > advanced features you've implemented including io-threads, afr,
>> > readahead, and writebehind. I am very satisfied with the write
>> > performance, but the file creation performance leaves much to be
>> > desired. What can we do to speed this up?
>> >
>> > Creating 100 empty files
>> >
>> > # time for i in {1..100}; do touch test.$i;done
>> > real 0m46.913s
>> > user 0m0.023s
>> > sys 0m0.067s
>> >
>> > That's about 0.500 seconds just to create an empty file.
>> >
>> >
>> > In general, what do you advise for tuning the performance of
>> > reading/writing tons of tiny files. Can the client use
>> io-threads to
>> > improve performance? Right now, our application stuffs all the
>> tiny
>> > files in a single directory. Eventually, we were planning on
>> hashing
>> > them out to directories. Would hashing them out into multiple
>> > directories positively and significantly affect the performance of
>> > GlusterFS?
>> >
>> >
>> >
>> > Best,
>> >
>> > Erik Osterman
>> >
>> >
>> >
>> >
>> >
>> > For what it's worth, here are my configurations:
>> >
>> > #
>> > # Master
>> > #
>> > volume posix0
>> > type storage/posix # POSIX FS translator
>> > option directory /home/glusterfs # Export this directory
>> > end-volume
>> >
>> > volume brick0
>> > type performance/io-threads
>> > option thread-count 8
>> > option queue-limit 1024
>> > subvolumes posix0
>> > end-volume
>> >
>> > ### Add network serving capability to above brick.
>> > volume server
>> > type protocol/server
>> > option transport-type tcp/server # For TCP/IP transport
>> > # option bind-address 192.168.1.10 <http://192.168.1.10> #
>> > Default is to listen on all
>> > interfaces
>> > option listen-port 6996 # Default is 6996
>> > option client-volume-filename /etc/glusterfs/client.vol
>> > subvolumes brick0
>> > option auth.ip.brick0.allow * # access to "brick" volume
>> > end-volume
>> >
>> >
>> >
>> > #
>> > # Mirror
>> > #
>> > volume posix0
>> > type storage/posix # POSIX FS translator
>> > option directory /home/glusterfs # Export this directory
>> > end-volume
>> >
>> > volume mirror0
>> > type performance/io-threads
>> > option thread-count 8
>> > option queue-limit 1024
>> > subvolumes posix0
>> > end-volume
>> >
>> > ### Add network serving capability to above brick.
>> > volume server
>> > type protocol/server
>> > option transport-type tcp/server # For TCP/IP transport
>> > # option bind-address 192.168.1.11 <http://192.168.1.11> #
>> > Default is to listen on all
>> > interfaces
>> > option listen-port 6996 # Default is 6996
>> > option client-volume-filename /etc/glusterfs/client.vol
>> > subvolumes mirror0
>> > option auth.ip.mirror0.allow * # access to "brick"
>> volume
>> > end-volume
>> >
>> >
>> > #
>> > # Client
>> > #
>> >
>> > ### Add client feature and attach to remote subvolume of server
>> > volume brick0
>> > type protocol/client
>> > option transport-type tcp/client # for TCP/IP transport
>> > option remote-host 216.182.237.155 <http://216.182.237.155> #
>> > IP address of the remote brick
>> > server
>> > option remote-port 6996 # default server port is
>> 6996
>> > option remote-subvolume brick0 # name of the remote
>> volume
>> > end-volume
>> >
>> > ### Add client feature and attach to remote mirror of brick0
>> > volume mirror0
>> > type protocol/client
>> > option transport-type tcp/client # for TCP/IP transport
>> > option remote-host 216.55.170.26 <http://216.55.170.26> #
>> > IP address of the remote
>> > mirror server
>> > option remote-port 6996 # default server port is
>> 6996
>> > option remote-subvolume mirror0 # name of the remote
>> volume
>> > end-volume
>> >
>> > ### Add AFR feature to brick
>> > volume afr0
>> > type cluster/afr
>> > subvolumes brick0 mirror0
>> > option replicate *:2 # All files 2 copies
>> (RAID-1)
>> > end-volume
>> >
>> > ### Add unify feature to cluster the servers. Associate an
>> > ### appropriate scheduler that matches your I/O demand.
>> > volume bricks
>> > type cluster/unify
>> > subvolumes afr0
>> > ### ** Round Robin (RR) Scheduler **
>> > option scheduler rr
>> > option rr.limits.min-free-disk 2GB
>> > end-volume
>> >
>> > ### Add performance feature
>> > volume writebehind
>> > type performance/write-behind
>> > option aggregate-size 131072 # aggregate block size in bytes
>> > subvolumes bricks
>> > end-volume
>> >
>> > ### Add performance feature
>> > volume readahead
>> > type performance/read-ahead
>> > option page-size 131072
>> > option page-count 16
>> > subvolumes writebehind
>> > end-volume
>> >
>> >
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > Gluster-devel mailing list
>> > Gluster-devel at nongnu.org <mailto:Gluster-devel at nongnu.org>
>> > http://lists.nongnu.org/mailman/listinfo/gluster-devel
>> > <http://lists.nongnu.org/mailman/listinfo/gluster-devel>
>> >
>> >
>>
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at nongnu.org
>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
More information about the Gluster-devel
mailing list