[Gluster-devel] Speeding up file creation

Mon Mar 12 18:35:56 UTC 2007

Yes, at the moment cluster/unify is just an extra layer. The goal was to 
write the config using all the layers we plan to use in such a way that 
we can easily add in another node/brick once we're comfortable with the 
stability.

Thanks,

Erik Osterman

Brent A Nelson wrote:
> This probably has little to nothing to do with your performance, but I 
> don't thing cluster/unify is really doing anything in your present 
> configuration; you would need more subvolumes for it to be useful.  I 
> assume you have it in there for when you do add more nodes?
>
> Thanks,
>
> Brent
>
> On Mon, 12 Mar 2007, T0aD wrote:
>
>> Hello,
>>
>> By looking at your configuration, my first guess is that you have 
>> latency
>> issues in your network, which would definitely explain that awful
>> performances. So using your configuration files, I've set up the same
>> architecture, but locally (one computer running on Sempron 2600+ with 2
>> hitachi SATA1 disks in RAID1 software) and here are the results I got:
>>
>> toad at web1:~/afr$ time for i in {1..100}; do touch ./mnt/test.$i; done
>>
>> real    0m0.560s
>> user    0m0.160s
>> sys     0m0.190s
>> toad at web1:~/afr$ rm -f ./mnt/*
>> toad at web1:~/afr$ time for i in {1..10000}; do touch ./mnt/test.$i; done
>>
>> real    1m8.060s
>> user    0m16.680s
>> sys     0m18.180s
>> toad at web1:~/afr$ find ./mnt/ -type f | xargs -n100 rm -f
>> toad at web1:~/afr$ time for i in {1..1000}; do touch ./mnt/test.$i; done
>>
>> real    0m5.829s
>> user    0m1.670s
>> sys     0m1.910s
>>
>>
>> So my advise would be: check your network :)
>>
>> Hope it helped,
>>
>> Have a nice day everyone
>>
>> Julien Perez
>>
>> On 3/12/07, Erik Osterman <e at osterman.com> wrote:
>>>
>>> I've configured a cluster with replication that uses most of the
>>> advanced features you've implemented including io-threads, afr,
>>> readahead, and writebehind. I am very satisfied with the write
>>> performance, but the file creation performance leaves much to be
>>> desired. What can we do to speed this up?
>>>
>>> Creating 100 empty files
>>>
>>> # time for i in {1..100}; do touch test.$i;done
>>> real    0m46.913s
>>> user    0m0.023s
>>> sys     0m0.067s
>>>
>>> That's about 0.500 seconds just to create an empty file.
>>>
>>>
>>> In general, what do you advise for tuning the performance of
>>> reading/writing tons of tiny files. Can the client use io-threads to
>>> improve performance? Right now, our application stuffs all the tiny
>>> files in a single directory. Eventually, we were planning on hashing
>>> them out to directories. Would hashing them out into multiple
>>> directories positively and significantly affect the performance of
>>> GlusterFS?
>>>
>>>
>>>
>>> Best,
>>>
>>> Erik Osterman
>>>
>>>
>>>
>>>
>>>
>>> For what it's worth, here are my configurations:
>>>
>>> #
>>> # Master
>>> #
>>> volume posix0
>>>   type storage/posix                   # POSIX FS translator
>>>   option directory /home/glusterfs        # Export this directory
>>> end-volume
>>>
>>> volume brick0
>>>   type performance/io-threads
>>>   option thread-count 8
>>>   option queue-limit 1024
>>>   subvolumes posix0
>>> end-volume
>>>
>>> ### Add network serving capability to above brick.
>>> volume server
>>>   type protocol/server
>>>   option transport-type tcp/server     # For TCP/IP transport
>>> # option bind-address 192.168.1.10     # Default is to listen on all
>>> interfaces
>>>   option listen-port 6996               # Default is 6996
>>>   option client-volume-filename /etc/glusterfs/client.vol
>>>   subvolumes brick0
>>>   option auth.ip.brick0.allow *         # access to "brick" volume
>>> end-volume
>>>
>>>
>>>
>>> #
>>> # Mirror
>>> #
>>> volume posix0
>>>   type storage/posix                   # POSIX FS translator
>>>   option directory /home/glusterfs     # Export this directory
>>> end-volume
>>>
>>> volume mirror0
>>>   type performance/io-threads
>>>   option thread-count 8
>>>   option queue-limit 1024
>>>   subvolumes posix0
>>> end-volume
>>>
>>> ### Add network serving capability to above brick.
>>> volume server
>>>   type protocol/server
>>>   option transport-type tcp/server     # For TCP/IP transport
>>> # option bind-address 192.168.1.11     # Default is to listen on all
>>> interfaces
>>>   option listen-port 6996               # Default is 6996
>>>   option client-volume-filename /etc/glusterfs/client.vol
>>>   subvolumes mirror0
>>>   option auth.ip.mirror0.allow *         # access to "brick" volume
>>> end-volume
>>>
>>>
>>> #
>>> # Client
>>> #
>>>
>>> ### Add client feature and attach to remote subvolume of server
>>> volume brick0
>>>   type protocol/client
>>>   option transport-type tcp/client     # for TCP/IP transport
>>>   option remote-host 216.182.237.155   # IP address of the remote brick
>>> server
>>>   option remote-port 6996              # default server port is 6996
>>>   option remote-subvolume brick0        # name of the remote volume
>>> end-volume
>>>
>>> ### Add client feature and attach to remote mirror of brick0
>>> volume mirror0
>>>   type protocol/client
>>>   option transport-type tcp/client     # for TCP/IP transport
>>>   option remote-host 216.55.170.26      # IP address of the remote
>>> mirror server
>>>   option remote-port 6996              # default server port is 6996
>>>   option remote-subvolume mirror0        # name of the remote volume
>>> end-volume
>>>
>>> ### Add AFR feature to brick
>>> volume afr0
>>>   type cluster/afr
>>>   subvolumes brick0 mirror0
>>>   option replicate *:2                 # All files 2 copies (RAID-1)
>>> end-volume
>>>
>>> ### Add unify feature to cluster the servers. Associate an
>>> ### appropriate scheduler that matches your I/O demand.
>>> volume bricks
>>>   type cluster/unify
>>>   subvolumes afr0
>>> ### ** Round Robin (RR) Scheduler **
>>>   option scheduler rr
>>>   option rr.limits.min-free-disk 2GB
>>> end-volume
>>>
>>> ### Add performance feature
>>> volume writebehind
>>>   type performance/write-behind
>>>   option aggregate-size 131072 # aggregate block size in bytes
>>>   subvolumes bricks
>>> end-volume
>>>
>>> ### Add performance feature
>>> volume readahead
>>>   type performance/read-ahead
>>>   option page-size 131072
>>>   option page-count 16
>>>   subvolumes writebehind
>>> end-volume
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at nongnu.org
>>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at nongnu.org
>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>