Fwd: [Gluster-devel] Performance

[Anand Avati]

Copying gluster-devel@

From: Brian Taber <btaber at diversecg.com>
Date: Oct 15, 2007 3:59 AM
Subject: Re: [Gluster-devel] Performance
To: Anand Avati <avati at zresearch.com>

Now there's a beautiful thing....

NFS Write:
time dd if=/dev/zero bs=65536 count=15625 of=/shared/1Gb.file
1024000000 bytes (1.0 GB) copied, 62.9818 seconds, 16.3 MB/s

Gluster Write:
time dd if=/dev/zero bs=65536 count=15625 of=/mnt/glusterfs/1Gb.file
1024000000 bytes (1.0 GB) copied, 41.74 seconds, 24.5 MB/s

NFS Read:
time dd if=/shared/1Gb.file bs=65536 count=15625 of=/dev/zero
1024000000 bytes (1.0 GB) copied, 44.4734 seconds, 23.0 MB/s

Gluster Read:
time dd if=/mnt/glusterfs/1Gb.file bs=65536 count=15625 of=/dev/zero
1024000000 bytes (1.0 GB) copied, 42.1526 seconds, 24.3 MB/s


this test is performed within a VMWare virtual machine, so network speed
isn't as good.   I tried it from outsite, 1000MB network:



NFS Write:
time dd if=/dev/zero bs=65536 count=15625 of=/shared/1Gb.file
1024000000 bytes (1.0 GB) copied, 27.619 seconds, 37.1 MB/s

Gluster Write:
time dd if=/dev/zero bs=65536 count=15625 of=/mnt/glusterfs/1Gb.file
1024000000 bytes (1.0 GB) copied, 11.1978 seconds, 91.4 MB/s

NFS Read:
time dd if=/shared/1Gb.file bs=65536 count=15625 of=/dev/zero
1024000000 bytes (1.0 GB) copied, 43.5323 seconds, 23.5 MB/s

Gluster Read:
time dd if=/mnt/glusterfs/1Gb.file bs=65536 count=15625 of=/dev/zero
1024000000 bytes (1.0 GB) copied, 30.6922 seconds, 33.4 MB/s



> Brian,
>  block size of 1kb is too small and expensive, especially for network or
> fuse based filesystems. Please try with a larger block size like 64kb.
>
> avati
>
> On 10/14/07, Brian Taber <btaber at diversecg.com> wrote:
>>
>> I am new to ClusterFS and I am looking for a replacement for my NFS
>> setup.
>> I have configured a clusterfs server on top of a raid 5 array on 4 SATA
>> hard drives.  Directly I can get a speed of 72.3 MB/s weh I do a:
>>
>> dd if=/dev/zero bs=1024 count=1000000 of=/data/1Gb.file
>>
>> If I do the same test over NFSv3 I get performance of 14.1 MB/s
>>
>> If I do the same test over the gluster mount, I get perfromance of
>> 3.6MB/s
>>
>> Am I doing something wrong here?  How can I increase my performance to
>> the
>> same or beyond my current NFS?
>>
>> I setup the server with this config:
>>
>> volume brick-ns
>>   type storage/posix
>>   option directory /gluster-ns
>> end-volume
>>
>> volume brick
>>   type storage/posix
>>   option directory /data/gluster
>> end-volume
>>
>> volume iothreads1    #iothreads can give performance a boost
>>    type performance/io-threads
>>    option thread-count 8
>>    subvolumes brick
>> end-volume
>>
>> volume server
>>   type protocol/server
>>   subvolumes iothreads1 brick-ns
>>   option transport-type tcp/server     # For TCP/IP transport
>>   option auth.ip.iothreads1.allow 192.168.*
>>   option auth.ip.brick-ns.allow 192.168.*
>> end-volume
>>
>>
>> and setup a client with:
>>
>> volume client1-ns
>>   type protocol/client
>>   option transport-type tcp/client
>>   option remote-host 192.168.200.201
>>   option remote-subvolume brick-ns
>> end-volume
>>
>> volume client1
>>   type protocol/client
>>   option transport-type tcp/client
>>   option remote-host 192.168.200.201
>>   option remote-subvolume iothreads1
>> end-volume
>>
>> volume bricks
>>   type cluster/unify
>>   subvolumes client1
>>   option namespace client1-ns
>>   option scheduler alu
>>   option alu.limits.min-free-disk  60GB              # Stop creating
>> files
>> when free-space lt 60GB
>>   option alu.limits.max-open-files 10000
>>   option alu.order
>> disk-usage:read-usage:write-usage:open-files-usage:disk-speed-usage
>>   option alu.disk-usage.entry-threshold 2GB          # Units in KB, MB
>> and
>> GB are allowed
>>   option alu.disk-usage.exit-threshold  60MB         # Units in KB, MB
>> and
>> GB are allowed
>>   option alu.open-files-usage.entry-threshold 1024
>>   option alu.open-files-usage.exit-threshold 32
>>   option alu.stat-refresh.interval 10sec
>> end-volume
>>
>> volume writebehind   #writebehind improves write performance a lot
>>   type performance/write-behind
>>   option aggregate-size 131072 # in bytes
>>   subvolumes bricks
>> end-volume
>>
>>
>> When trying the
>>
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at nongnu.org
>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
>
>
>
> --
> It always takes longer than you expect, even when you take into account
> Hofstadter's Law.
>
> -- Hofstadter's Law
>




-- 
It always takes longer than you expect, even when you take into account
Hofstadter's Law.

-- Hofstadter's Law