[Gluster-devel] clustered afr
Brent A Nelson
brent at phys.ufl.edu
Tue Mar 13 16:00:06 UTC 2007
Could you try 10GB files? My read leaks with read-ahead were triggered
part-way through reading files of this size.
I, too, was seeing slow increases in glusterfs; you might see if it also
happens without stat-prefetch, as I believe my tests did.
Thanks,
Brent
On Tue, 13 Mar 2007, Tibor Veres wrote:
>> Scheduler is an option of unify translator. As of now you are using
>> plain AFR across three bricks with 2 copies. To achieve what you want,
>> you can try like this. Create 3 AFR volumes across 3 bricks in pairs
>> of two like this: 1-2, 2-3 and 3-1. Then unify all the three volumes
>> using rr scheduler.
>
> i set up the test enviroment like this, and it works fine, though
> setting it up for more nodes/replicas would require writing a script
> to generate my config files, but its doable :)
>
> anyway, i checked out the latest source (patch-75), recompiled with
> -O3 and enabled all the performance translators
> iothreads in the server config, after the brick, before the
> protocol/server volume
> writeback/readahead/statprefetch in the client config, just like in
> client.vol.sample
>
> i did some tests to check for memory leaks
> didn't see any memory leak at write, nor at read
>
> but statprefetch is leaking memory:
> gluster-cl2:/mnt/gluster/test/lib# for j in `seq 1 4`; do ps uax |grep
> -v grep |grep glus ; for i in `seq 1 100`; do ls -la >/dev/null; done;
> done
> root 29575 0.6 0.1 1916 988 ? Ss 11:53 0:28 [glusterfsd]
> root 29597 2.2 2.7 15396 14324 ? Ss 11:54 1:40 [glusterfs]
>
> root 29575 0.6 0.1 1916 988 ? Ss 11:53 0:30 [glusterfsd]
> root 29597 2.3 3.2 18196 17108 ? Ss 11:54 1:45 [glusterfs]
>
> root 29575 0.7 0.1 1916 988 ? Ss 11:53 0:32 [glusterfsd]
> root 29597 2.4 3.8 21236 20104 ? Ss 11:54 1:51 [glusterfs]
>
> root 29575 0.7 0.1 1916 988 ? Ss 11:53 0:34 [glusterfsd]
> root 29597 2.5 4.4 24292 23168 ? Ss 11:54 1:56 [glusterfs]
>
> write/read tests and performance
> no write memory leak:
> root 29575 0.6 0.1 1916 988 ? Ss 11:53 0:17 [glusterfsd]
> root 29597 1.7 2.2 12772 11716 ? Ss 11:54 0:47 [glusterfs]
> gluster-cl2:/mnt/gluster/test# for i in `seq 1 10`; do dd if=/dev/zero
> of=test_$i bs=10000 count=10000; done
> 100000000 bytes (100 MB) copied, 2.93355 seconds, 34.1 MB/s
> 100000000 bytes (100 MB) copied, 3.47932 seconds, 28.7 MB/s
> 100000000 bytes (100 MB) copied, 3.38781 seconds, 29.5 MB/s
> 100000000 bytes (100 MB) copied, 7.33118 seconds, 13.6 MB/s
> 100000000 bytes (100 MB) copied, 3.45481 seconds, 28.9 MB/s
> 100000000 bytes (100 MB) copied, 9.05226 seconds, 11.0 MB/s
> 100000000 bytes (100 MB) copied, 9.37289 seconds, 10.7 MB/s
> 100000000 bytes (100 MB) copied, 3.17613 seconds, 31.5 MB/s
> 100000000 bytes (100 MB) copied, 6.28839 seconds, 15.9 MB/s
> 100000000 bytes (100 MB) copied, 6.46795 seconds, 15.5 MB/s
> gluster-cl2:/mnt/gluster/test# ps uax |grep glus
> root 29575 0.6 0.1 1916 988 ? Ss 11:53 0:20 [glusterfsd]
> root 29597 2.4 2.2 12772 11716 ? Ss 11:54 1:09 [glusterfs]
>
> no read memory leak:
> gluster-cl2:/mnt/gluster/test# for i in `seq 1 10`; do cat test_$i
>> /dev/null; done
> root 29575 0.6 0.1 1916 988 ? Ss 11:53 0:24 [glusterfsd]
> root 29597 2.2 2.3 13100 12060 ? Ss 11:54 1:27 [glusterfs]
> gluster-cl2:/mnt/gluster/test# for i in `seq 1 10`; do cat test_$i
>> /dev/null; done
> root 29575 0.6 0.1 1916 988 ? Ss 11:53 0:24 [glusterfsd]
> root 29597 2.2 2.3 13100 12060 ? Ss 11:54 1:28 [glusterfs]
> gluster-cl2:/mnt/gluster/test# for i in `seq 1 10`; do cat test_$i
>> /dev/null; done
> root 29575 0.6 0.1 1916 988 ? Ss 11:53 0:25 [glusterfsd]
> root 29597 2.2 2.3 13164 12124 ? Ss 11:54 1:30 [glusterfs]
> gluster-cl2:/mnt/gluster/test# for i in `seq 1 10`; do cat test_$i
>> /dev/null; done
> root 29575 0.6 0.1 1916 988 ? Ss 11:53 0:25 [glusterfsd]
> root 29597 2.3 2.3 13100 12060 ? Ss 11:54 1:31 [glusterfs]
>
> read performance:
> gluster-cl2:/mnt/gluster/test# for i in `seq 1 10`; do time cat
> test_$i >/dev/null; done
> real 0m2.001s
> real 0m2.660s
> real 0m2.344s
> real 0m1.979s
> real 0m2.314s
> real 0m2.627s
> real 0m2.635s
> real 0m2.815s
> real 0m2.907s
> real 0m2.088s
>
>
>
> --
> Tibor Veres
> tibor.veres at gmail.com
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
More information about the Gluster-devel
mailing list