[Gluster-devel] clustered afr

[Brent A Nelson]

Could you try 10GB files? My read leaks with read-ahead were triggered 
part-way through reading files of this size.

I, too, was seeing slow increases in glusterfs; you might see if it also 
happens without stat-prefetch, as I believe my tests did.

Thanks,

Brent

On Tue, 13 Mar 2007, Tibor Veres wrote:

>> Scheduler is an option of unify translator. As of now you are using
>> plain AFR across three bricks with 2 copies. To achieve what you want,
>> you can try like this. Create 3 AFR volumes across 3 bricks in pairs
>> of two like this: 1-2, 2-3 and 3-1. Then unify all the three volumes
>> using rr scheduler.
>
> i set up the test enviroment like this, and it works fine, though
> setting it up for more nodes/replicas would require writing a script
> to generate my config files, but its doable :)
>
> anyway, i checked out the latest source (patch-75), recompiled with
> -O3 and enabled all the performance translators
> iothreads in the server config, after the brick, before the
> protocol/server volume
> writeback/readahead/statprefetch in the client config, just like in
> client.vol.sample
>
> i did some tests to check for memory leaks
> didn't see any memory leak at write, nor at read
>
> but statprefetch is leaking memory:
> gluster-cl2:/mnt/gluster/test/lib# for j in `seq 1 4`; do ps uax |grep
> -v grep |grep glus ; for i in `seq 1 100`; do ls -la >/dev/null; done;
> done
> root     29575  0.6  0.1   1916   988 ?        Ss   11:53   0:28 [glusterfsd]
> root     29597  2.2  2.7  15396 14324 ?        Ss   11:54   1:40 [glusterfs]
>
> root     29575  0.6  0.1   1916   988 ?        Ss   11:53   0:30 [glusterfsd]
> root     29597  2.3  3.2  18196 17108 ?        Ss   11:54   1:45 [glusterfs]
>
> root     29575  0.7  0.1   1916   988 ?        Ss   11:53   0:32 [glusterfsd]
> root     29597  2.4  3.8  21236 20104 ?        Ss   11:54   1:51 [glusterfs]
>
> root     29575  0.7  0.1   1916   988 ?        Ss   11:53   0:34 [glusterfsd]
> root     29597  2.5  4.4  24292 23168 ?        Ss   11:54   1:56 [glusterfs]
>
> write/read tests and performance
> no write memory leak:
> root     29575  0.6  0.1   1916   988 ?        Ss   11:53   0:17 [glusterfsd]
> root     29597  1.7  2.2  12772 11716 ?        Ss   11:54   0:47 [glusterfs]
> gluster-cl2:/mnt/gluster/test# for i in `seq 1 10`; do dd if=/dev/zero
> of=test_$i bs=10000 count=10000; done
> 100000000 bytes (100 MB) copied, 2.93355 seconds, 34.1 MB/s
> 100000000 bytes (100 MB) copied, 3.47932 seconds, 28.7 MB/s
> 100000000 bytes (100 MB) copied, 3.38781 seconds, 29.5 MB/s
> 100000000 bytes (100 MB) copied, 7.33118 seconds, 13.6 MB/s
> 100000000 bytes (100 MB) copied, 3.45481 seconds, 28.9 MB/s
> 100000000 bytes (100 MB) copied, 9.05226 seconds, 11.0 MB/s
> 100000000 bytes (100 MB) copied, 9.37289 seconds, 10.7 MB/s
> 100000000 bytes (100 MB) copied, 3.17613 seconds, 31.5 MB/s
> 100000000 bytes (100 MB) copied, 6.28839 seconds, 15.9 MB/s
> 100000000 bytes (100 MB) copied, 6.46795 seconds, 15.5 MB/s
> gluster-cl2:/mnt/gluster/test# ps uax |grep glus
> root     29575  0.6  0.1   1916   988 ?        Ss   11:53   0:20 [glusterfsd]
> root     29597  2.4  2.2  12772 11716 ?        Ss   11:54   1:09 [glusterfs]
>
> no read memory leak:
> gluster-cl2:/mnt/gluster/test# for i in `seq 1 10`; do cat test_$i
>> /dev/null; done
> root     29575  0.6  0.1   1916   988 ?        Ss   11:53   0:24 [glusterfsd]
> root     29597  2.2  2.3  13100 12060 ?        Ss   11:54   1:27 [glusterfs]
> gluster-cl2:/mnt/gluster/test# for i in `seq 1 10`; do cat test_$i
>> /dev/null; done
> root     29575  0.6  0.1   1916   988 ?        Ss   11:53   0:24 [glusterfsd]
> root     29597  2.2  2.3  13100 12060 ?        Ss   11:54   1:28 [glusterfs]
> gluster-cl2:/mnt/gluster/test# for i in `seq 1 10`; do cat test_$i
>> /dev/null; done
> root     29575  0.6  0.1   1916   988 ?        Ss   11:53   0:25 [glusterfsd]
> root     29597  2.2  2.3  13164 12124 ?        Ss   11:54   1:30 [glusterfs]
> gluster-cl2:/mnt/gluster/test# for i in `seq 1 10`; do cat test_$i
>> /dev/null; done
> root     29575  0.6  0.1   1916   988 ?        Ss   11:53   0:25 [glusterfsd]
> root     29597  2.3  2.3  13100 12060 ?        Ss   11:54   1:31 [glusterfs]
>
> read performance:
> gluster-cl2:/mnt/gluster/test# for i in `seq 1 10`; do time cat
> test_$i >/dev/null; done
> real    0m2.001s
> real    0m2.660s
> real    0m2.344s
> real    0m1.979s
> real    0m2.314s
> real    0m2.627s
> real    0m2.635s
> real    0m2.815s
> real    0m2.907s
> real    0m2.088s
>
>
>
> -- 
> Tibor Veres
> tibor.veres at gmail.com
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>