[Gluster-users] GlusterFS performance

[Nikita A Kardashin]

Hello!

I have GlusterFS installation with parameters:

- 4 servers, connected by 1Gbit/s network (760-800 Mbit/s by iperf)
- Distributed-replicated volume with 4 bricks and 2x4 redundancy formula.
- Replicated volume with 2 bricks and 2x2 formula.

I found some trouble: if I try to copy huge amount of files (94000 files,
3Gb size), this process takes terribly long time (from 20 to 40 minutes). I
perform some tests and results is:

Directly to storage (single 2TB HDD): 158MB/s
Directly to storage (RAID1 of 2 HDDs): 190MB/s
To Replicated gluster volume: 89MB/s
To Distributed-replicated gluster volume: 49MB/s

Test command is: sync && echo 3 > /proc/sys/vm/drop_caches && dd
if=/dev/zero of=gluster.test.bin bs=1G count=1

Switching direct-io on and off doesn't have effect.
Playing with glusterfs options too.

What I can do with performance?

My volumes:

Volume Name: nginx
Type: Replicate
Volume ID: e3306431-e01d-41f8-8b2d-86a61837b0b2
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: control1:/storage/nginx
Brick2: control2:/storage/nginx

Volume Name: instances
Type: Distributed-Replicate
Volume ID: d32363fc-4b53-433c-87b7-ad51acfa4125
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: control1:/storage/instances
Brick2: control2:/storage/instances
Brick3: compute1:/storage/instances
Brick4: compute2:/storage/instances
Options Reconfigured:
cluster.self-heal-window-size: 1
cluster.data-self-heal-algorithm: diff
performance.stat-prefetch: 1
features.quota-timeout: 3600
performance.write-behind-window-size: 512MB
performance.cache-size: 1GB
performance.io-thread-count: 64
performance.flush-behind: on
performance.cache-min-file-size: 0
performance.write-behind: on

Mounted with default options by Gluster-FUSE.


-- 
With best regards,
differentlocal (www.differentlocal.ru | differentlocal at gmail.com),
System administrator.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130227/7ee8d468/attachment.html>