[Gluster-users] Write performance in a replicated/distributed setup with KVM?

[Harald Hannelius]

This has probably been discussed before, but since I'm new on the list I hope 
You have patience with me.

I have a four brick distributed/replicated setup. The computers are multi-core 
16GB memory and 2*2.0TB in raid1 SATA-disks locally. The nodes are connected by 
1 GB ethernet. All nodes have glusterfs 3.3beta2 installed and they are running 
debian 6 64bit. The underlying filesystems are xfs.

I have setup a volume like so;

gluster volume create virtuals replica 2 transport tcp \
   adraste:/data/brick alcippe:/data/brick aethra:/data/brick helen:/data/brick

Which resulted in a nice volume;

# gluster volume info virtuals

Volume Name: virtuals
Type: Distributed-Replicate
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: adraste:/data/brick
Brick2: alcippe:/data/brick
Brick3: aethra:/data/brick
Brick4: helen:/data/brick

All seems OK so far, but write performance seems very slow. When writing to 
localhost:/virtuals I get single-digit MB/s performance which isn't really what 
I had expected. I know that the write has to go to at least two (?) nodes at 
the same time, but still?

A single scp of a 1GB file from a node to another gives something like 
~100MBps.

A copy of a virtual image took 17 minutes;

# time cp debtest.raw /gluster/debtest.img

real	17m36.727s
user	0m1.832s
sys	0m14.081s

# ls -lah /gluster/debtest.img
-rw------- 1 root root 20G Mar  1 12:35 /gluster/debtest.img
# du -ah /gluster/debtest.img
4.5G	/gluster/debtest.img

I noted that the processlist shows that direct-io-mode is disabled. Default 
should be on, or should it?

Any help is really appreciated!

-- 

Harald Hannelius | harald.hannelius/a\arcada.fi | +358 50 594 1020