[Gluster-users] Slow speed with Gluster Backed Xen DomU's

[Sheng Yeo]

Hi Everyone,

Hope your day has been going well. I tried emailing this in a week or  
so ago, but i do not think that it got emailed to the group. I am just  
trying again, but apologies in advanced if you have already read this.  
I am currently using GlusterFS as a distributed SAN backend for the  
Xen based cloud platform we are developing.

We deploy Xen virtuals on pairs of servers using GlusterFs V3 in  
replicate mode on Debian Stable (Lenny) with Xen 3.2.1 as the  
hypervisor. I am currently experiencing a weird issue where within the
virtual machines (DomU) running on a GlusterFS mount only receive  
around 10-18MB/s write speeds, but full speed reads.

Our hardware for each node is Dual Core Xeon Processors, 8GB of RAM  
and 4 * High Speed SATA drives (RAID 10, around 160MB/s writes and  
reads).

If I write a file to the Gluster mount in the Dom0 (host) we receive  
around 90-100MB/s writes (maxing out the GigE link). If I run the  
virtual machine on the disks without Gluster I get much higher speeds
within the DomU of around 80-90MB/s.

This slow down only appears to occur on writes. Does anyone with a  
better understanding of GlusterFS, Fuse and filesystems have an idea  
why this is slowing down. The underlying file system is Ext3 using  
TAP:AIO within Xen to connect to a file image based disk. This is  
without using gluster fuse client (what benefits does this give?) and  
Gluster version 3.0.4.

Many Thanks
Sheng

Here is the current configuration of the servers in replicate:

Server:

volume posix
   type storage/posix
   option directory /export
end-volume

volume locks
   type features/locks
   subvolumes posix
end-volume

volume brick
   type performance/io-threads
   option thread-count 8
   subvolumes locks
end-volume

volume server
   type protocol/server
   option transport-type tcp
   option auth.addr.brick.allow 10.*.*.*
   subvolumes brick
end-volume

Client:

volume remote1
   type protocol/client
   option transport-type tcp
   option remote-host node01
   option remote-subvolume brick
end-volume

volume remote2
   type protocol/client
   option transport-type tcp
   option remote-host node02
   option remote-subvolume brick
end-volume

volume replicate1
   type cluster/replicate
   subvolumes remote1 remote2
end-volume


volume writebehind
   type performance/write-behind
   option window-size 1MB
   subvolumes replicate1
end-volume

volume cache
   type performance/io-cache
   option cache-size 512MB
   subvolumes writebehind
end-volume