[Gluster-users] Slow speed with Gluster Backed Xen DomU's

Sheng Yeo shengyeo at gmail.com
Mon Jul 19 22:38:34 UTC 2010

Hi Everyone,

Hope your day has been going well. I tried emailing this in a week or  
so ago, but i do not think that it got emailed to the group. I am just  
trying again, but apologies in advanced if you have already read this.  
I am currently using GlusterFS as a distributed SAN backend for the  
Xen based cloud platform we are developing.

We deploy Xen virtuals on pairs of servers using GlusterFs V3 in  
replicate mode on Debian Stable (Lenny) with Xen 3.2.1 as the  
hypervisor. I am currently experiencing a weird issue where within the
virtual machines (DomU) running on a GlusterFS mount only receive  
around 10-18MB/s write speeds, but full speed reads.

Our hardware for each node is Dual Core Xeon Processors, 8GB of RAM  
and 4 * High Speed SATA drives (RAID 10, around 160MB/s writes and  

If I write a file to the Gluster mount in the Dom0 (host) we receive  
around 90-100MB/s writes (maxing out the GigE link). If I run the  
virtual machine on the disks without Gluster I get much higher speeds
within the DomU of around 80-90MB/s.

This slow down only appears to occur on writes. Does anyone with a  
better understanding of GlusterFS, Fuse and filesystems have an idea  
why this is slowing down. The underlying file system is Ext3 using  
TAP:AIO within Xen to connect to a file image based disk. This is  
without using gluster fuse client (what benefits does this give?) and  
Gluster version 3.0.4.

Many Thanks

Here is the current configuration of the servers in replicate:


volume posix
   type storage/posix
   option directory /export

volume locks
   type features/locks
   subvolumes posix

volume brick
   type performance/io-threads
   option thread-count 8
   subvolumes locks

volume server
   type protocol/server
   option transport-type tcp
   option auth.addr.brick.allow 10.*.*.*
   subvolumes brick


volume remote1
   type protocol/client
   option transport-type tcp
   option remote-host node01
   option remote-subvolume brick

volume remote2
   type protocol/client
   option transport-type tcp
   option remote-host node02
   option remote-subvolume brick

volume replicate1
   type cluster/replicate
   subvolumes remote1 remote2

volume writebehind
   type performance/write-behind
   option window-size 1MB
   subvolumes replicate1

volume cache
   type performance/io-cache
   option cache-size 512MB
   subvolumes writebehind

More information about the Gluster-users mailing list