[Gluster-users] Slow speed with Gluster Backed Xen DomU's

[Samuli Heinonen]

Hello Sheng,

Sorry for the delay with my answer. I got your message earlier but I was
hoping that someone else would answer with more precise information.

I did some testing almost a year ago to use GlusterFS to store Xen images.
I was testing with pre 2.10 version of GlusterFS but my findings were very
similar to yours. Writing speed was fine when done from Dom0 to GlusterFS
mount, I was even able to max out GigE link with my cheap hardware.
However when I did same tests on DomU (image inside the GlusterFS mount)
the write speeds dropped drastically. Read speeds were ok all the time.

Unfortunately I didn't have time to investigate it much further back then.
Although I did some performance tests with Bonnie++ on Dom0 and noticed
that block write speeds were almost the same as write speeds in general on
DomU. So, my guess is that block write speeds needs somekind of on
GlusterFS side?

-samuli

> Hope your day has been going well. I tried emailing this in a week or
> so ago, but i do not think that it got emailed to the group. I am just
> trying again, but apologies in advanced if you have already read this.
> I am currently using GlusterFS as a distributed SAN backend for the
> Xen based cloud platform we are developing.
>
> We deploy Xen virtuals on pairs of servers using GlusterFs V3 in
> replicate mode on Debian Stable (Lenny) with Xen 3.2.1 as the
> hypervisor. I am currently experiencing a weird issue where within the
> virtual machines (DomU) running on a GlusterFS mount only receive
> around 10-18MB/s write speeds, but full speed reads.
>
> Our hardware for each node is Dual Core Xeon Processors, 8GB of RAM
> and 4 * High Speed SATA drives (RAID 10, around 160MB/s writes and
> reads).
>
> If I write a file to the Gluster mount in the Dom0 (host) we receive
> around 90-100MB/s writes (maxing out the GigE link). If I run the
> virtual machine on the disks without Gluster I get much higher speeds
> within the DomU of around 80-90MB/s.
>
> This slow down only appears to occur on writes. Does anyone with a
> better understanding of GlusterFS, Fuse and filesystems have an idea
> why this is slowing down. The underlying file system is Ext3 using
> TAP:AIO within Xen to connect to a file image based disk. This is
> without using gluster fuse client (what benefits does this give?) and
> Gluster version 3.0.4.