[Gluster-users] Performance in VM guests when hosting VM images on Gluster

[Torbjørn Thorsen]

I'm seeing less-than-stellar performance on my Gluster deployment when
hosting VM images on the FUSE mount.
I've seen that this topic has surfaced before, but my googling and
perusing of the list archive haven't turned out very conclusive.

I'm on a 2-node distribute+replicate cluster, the clients use Gluster
via the FUSE mount.
torbjorn at storage01:~$ sudo gluster volume info

Volume Name: gluster0
Type: Distributed-Replicate
Volume ID: 81bbf681-ecdb-4866-9b45-41d5d2df7b35
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: storage01.gluster.trollweb.net:/srv/gluster/brick0
Brick2: storage02.gluster.trollweb.net:/srv/gluster/brick0
Brick3: storage01.gluster.trollweb.net:/srv/gluster/brick1
Brick4: storage02.gluster.trollweb.net:/srv/gluster/brick1

The naive dd case from one client, on the dom0, looks like this:
torbjorn at xen01:/srv/ganeti/shared-file-storage/tmp$ sudo dd
if=/dev/zero of=bigfile bs=1024k count=2000
2097152000 bytes (2.1 GB) copied, 22.9161 s, 91.5 MB/s

The clients see each node on a separate 1Gbps NIC, so this is pretty
close to the expected transfer rate.

Writing with the sync flag, from dom0, looks like so:
torbjorn at xen01:/srv/ganeti/shared-file-storage/tmp$ sudo dd
if=/dev/zero of=bigfile bs=1024k count=2000 oflag=sync
2097152000 bytes (2.1 GB) copied, 51.1271 s, 41.0 MB/s

If we use a file on the gluster mount as backing for a loop device,
and do a sync write:
torbjorn at xen01:/srv/ganeti/shared-file-storage/tmp$ sudo dd
if=/dev/zero of=/dev/loop1 bs=1024k count=2000 oflag=sync
2097152000 bytes (2.1 GB) copied, 56.3729 s, 37.2 MB/s

The Xen instances are managed by Ganeti, using the loopback interface
over a file on Gluster.
Inside the Xen instance the performance is not quite what I was hoping.
torbjorn at hennec:~$ sudo dd if=/dev/zero of=bigfile bs=1024k count=2000
2097152000 bytes (2.1 GB) copied, 1267.39 s, 1.7 MB/s

The transfer rate is similar when using sync or direct flags with dd.

Are these expected performance levels ?
A couple of threads[1] talk about performance, and seem to indicate my
situation isn't unique.
However, I'm under the impression that other are using a similar setup
with much better performance.


[1]:
* http://www.gluster.org/pipermail/gluster-users/2012-January/032369.html
* http://www.gluster.org/pipermail/gluster-users/2012-July/033763.html

--
Vennlig hilsen
Torbjørn Thorsen
Utvikler / driftstekniker

Trollweb Solutions AS
- Professional Magento Partner
www.trollweb.no

Telefon dagtid: +47 51215300
Telefon kveld/helg: For kunder med Serviceavtale

Besøksadresse: Luramyrveien 40, 4313 Sandnes
Postadresse: Maurholen 57, 4316 Sandnes

Husk at alle våre standard-vilkår alltid er gjeldende