[Gluster-users] Performance in VM guests when hosting VM images on Gluster

[Brian Foster]

On 02/27/2013 10:14 AM, Torbjørn Thorsen wrote:
> I'm seeing less-than-stellar performance on my Gluster deployment when
> hosting VM images on the FUSE mount.
> I've seen that this topic has surfaced before, but my googling and
> perusing of the list archive haven't turned out very conclusive.
> 
> I'm on a 2-node distribute+replicate cluster, the clients use Gluster
> via the FUSE mount.
> torbjorn at storage01:~$ sudo gluster volume info
> 
> Volume Name: gluster0
> Type: Distributed-Replicate
> Volume ID: 81bbf681-ecdb-4866-9b45-41d5d2df7b35
> Status: Started
> Number of Bricks: 2 x 2 = 4
> Transport-type: tcp
> Bricks:
> Brick1: storage01.gluster.trollweb.net:/srv/gluster/brick0
> Brick2: storage02.gluster.trollweb.net:/srv/gluster/brick0
> Brick3: storage01.gluster.trollweb.net:/srv/gluster/brick1
> Brick4: storage02.gluster.trollweb.net:/srv/gluster/brick1
> 
> The naive dd case from one client, on the dom0, looks like this:
> torbjorn at xen01:/srv/ganeti/shared-file-storage/tmp$ sudo dd
> if=/dev/zero of=bigfile bs=1024k count=2000
> 2097152000 bytes (2.1 GB) copied, 22.9161 s, 91.5 MB/s
> 
> The clients see each node on a separate 1Gbps NIC, so this is pretty
> close to the expected transfer rate.
> 
> Writing with the sync flag, from dom0, looks like so:
> torbjorn at xen01:/srv/ganeti/shared-file-storage/tmp$ sudo dd
> if=/dev/zero of=bigfile bs=1024k count=2000 oflag=sync
> 2097152000 bytes (2.1 GB) copied, 51.1271 s, 41.0 MB/s
> 
> If we use a file on the gluster mount as backing for a loop device,
> and do a sync write:
> torbjorn at xen01:/srv/ganeti/shared-file-storage/tmp$ sudo dd
> if=/dev/zero of=/dev/loop1 bs=1024k count=2000 oflag=sync
> 2097152000 bytes (2.1 GB) copied, 56.3729 s, 37.2 MB/s
> 

What you might want to try is compare each case with gluster profiling
enabled on your volume (e.g., run a 'gluster ... profile info' to clear
the interval stats, run your test, run another 'profile info' and see
how many write requests occurred, divide the amount of data transferred
by the number of requests).

Running similar tests on a couple random servers around here brings me
from 70-80MB/s down to 10MB/s over loop. The profile data clearly shows
that loop is breaking what were previously 128k (max) write requests
into 4k requests. I don't know enough about the block layer to say why
that occurs, but I'd be suspicious of the combination of the block
interface on top of a filesystem (fuse) with synchronous request
submission (no caching, writes are immediately submitted to the client
fs). That said, I'm on an older kernel (or an older loop driver anyways,
I think) and your throughput above doesn't seem to be much worse with
loop alone...

Brian

> The Xen instances are managed by Ganeti, using the loopback interface
> over a file on Gluster.
> Inside the Xen instance the performance is not quite what I was hoping.
> torbjorn at hennec:~$ sudo dd if=/dev/zero of=bigfile bs=1024k count=2000
> 2097152000 bytes (2.1 GB) copied, 1267.39 s, 1.7 MB/s
> 
> The transfer rate is similar when using sync or direct flags with dd.
> 
> Are these expected performance levels ?
> A couple of threads[1] talk about performance, and seem to indicate my
> situation isn't unique.
> However, I'm under the impression that other are using a similar setup
> with much better performance.
> 
> 
> [1]:
> * http://www.gluster.org/pipermail/gluster-users/2012-January/032369.html
> * http://www.gluster.org/pipermail/gluster-users/2012-July/033763.html
> 
> --
> Vennlig hilsen
> Torbjørn Thorsen
> Utvikler / driftstekniker
> 
> Trollweb Solutions AS
> - Professional Magento Partner
> www.trollweb.no
> 
> Telefon dagtid: +47 51215300
> Telefon kveld/helg: For kunder med Serviceavtale
> 
> Besøksadresse: Luramyrveien 40, 4313 Sandnes
> Postadresse: Maurholen 57, 4316 Sandnes
> 
> Husk at alle våre standard-vilkår alltid er gjeldende
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>