[Gluster-users] GlusterFS HA testing feedback

Tue Oct 22 20:13:27 UTC 2013

On 10/22/2013 02:42 AM, José A. Lausuch Sales wrote:
> Hi,
>
> we are currently evaluating GlusterFS for a production environment. 
> Our focus is on the high-availability features of GlusterFS. However, 
> our tests have not worked out well. Hence I am seeking feedback from you.
>
>
> In our planned production environment, Gluster should provide shared 
> storage for VM disk images. So, our very basic initial test setup is 
> as follows:
>
>
> We are using two servers, each providing a single brick of a 
> replicated gluster volume (Gluster 3.4.1). A third server runs a 
> test-VM (Ubuntu 13.04 on QEMU 1.3.0 and libvirt 1.0.3) which uses a 
> disk image file stored on the gluster volume as block device 
> (/dev/vdb). For testing purposes, the root file system of this VM 
> (/dev/vda) is a disk image NOT stored on the gluster volume.
>
>
> To test the high-availability features of gluster under load, we run 
> FIO inside the VM directly on the vdb block device (see configuration 
> below). Up to now, we tested reading only. The test procedure is as 
> follows:
>
> 1.We start FIO inside the VM and observe by means of "top" which of 
> the two servers receives the read requests (i.e., increased CPU load 
> of the glusterd process). Let's say that Server1 has the CPU load by 
> glusterfsd.
>
> 2.While FIO is running, we take down the network of this Server1 and 
> observe if the Server2 takes over.

You're bringing server1 down by taking down the NIC (assuming from #5). 
This does take down the connection but it does so without closing the 
TCP connection. Though this does represent worst-case scenarios, see 
http://joejulian.name/blog/keeping-your-vms-from-going-read-only-when-encountering-a-ping-timeout-in-glusterfs/

>
> 3.This "fail over" works (almost 100% of the times), we see the CPU 
> load from glusterfsd on Server2. As expected, Server1 does not have 
> any load because is "offline".
>
> 4.After a while we bring up the NIC on Server1 again. In this step we 
> realized that the expected behavior is that when bringing up this NIC, 
> this server should take over again (something like active-passive 
> behavior) but this happens only 5-10% of the times.  The CPU load is 
> still on Server2.
I'm not sure I would have that expectation. The second server will have 
taken over the open FD and the reads should come from there. The reads 
for a given fd come from the first-to-respond to the lookup().

>
> 5.After some time, we bring down the NIC on Server2 expecting that 
> Server1 takes over.  This second "fail over" crashes. The VM complains 
> about I/O errors which can only be resolved by restarting the VM and 
> sometimes even removing and creating the volume again.
>
>
> After some test, we realized that if restarting the glusterd daemon 
> (/etc/init.d/glusterd restart) on Server1 after step 3 or before step 
> 4, the Server1 takes over automatically without bringing down Server2 
> or anything like that.
Check the logs for glusterd 
(/var/log/glusterfs/etc-glusterfs-glusterd.vol.log) for clues. Perhaps 
the /way/ you're taking down the NIC is exposing some bug. Perhaps 
instead of taking it down, use iptables or just killall glusterfsd.
>
>
> We tested this using the normal FUSE mount and libgfapi. If using 
> FUSE, the local mount sometimes becomes unavailable (ls shows not more 
> files) if the failover fails.
>
>
> We have a few fundamental questions in this regard:
>
> i) Is Gluster supposed to handle such a scenario or are we making 
> wrong assumptions? Because the only solution we found is to restart 
> the daemon when a network outage occurs, but this is not acceptable in 
> a real scenario with VMs running real applications.
I host my (raw and qcow2) vm images on a gluster volume. Since my 
servers are not expected to hard-crash a lot, I take them down for 
maintenance (kernel updates and such) gracefully, killing the processes 
first. This closes the TCP connections and everything just keeps humming 
along.

>
> ii) What is the recommended configuration in terms of caching (QEMU: 
> cache=none/writethrough/writeback) and direct I/O (FIO and Gluster) to 
> maximize the reliability of the failover process? We varied the 
> parameters but could find a working configuration. Do these parameters 
> have an impact at all?
To the best of my knowledge, none of those should affect reliability.

>
>
>
>
> FIO test specification:
>
> [global]
> direct=1
> ioengine=libaio
> iodepth=4
> filename=/dev/vdb
> runtime=300
> numjobs=1
>
> [maxthroughput]
> rw=read
> bs=16k
>
>
>
> VM configuration:
>
> <domain type='kvm' id='6'>
>   <name>testvm</name>
> <uuid>93877c03-605b-ed67-1ab2-2ba16b5fb6b5</uuid>
>   <memory unit='KiB'>2097152</memory>
>   <currentMemory unit='KiB'>2097152</currentMemory>
>   <vcpu placement='static'>1</vcpu>
>   <os>
>     <type arch='x86_64' machine='pc-1.1'>hvm</type>
>     <boot dev='hd'/>
>   </os>
>   <features>
>     <acpi/>
>     <apic/>
>     <pae/>
>   </features>
>   <clock offset='utc'/>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>restart</on_crash>
>   <devices>
>     <emulator>/usr/bin/kvm</emulator>
>     <disk type='block' device='disk'>
>       <driver name='qemu' type='raw' cache='writethrough'/>
>       <source dev='/mnt/local/io-perf.img'/>
>       <target dev='vda' bus='virtio'/>
>       <alias name='virtio-disk0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x04' 
> function='0x0'/>
>     </disk>
>     <disk type='block' device='disk'>
>       <driver name='qemu' type='raw' cache='writethrough'/>
>       <source dev='/mnt/shared/io-perf-testdisk.img'/>
>       <target dev='vdb' bus='virtio'/>
>       <alias name='virtio-disk1'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x07' 
> function='0x0'/>
>     </disk>
>     <controller type='usb' index='0'>
>       <alias name='usb0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
> function='0x2'/>
>     </controller>
>     <interface type='network'>
>       <mac address='52:54:00:36:5f:dd'/>
>       <source network='default'/>
>       <target dev='vnet0'/>
>       <model type='virtio'/>
>       <alias name='net0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' 
> function='0x0'/>
>     </interface>
>     <input type='mouse' bus='ps2'/>
>     <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
>       <listen type='address' address='127.0.0.1'/>
>     </graphics>
>     <video>
>       <model type='cirrus' vram='9216' heads='1'/>
>       <alias name='video0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' 
> function='0x0'/>
>     </video>
>     <memballoon model='virtio'>
>       <alias name='balloon0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x05' 
> function='0x0'/>
>     </memballoon>
>   </devices>
>   <seclabel type='none'/>
> </domain>
>
>
>
>
> Thank you very much in advance,
> Jose Lausuch
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131022/1b8b7c4f/attachment.html>