[Gluster-users] GlusterFS as virtual machine storage

[Pavel Szalbot]

Sorry, I did not start the glusterfsd on the node I was shutting
yesterday and now killed another one during FUSE test, so it had to
crash immediately (only one of three nodes were actually up). This
definitely happened for the first time (only one node had been killed
yesterday).

Using FUSE seems to be OK with replica 3. So this can be gfapi related
or maybe rather libvirt related.

I tried ioengine=gfapi with fio and job survived reboot.


-ps


On Sat, Sep 9, 2017 at 8:05 AM, Pavel Szalbot <pavel.szalbot at gmail.com> wrote:
> Hi,
>
> On Sat, Sep 9, 2017 at 2:35 AM, WK <wkmail at bneit.com> wrote:
>> Pavel.
>>
>> Is there a difference between native client (fuse) and libgfapi in regards
>> to the crashing/read-only behaviour?
>
> I switched to FUSE now and the VM crashed (read-only remount)
> immediately after one node started rebooting.
>
> I tried to mount.glusterfs same volume on different server (not VM),
> running Ubuntu Xenial and gluster client 3.10.5.
>
> mount -t glusterfs -o backupvolfile-server=10.0.1.202
> 10.0.1.201:/gv_openstack_1 /mnt/gv_openstack_1/
>
> I ran fio job I described earlier. As soon as I killall glusterfsd,
> fio reported:
>
> fio: io_u error on file /mnt/gv_openstack_1/fio.data: Transport
> endpoint is not connected: read offset=7022575616, buflen=262144
> fio: pid=7205, err=107/file:io_u.c:1582, func=io_u error,
> error=Transport endpoint is not connected
>
> And crashed. I still cannot believe I am the only one experiencing
> these problems and that tells me, that there must be some problem in
> my setup. However I have not experienced any crashes if all nodes were
> up. Ever. I suspected disks and network as culprit, but we run SMART
> tests frequently (short and long), bricks are on RAID10 (6xSSDs),
> switches are Juniper EX4550s (shallow packet buffer, but no drops in
> the statistics) pretty much dedicated to Gluster and we ran many VMs
> and stored and heavily used other data there. And gluster logs or
> system logs do not provide any hint of HW/network failure.
>
>> We use Rep2 + Arb and can shutdown a node cleanly, without issue on our VMs.
>> We do it all the time for upgrades and maintenance.
>>
>> However we are still on native client as we haven't had time to work on
>> libgfapi yet. Maybe that is more tolerant.
>>
>> We have linux VMs mostly with XFS filesystems.
>
> We use whatever official cloud (Openstack) images provide, all tests I
> describe are on Ubuntu Xenial VMs, ext4.
>
>> During the downtime, the VMs continue to run with normal speed.
>>
>> In this case we migrated to the VM so date node 2 (c2g.gluster) and shutdown
>> c1g.gluster to do some upgrades.
>>
>> # gluster peer status
>> Number of Peers: 2
>>
>> Hostname: c1g.gluster
>> Uuid: 91be2005-30e6-462b-a66e-773913cacab6
>> State: Peer in Cluster (Disconnected)
>> Hostname: arb-c2.gluster
>> Uuid: 20862755-e54e-4b79-96a8-59e78c6a6a2e
>> State: Peer in Cluster (Connected)
>>
>> # gluster volume status
>> Status of volume: brick1
>> Gluster process                             TCP Port  RDMA Port Online  Pid
>> ------------------------------------------------------------------------------
>> Brick c2g.gluster:/GLUSTER/brick1       49152     0 Y       5194
>> Brick arb-c2.gluster:/GLUSTER/brick1        49152     0 Y       3647
>> Self-heal Daemon on localhost               N/A       N/A Y       5214
>> Self-heal Daemon on arb-c2.gluster          N/A       N/A Y       3667
>>
>> Task Status of Volume brick1
>> ------------------------------------------------------------------------------
>> There are no active volume tasks
>>
>> When we return the c1g node, we do see a "pause" in the VMs as the shards
>> heal. By pause meaning a terminal session gets spongy, but that passes
>> pretty quickly.
>
> Hmm, do you see any errors in VM's dmesg? Or any other reasons for "sponginess"?
>
>> Also are your VMs mounted in libvirt with caching? We always use
>> cache='none' so we can migrate around easily.
>
> No cache, virtio:
>
>     <disk type='network' device='disk'>
>       <driver name='qemu' type='raw' cache='none'/>
>       <source protocol='gluster'
> name='gv_openstack_1/volume-3a7eaf5a-8348-4f01-b59f-f28cd8cea771'>
>         <host name='10.0.1.201' port='24007'/>
>       </source>
>       <backingStore/>
>       <target dev='vda' bus='virtio'/>
>       <serial>3a7eaf5a-8348-4f01-b59f-f28cd8cea771</serial>
>       <alias name='virtio-disk0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
> function='0x0'/>
>     </disk>
>
>> Finally, you seem to be using oVirt/RHEV. Is it possible that your platform
>> is triggering a protective response on the VMs (by suspending).
>
> No, this is Openstack environment, I am not aware of any protective mechanisms.
>
> -ps