[Gluster-users] VM fs becomes read only when one gluster node goes down
André Bauer
abauer at magix.net
Mon Oct 26 19:47:07 UTC 2015
Just some. But i think the reason is some vm images are replicated on
node 1 & 2 and some on node 3 & 4 because i use distributed/replicated
volume.
You're right. I think i have to try it on a testsetup.
At the moment i'm also no completly sure, if its a Glusterfs problem
(not connecting to the node with the replicated file immediately, when
read/write fails) or a problem of the filesystem (ext4 fs goes read only
on error to early)?
Regards
André
Am 26.10.2015 um 20:23 schrieb Josh Boon:
> Hmm even five should be OK. Do you lose all VMs or just some?
>
> Also, we had issues with
>
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
>
> and had to instead go with
>
> cluster.server-quorum-type: none
> cluster.quorum-type: none
>
> though we only replicate instead distribute and replicate so I'd be wary of changing those without advice from folks more familiar with the impact on your config.
>
> gfapi upon connect gets the volume file and is aware of the configuration and changes to it so it should be OK when a node is lost since it knows where the other nodes are.
>
> If you have a lab with your gluster config setup and you lose all of your VM's I'd suggest trying my config to see what happens. The gluster logs and qemu clients could also have some tips on what happens when a node disappears.
> ----- Original Message -----
> From: "André Bauer" <abauer at magix.net>
> To: "Josh Boon" <gluster at joshboon.com>
> Cc: "Krutika Dhananjay" <kdhananj at redhat.com>, "gluster-users" <gluster-users at gluster.org>, gluster-devel at gluster.org
> Sent: Monday, October 26, 2015 7:08:15 PM
> Subject: Re: [Gluster-users] VM fs becomes read only when one gluster node goes down
>
> Thanks guys!
> My volume info is attached at the bottom of this mail...
>
> @ Josh
> As you can see, i already have a 5 second ping timeout set. I will try
> it with 3 seconds.
>
> Not sure, if i want to have errors=continue on the fs level but i will
> give it a try, if its the only possibility to get automatic failover work.
>
>
> @ Roman
> I use qemu with libgfapi to access the images. So no glusterfs entries
> in fstab for my vm hosts. It also seems this is kind of deprecated:
>
> http://blog.gluster.org/category/mount-glusterfs/
>
> "`backupvolfile-server` - This option did not really do much rather than
> provide a 'shell' script based failover which was highly racy and
> wouldn't work during many occasions. It was necessary to remove this to
> make room for better options (while it is still provided for backward
> compatibility in the code)"
>
>
> @ all
> Can anybody tell me how Glusterfs handles this internaly?
> Is the libgfapi client already aware of the server which replicates the
> image?
> Is there a way i can configure it manualy for a volume?
>
>
>
>
> Volume Name: vmimages
> Type: Distributed-Replicate
> Volume ID: 029285b2-dfad-4569-8060-3827c0f1d856
> Status: Started
> Number of Bricks: 2 x 2 = 4
> Transport-type: tcp
> Bricks:
> Brick1: storage1.domain.local:/glusterfs/vmimages
> Brick2: storage2.domain.local:/glusterfs/vmimages
> Brick3: storage3.domain.local:/glusterfs/vmimages
> Brick4: storage4.domain.local:/glusterfs/vmimages
> Options Reconfigured:
> network.ping-timeout: 5
> performance.quick-read: off
> performance.read-ahead: off
> performance.io-cache: off
> performance.stat-prefetch: off
> cluster.eager-lock: enable
> network.remote-dio: enable
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> auth.allow:
> 192.168.0.21,192.168.0.22,192.168.0.23,192.168.0.24,192.168.0.25,192.168.0.26
> server.allow-insecure: on
> storage.owner-uid: 2000
> storage.owner-gid: 2000
>
>
>
> Regards
> André
>
>
> Am 26.10.2015 um 17:41 schrieb Josh Boon:
>> Andre,
>>
>> I've not explored using a DNS solution to publish the gluster cluster
>> addressing space but things you'll want to check out
>> are network.ping-timeout and whether or not your VM goes read-only on
>> filesystem error. If your network is consistent and robust
>> tuning network.ping-timeout to a very low value such as three seconds
>> will instruct the client to drop that client on failure. The default
>> value for this is 42 seconds which will cause your VM to go read-only as
>> you've seen. You could also choose to have your VM's mount their
>> partitions errors=continue as well depending on the filesystem they run.
>> Our setup has timeout at seven seconds and errors=continue and has
>> survived both testing and storage node segfaults. No data integrity
>> issues have presented yet but our data is mostly temporal so integrity
>> hasn't been tested thoroughly. Also we're qemu 2.0 running gluster 3.6
>> on ubuntu 14.04 for those curious.
>>
>> Best,
>> Josh
>>
>> ------------------------------------------------------------------------
>> *From: *"Roman" <romeo.r at gmail.com>
>> *To: *"Krutika Dhananjay" <kdhananj at redhat.com>
>> *Cc: *"gluster-users" <gluster-users at gluster.org>, gluster-devel at gluster.org
>> *Sent: *Monday, October 26, 2015 1:33:57 PM
>> *Subject: *Re: [Gluster-users] VM fs becomes read only when one gluster
>> node goes down
>>
>> Hi,
>> got backupvolfile-server=NODE2NAMEHERE in fstab ? :)
>>
>> 2015-10-23 5:24 GMT+03:00 Krutika Dhananjay <kdhananj at redhat.com
>> <mailto:kdhananj at redhat.com>>:
>>
>> Could you share the output of 'gluster volume info', and also
>> information as to which node went down on reboot?
>>
>> -Krutika
>> ------------------------------------------------------------------------
>>
>> *From: *"André Bauer" <abauer at magix.net <mailto:abauer at magix.net>>
>> *To: *"gluster-users" <gluster-users at gluster.org
>> <mailto:gluster-users at gluster.org>>
>> *Cc: *gluster-devel at gluster.org <mailto:gluster-devel at gluster.org>
>> *Sent: *Friday, October 23, 2015 12:15:04 AM
>> *Subject: *[Gluster-users] VM fs becomes read only when one
>> gluster node goes down
>>
>> Hi,
>>
>> i have a 4 node Glusterfs 3.5.6 Cluster.
>>
>> My VM images are in an replicated distributed volume which is
>> accessed
>> from kvm/qemu via libgfapi.
>>
>> Mount is against storage.domain.local which has IPs for all 4
>> Gluster
>> nodes set in DNS.
>>
>> When one of the Gluster nodes goes down (accidently reboot) a
>> lot of the
>> vms getting read only filesystem. Even when the node comes back up.
>>
>> How can i prevent this?
>> I expect that the vm just uses the replicated file on the other
>> node,
>> without getting ro fs.
>>
>> Any hints?
>>
>> Thanks in advance.
>>
>> --
>> Regards
>> André Bauer
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>>
>> --
>> Best regards,
>> Roman.
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
>
>
--
Mit freundlichen Grüßen
André Bauer
MAGIX Software GmbH
André Bauer
Administrator
August-Bebel-Straße 48
01219 Dresden
GERMANY
tel.: 0351 41884875
e-mail: abauer at magix.net
abauer at magix.net <mailto:Email>
www.magix.com <http://www.magix.com/>
Geschäftsführer | Managing Directors: Dr. Arnd Schröder, Klaus Schmidt
Amtsgericht | Commercial Register: Berlin Charlottenburg, HRB 127205
Find us on:
<http://www.facebook.com/MAGIX> <http://www.twitter.com/magix_de>
<http://www.youtube.com/wwwmagixcom> <http://www.magixmagazin.de>
----------------------------------------------------------------------
The information in this email is intended only for the addressee named
above. Access to this email by anyone else is unauthorized. If you are
not the intended recipient of this message any disclosure, copying,
distribution or any action taken in reliance on it is prohibited and
may be unlawful. MAGIX does not warrant that any attachments are free
from viruses or other defects and accepts no liability for any losses
resulting from infected email transmissions. Please note that any
views expressed in this email may be those of the originator and do
not necessarily represent the agenda of the company.
----------------------------------------------------------------------
More information about the Gluster-users
mailing list