[Gluster-users] Strange file corruption

[Lindsay Mathieson]

Hi Udo, thanks for posting your volume info settings. Please note for 
the following, I am not one of the devs, just a user, so unfortunately I 
have no authoritative answers :(

I am running a very similar setup - Proxmox 4.0, three nodes, but using 
ceph for our production storage. Am heavily testing gluster 3.7 on the 
side. We find the performance of ceph slow on these small setups and 
management of it a PITA.


Some more questions

- how are your VM images being accessed by Proxmox? gfapi? (Proxmox 
Gluster storage type) or by using the fuse mount?

- whats your underlying filesystem (ext4, zfs etc)

- Are you using the HA/Watchdog system in Proxmox?



On 07/12/15 21:03, Udo Giacomozzi wrote:
> esterday I had a strange situation where Gluster healing corrupted 
> *all* my VM images.
>
>
> In detail:
> I had about 15 VMs running (in Proxmox 4.0) totaling about 600 GB of 
> qcow2 images. Gluster is used as storage for those images in replicate 
> 3 setup (ie. 3 physical servers replicating all data).
> All VMs were running on machine #1 - the two other machines (#2 and 
> #3) were *idle*.
> Gluster was fully operating (no healing) when I rebooted machine #2.
> For other reasons I had to reboot machines #2 and #3 a few times, but 
> since all VMs were running on machine #1 and nothing on the other 
> machines was accessing Gluster files, I was confident that this 
> wouldn't disturb Gluster.
> But anyway this means that I rebootet Gluster nodes during a healing 
> process.
>
> After a few minutes, Gluster files began showing corruption - up to 
> the point that the qcow2 files became unreadable and all VMs stopped 
> working. 


:( sounds painful - my sympathies.

You're running 3.5.2 - thats getting rather old. I use the gluster 
debian repos:

   3.6.7 : 
http://download.gluster.org/pub/gluster/glusterfs/3.6/LATEST/Debian/
   3.7.6 : 
http://download.gluster.org/pub/gluster/glusterfs/LATEST/Debian/jessie/

3.6.x is the latest stable, 3.7 is close to stable(?) 3.7 has some nice 
new features such as sharding, which is very useful for VM hosting - it 
enables much faster heal times.

Regards what happened with your VM's, I'm not sure. Having two servers 
down should have disabled the entire store making it not readable or 
writable. I note that you are missing some settings that need to be set 
for VM stores - there will be corruption problems if you live migrate 
without them.

quick-read=off
read-ahead=off
io-cache=off
stat-prefetch=off
eager-lock=enable
remote-dio=enable
quorum-type=auto
server-quorum-type=server


"stat-prefetch=off" is particularly important.






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20151208/38411c27/attachment.html>