[Gluster-users] GlusteFS VS DRBD + power failure ?

[Joe Julian]

On 10/27/2014 01:34 PM, Tytus Rogalewski wrote:
> Hi guys,
> I wanted to ask you about what happen in case of power failure.
> I have 2 node proxmox cluster with glusterfs as sdb1 XFS, and mounted 
> on each node as localhost/glusterstorage.
> I am storing VMs on it as qcow2(and inside ext4 filesystem).
>  Live migration works ok WOW.. Everything works fine.
> But tell me will something bad happen when the power will fail on 
> whole datacenter ?
> Will be data corrupted and will be the same thing if i am using drbd ?
> DRBD doesnt give me so much flexability(because i cant use qcow2 and 
> store files like iso or backups on drbd), but glusterfs does give me 
> much flexability !
> Anyway yesterday i created glusterfs with ext4, and VM qcow with ext4 
> on it and when i made "reboot -f"(i assume this is the same as i will 
> pull power cord off ?) - after node went online again, VM data was 
> corrupted and i had many ext failures inside VM.
> Tell me was that because i used ext4 on top of sdb1 glusterfs storage 
> or will that work the same with XFS ?
> Is drbd better  protection in case of power failure ?
My experience with DRBD is really old, but I became a gluster user 
because of my experience with drbd. After it destroyed my filesystem for 
the 3rd time, it was "replace that or find somewhere else to work" time.

I chose gluster because you can create a fullly redundant system from 
the client to each replica server, all the way through all the hardware 
by creating parallel network paths.

What you experienced is a result of the ping timeout. Ping-timeouts 
happen when the TCP connection is not closed, like when you pull the 
plug. The timeout exists to allow the filesystem to recover gracefully 
in the event of a temporary network problem. Without that, there's an 
increased load on the server while all the file descriptors are 
re-established. This can be a fairly heavy load, to the point where tcp 
pings are delayed. If they're delayed longer than ping-timeout, you have 
a race condition from which you'll never recover. For that reason, the 
ping-timeout is longer. You *can* adjust that timeout as long as you 
sufficiently test around the actual loads you're expecting.

Keep in mind your SLA/OLA expectations and engineer for them using the 
actual mathematical calculations, not just some gut expectations. Your 
DC power should be more reliable than most industries requirements.

>
> Anyway second question, if i have 2 nodes with glusterfs.
> node1 is changing file1.txt
> node2 is changing file2.txt
> then i will disconnect glusterfs in network, and data keeps changing 
> on both nodes)
> After i will reconnect glusterfs how this will go?
> Newer changed file1 from node1 will overwrite file1 on node2?
> and newer file2 changed on node2 will overwrite file2 on node1 ?
> Am i correct ?
>
> Thx for answer :)
>
Each client intends to write to both (all) replicas. The intent count is 
incremented in extended attributes, the write executes on a replica, the 
intent count is decremented for that replica. With the disconnect, each 
of those files will show pending changed destined for the other replica. 
When they are reconnected, the self-heal daemon (or a client attempting 
to access those files) will note the changes destined for the other 
brick and repair it.

Split-brain occurs when each side of that netsplit writes to the same 
file. That file indicates pending changes for the other brick. When the 
connection returns, they compare those pending flags and see changes to 
each that are unwritten on the other. They refuse and leave each file 
intact, forcing manual intervention to clear the split-brain.

You can avoid split-brain by using replica 3 and volume-level quorum, or 
with replica 2 and some 3rd observer, server quorum. It is also possible 
to have quorum with only 2 servers or replicas, but I wouldn't recommend 
it. With volume based quorum, the volume will go read only if the client 
loses connection with either server. With server quorum and only two 
servers, the server will shut down if it loses quorum completely 
removing access to the volume.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141028/a27dd415/attachment.html>