[Gluster-users] Data corruption

[Bob Black]

Hello community of Gluster,

Sorry for the long post.
TL;DR: stock gluster 3.3.0 on 5 nodes results in massive data
corruption on brick "failure" or peer disconnection.

We are having problems with data corruption on VM volumes with VMs
running on top of Gluster 3.3.0 when introducing brick failures and/or
node disconnects.
In our setup we have 5 storage nodes with 16 core AMD Opteron(tm)
Processor 6128, 32GB ram and 34 2TB SATA disks. To utilize the the
storage nodes we have 20 compute nodes with 24 core AMD Opteron(TM)
Processor 6238 and 128GB ram.

To be able to test and verify this setup we installed Gluster 3.3.0 on
the storage nodes and GlusterFS 3.3.0 client on the compute nodes.
We created one brick for each hard drive and created
Distributed-Replicate volume with the bricks with tcp,rdma transport.
The volume was mounted with glusterfs over tcp transport over
infiniband on all the compute nodes. We created 500 virtual machines
on the compute nodes and made them do heavy IO benchmarking on the
volume and Gluster performed as expected.
Then we created sanity test script that creates files, copies over and
over again and does md5 sums of all written data and does md5 check of
all the operating system. We ran this test on all the VMs
successfully, then we did it again and stopped one storage node for
few minutes and started it again, which gluster recovered from
successfully.
Then we ran this test again but with kill -9 on all Gluster processes
on one node for more than an hour. We kept the tests running to
emulate load and then started the Gluster deamon on the storage node
again. Now around 10% of all VMs lost connection to Gluster and failed
to "read-only" file-system and more instances got some data
corruption, missing or broken files. Very bad!

We wiped the VMs and created new ones instead. Started the same test
again but now we terminated 4 bricks on one node and carried out load
testing to test shrinking and re-balancing. Before we got the chance
to remove/move bricks we started getting bunch of corrupted VMs and
data corruption and after re-balancing we got a load of kernel panics
on the VMs. Very bad indeed!

Are anyone else having the same problem, is there anything we are
doing wrong, is this lost cause?

Thanks for any input.
-Bob