[Gluster-users] Deleted files reappearing

[Øystein Viggen]

Hi,

I have a small test setup on Ubuntu 12.04, using the
3.4.1-ubuntu1~precise1 packages of glusterfs from the recommended PPA.
There are two gluster servers (replicate 2) and one client.  Bricks are
16 GB xfs -i size=512 filesystems.  All servers run on vmware.

I've been using the linux kernel source for some simple performance and
stability tests with many small files.  When deleting the linux kernel
tree with rm -Rf while rebooting one glusterfs server, it seems that
some deletes are missed, or "recreated".  Here's how it goes:

root at client:/mnt# rm -Rf linux-3.12

At this point, I run "shutdown -r now" on one server.  The deletion
seems to keep running just fine, but just as the server comes back up, I
get something like this on the client:

rm: cannot remove `linux-3.12/arch/mips/ralink/dts': Directory not empty

After the rm has run to completion:

root at client:/mnt# find linux-3.12 -type f
linux-3.12/arch/mips/ralink/dts/Makefile

Sometimes it's more than one file, too.  "gluster volume heal volname
info" shows no outstanding entries. 

If I turn off one server before running rm, and turn it on during the rm
run, a similar thing happens, only it seems worse.  In one test, I had
9220 files left after rm had finished.

If both servers are up during the rm run, all files are deleted as
expected every time.


What is happening here, and can I do something to avoid it?

I was hoping that in a replica 2 cluster, you could safely reboot one
server at a time (with sync-up time in between) to, say, apply OS
patches without taking the gluster volume offline.


I'm thankful for any help.

Øystein