[Gluster-users] data disappearing with 'mv' command executed on a directory.

[Andrew Smith]

Hi, 

Recently, I moved a few directories and found that not all the
data was moved. 

I had several directories that had what looked like valid data 
inside. Then I performed a ‘mv’ command to relocate the data 
on the same volume. The data was not copied, just moved to 
a different spot. 

After the mv, I found that a large number of files were missing
from within the directory. I found that one of my bricks still
had the files in the old location. I was able to recover the files.

To try to avoid this problem, I have run the ‘backend-cleanup.sh’
script on my bricks. I assume that I had some corrupted pointers
somewhere. This seems to have kept the problem from recurring,
but my statistics are low.

So, I have some questions:

1) The orphan files are still living on that brick. Is it OK to
just delete them? I am always reluctant to just delete files off a
brick.

2) What is backend-clenaup.sh doing? It finds many links to delete,
then read and write to that file system for a time and run it again
only to find that it deletes many more files. Does this mean that 
normal activity is causing problematic links to appear on my bricks,
or is it deleting good links along with the bad ones?

3) I am currently rebalancing my system, which will take some time,
maybe weeks. Is it OK to move directories around during a rebalance?
Could it just miss those directories, or rebalance them twice, or 
could bad things happen?

System Configuration: 

8 brick Gluster system with a total capacity of 625TB
In the past few months I have performed several “remove-brick’
and “add-brick” each time followed by a rebalance with a
“fix-layout”, since an complete rebalance would take too long.

As a result, my system is quite out of balance, which 
I assumed could impact performance, not reliability.

Best,
Andrew Smith