[Gluster-users] Gluster File Abnormalities

[Kevin Leigeb]

All -

We recently moved from an old cluster running 3.7.9 to a new one running 3.8.4. To move the data we rsync'd all files from the old gluster nodes that were not in the .glusterfs directory and had a size of greater-than zero (to avoid stub files) through the front-end of the new cluster. However, it has recently come to our attention that some of the files copied over were already "corrupted" on the old back-end. That is, these files had permissions of 1000 (like a stub file) yet were the full size of the actual file. In some cases, these were the only copies of the file that existed at all on any of the bricks, in others, another version of the file existed that was also full size and had the proper permissions. In some cases, we believe, these correct files were rsync'd but then overwritten by the 1000 permission version resulting in a useless file on the new cluster.

These files are thought by the OS to be binaries when trying to open them using vim, but they are actually text files (or at least were originally). We can cat the file to see that it has a length of zero and so far that is our only reliable test to find which files are indeed corrupted (find . -type f | xargs wc -l). With nearly 50 million files on our cluster, this is really a non-starter because of the speed.

Has anyone seen this issue previously? We're hoping to find a solution that doesn't involve overthinking the problem and thought this might be a great place to start.

Let me know if there's any info I may have omitted that could be of further use.

Thanks,
Kevin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161114/57c93c36/attachment.html>