[Gluster-users] Problems with .gluster structure - bad symlinks

[Shawn Heisey]

Some background:

----

On version 3.3.1, we tried to rebalance after adding storage.  It blew
up badly due to this bug:

https://bugzilla.redhat.com/show_bug.cgi?id=859387

We have now upgraded to 3.4.2.  A new rebalance attempt resulted in a
several dozen entries showing up in the 'gluster volume heal $vol info'
output.

----

With the help of Joe Julian in the IRC channel, I made my way through
the heal problems, but I continue to get errors in my server logs.

I have now learned that there are a bunch of bad symlinks in the
.glusterfs structure on each of my bricks.  All of them say too many
levels of symbolic links.  I do not believe they are loops ... when I
manually checked a couple of them, they were actually valid, but had
more than the allowed number of symlinks in the chain.

cat:
/bricks/d00v00/mdfs/.glusterfs/65/30/6530ce82-310d-4c7c-8d14-135655328a77:
Too many levels of symbolic links

What do I need to do to fix this problem?  Is there something I can do
for each of the bad symlinks?  Would a 'heal full' do anything useful?
Do I need to do something more drastic, like take the volume down and
entirely remove (or rename) the .glusterfs structure from all 32 bricks
(16x2 distributed-replicate)?  I don't want to cause myself more
problems, but I want to get the volume in a completely pristine state
and NOT risk losing any of the 52 terabytes of data that's in the volume.

Thanks,
Shawn