[Gluster-users] brick does not come online with gluster 5.0, even with fresh install

[Computerisms Corporation]

Hi,

Fortunately I am playing in a sandbox right now, but I am good and stuck 
and hoping someone can point me in the right direction.

I have been playing for about 3 months with a gluster that currently has 
one brick.  The idea is that I have a server with data, I need to 
migrate that server onto the new gluster-capable server, then I can use 
the original server to make a 2nd brick, then I will be able to make 
some room on a 3rd server for an arbiter brick.  So I am building and 
testing to be sure it all works before I try it in production.

Yesterday morning I was plugging away at figuring out how to make stuff 
work on the new gluster server when I ran into an issue trying to rm -rf 
a directory and it telling me it wasn't empty when ls -al showed that it 
was.  This has happened to me before, and what I did to fix it before 
was unmount the Glusterfs, go into the brick, delete the files, and 
remount the Glusterfs.  I did that and it appeared to mount fine, but 
when I tried to access the gluster mount, it gave me an error that there 
were too many levels of symlinks.

I spent my day yesterday trying pretty much everything I could find on 
google and a few things I couldn't.  In the past when stuff has gone 
funny with gluster on this box, I have always shut everything down and 
checked if there was a new version of gluster, and indeed there was 
version 5.0 available.  So I did the upgrade quite early in the day. 
Sadly it didn't fix my problem, but it did give me an error that led me 
to modifying my hosts file to be ipv6 resolvable.  Also after that, the 
only time the gluster would mount was at reboot, but always with the 
symlinks error, and it wasn't really mounted as reported by mount, but 
the directory could be unmounted.

Having struck out completely yesterday, today I decided to start with a 
new machine.  I kept a history of the commands I had used to build the 
gluster a few months back and pasted them all in.  Found that the 5.0 
package does not enable systemd, found that I needed the ipv6 entries in 
the hosts file again, and also found the same problem in that the 
glusterfs would not mount, the symlinks error at reboot, and the same 
log entries.

I am still pretty new with gluster, so my best may not be that good, but 
as best as I can tell the issue is that the brick will not start, even 
with the force option.  I think the problem boils down to one or both of 
two lines in the logs.  In the glusterd.log I have a line:

0-: Unable to read pidfile: 
/var/run/gluster/vols/moogle-gluster/sand1lian.computerisms.ca-var-GlusterBrick-moogle-gluster.pid

The file exists, and I can't see anything wrong with permissions on the 
file or the file tree leading to it, but it is a zero-bit file, so I am 
thinking the problem is not the file, but that it can't read the 
contents of the file because there aren't any.

The other log entry is in the brick log:

0-glusterfs: connection attempt on  failed, (Invalid argument)

When I looked this up, it seems in my case there should be an attempt to 
connect on 127.0.0.1, but given the double space I am thinking the host 
argument is null, hence the invalid argument.  It occurs that maybe I 
still need some other entry in my hosts file to satisfy this, but I 
can't think what it would be.  I have created DNS entries; dig works, 
and both hostname and FQDN resolve.

I have tried to change a lot of things today, so probably things are 
buggered up beyond hope right now so even if I do find the solution 
maybe it won't work.  will wipe the new machine and start over again 
tomorrow.

I realize the post is kinda long, sorry for that, but I want to make 
sure I get every thing important.  In fairness, though, I could easily 
double the length of this post with possibly relevant things (if you are 
interested).  If you are still reading, thank you so much, I would 
appreciate anything, even a wild guess, as to how to move forward on this?

-- 
Bob Miller
Cell: 867-334-7117
Office: 867-633-3760
www.computerisms.ca