[Gluster-users] 3.3.1 Replicate only replicating one way

[Marcus Bointon]

On 6 Mar 2013, at 09:02, Todd Stansell <todd at stansell.org> wrote:

> In our recent testing, we saw all kinds of weird problems while testing
> rebuilding a failed brick in the same 2 node replicate cluster.  Several times
> we had to kill off all gluster processes and restart things from scratch to
> get the two sides talking correctly again (where both sides thought they were
> happily talking to the other side, but self-heal wasn't doing anything).  We'd
> run a full heal or stat some files and they wouldn't replicate back to the
> other side.  After restarting the processes (not just glusterd, but all of the
> glusterfs ones too), things would start working.  Once things were running and
> the nodes were properly replicating, it appeared to flow both ways nicely.

Thanks. I've managed to fix it by deleting every bit of gluster I could find, reinstalling and copying all the data back on.

I also saw that bug report of NFS hangs with 3.3.1, so I downgraded to 3.3.0 (which also meant switching to the older ppa).

It would be good to have a definitive list of where gluster puts everything - after uninstalling and deleting everything I could find, it still clearly had some info about my old config. I did this after unmounting volume and stopping all gluster services:

aptitude remove glusterfs-client glusterfs-server
aptitude purge glusterfs-client glusterfs-server
rm -rf /etc/glusterfs
rm -rf /var/log/glusterfs
rm -rf /var/lib/glusterfs
rm -rf /usr/lib/glusterfs
rm -rf /var/shared (my gluster storage area)

and yet when I reinstalled, I saw entries in the logs mentioning the brick storage area from my old installation - I've no idea where that info was lurking for it to find it again.

Incidentally, that also caused a bug of sorts. When starting glusterd it hung up, filling /var/log/glusterfs/etc-glusterfs-glusterd.vol.log with repeats of this:

[2013-03-05 19:24:21.137209] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.3.0
[2013-03-05 19:24:21.137393] E [glusterfsd.c:1296:glusterfs_pidfile_setup] 0-glusterfsd: pidfile /var/run/glusterd.pid lock error (Resource temporarily unavailable)

I couldn't find anything on google relating this error, but it seems it's caused when gluster can't find its storage area. I my case, creating /var/shared fixed this problem. I've no idea why it would report that as an issue with the pid file, but hopefully this will help someone else.

Marcus
-- 
Marcus Bointon
Synchromedia Limited: Creators of http://www.smartmessages.net/
UK info at hand CRM solutions
marcus at synchromedia.co.uk | http://www.synchromedia.co.uk/