[Bugs] [Bug 1322145] Glusterd fails to restart after replacing a failed GlusterFS node and a volume has a snapshot

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1322145



--- Comment #6 from Ben Werthmann <ben at apcera.com> ---
Another Correction: We remove the dead peer via 'gluster peer detach
$failed_peer force', AFTER 'gluster volume replace-brick $vol $failed_peer
$new_peer_ip:$new_brick commit force'. 

Found this thread:
https://www.gluster.org/pipermail/gluster-users/2015-June/022264.html

Current IPs are 172.27.20.205, 172.27.18.105, and 172.27.21.136. 172.27.20.50
was the IP of the node which was replaced.


ubuntu at bwerthmann-743d2d98:~$ sudo grep -iR 172 /var/lib/glusterd/* | grep -v
20\.205 | grep -v 18\.105 | grep -v 21\.136
/var/lib/glusterd/snaps/s1_GMT-2016.03.30-23.30.58/7c2ef5a4775648a1adf20bc1f7ae764b/info:brick-1=172.27.20.50:-run-gluster-snaps-7c2ef5a4775648a1adf20bc1f7ae764b-brick2-apcfs
/var/lib/glusterd/snaps/s1_GMT-2016.03.30-23.30.58/7c2ef5a4775648a1adf20bc1f7ae764b/bricks/172.27.20.50:-run-gluster-snaps-7c2ef5a4775648a1adf20bc1f7ae764b-brick2-apcfs:hostname=172.27.20.50
/var/lib/glusterd/snaps/s1_GMT-2016.03.30-23.30.58/7c2ef5a4775648a1adf20bc1f7ae764b/trusted-7c2ef5a4775648a1adf20bc1f7ae764b.tcp-fuse.vol:
   option remote-host 172.27.20.50
/var/lib/glusterd/snaps/s1_GMT-2016.03.30-23.30.58/7c2ef5a4775648a1adf20bc1f7ae764b/7c2ef5a4775648a1adf20bc1f7ae764b.tcp-fuse.vol:
   option remote-host 172.27.20.50
/var/lib/glusterd/snaps/s0_GMT-2016.03.30-23.18.02/2996d3515da24ddd859f32c294587ca8/2996d3515da24ddd859f32c294587ca8.tcp-fuse.vol:
   option remote-host 172.27.20.50
/var/lib/glusterd/snaps/s0_GMT-2016.03.30-23.18.02/2996d3515da24ddd859f32c294587ca8/info:brick-1=172.27.20.50:-run-gluster-snaps-2996d3515da24ddd859f32c294587ca8-brick2-apcfs
/var/lib/glusterd/snaps/s0_GMT-2016.03.30-23.18.02/2996d3515da24ddd859f32c294587ca8/bricks/172.27.20.50:-run-gluster-snaps-2996d3515da24ddd859f32c294587ca8-brick2-apcfs:hostname=172.27.20.50
/var/lib/glusterd/snaps/s0_GMT-2016.03.30-23.18.02/2996d3515da24ddd859f32c294587ca8/trusted-2996d3515da24ddd859f32c294587ca8.tcp-fuse.vol:
   option remote-host 172.27.20.50
/var/lib/glusterd/snaps/s2_GMT-2016.03.30-23.31.03/86baab41a3174d6bba13325a85d9a46b/info:brick-1=172.27.20.50:-run-gluster-snaps-86baab41a3174d6bba13325a85d9a46b-brick2-apcfs
/var/lib/glusterd/snaps/s2_GMT-2016.03.30-23.31.03/86baab41a3174d6bba13325a85d9a46b/bricks/172.27.20.50:-run-gluster-snaps-86baab41a3174d6bba13325a85d9a46b-brick2-apcfs:hostname=172.27.20.50
/var/lib/glusterd/snaps/s2_GMT-2016.03.30-23.31.03/86baab41a3174d6bba13325a85d9a46b/86baab41a3174d6bba13325a85d9a46b.tcp-fuse.vol:
   option remote-host 172.27.20.50
/var/lib/glusterd/snaps/s2_GMT-2016.03.30-23.31.03/86baab41a3174d6bba13325a85d9a46b/trusted-86baab41a3174d6bba13325a85d9a46b.tcp-fuse.vol:
   option remote-host 172.27.20.50

ubuntu at bwerthmann-743d2d98:~$ find /var/lib/glusterd/ -name *\.20\.50*
/var/lib/glusterd/snaps/s1_GMT-2016.03.30-23.30.58/7c2ef5a4775648a1adf20bc1f7ae764b/7c2ef5a4775648a1adf20bc1f7ae764b.172.27.20.50.run-gluster-snaps-7c2ef5a4775648a1adf20bc1f7ae764b-brick2-apcfs.vol
/var/lib/glusterd/snaps/s1_GMT-2016.03.30-23.30.58/7c2ef5a4775648a1adf20bc1f7ae764b/bricks/172.27.20.50:-run-gluster-snaps-7c2ef5a4775648a1adf20bc1f7ae764b-brick2-apcfs
/var/lib/glusterd/snaps/s0_GMT-2016.03.30-23.18.02/2996d3515da24ddd859f32c294587ca8/2996d3515da24ddd859f32c294587ca8.172.27.20.50.run-gluster-snaps-2996d3515da24ddd859f32c294587ca8-brick2-apcfs.vol
/var/lib/glusterd/snaps/s0_GMT-2016.03.30-23.18.02/2996d3515da24ddd859f32c294587ca8/bricks/172.27.20.50:-run-gluster-snaps-2996d3515da24ddd859f32c294587ca8-brick2-apcfs
/var/lib/glusterd/snaps/s2_GMT-2016.03.30-23.31.03/86baab41a3174d6bba13325a85d9a46b/86baab41a3174d6bba13325a85d9a46b.172.27.20.50.run-gluster-snaps-86baab41a3174d6bba13325a85d9a46b-brick2-apcfs.vol
/var/lib/glusterd/snaps/s2_GMT-2016.03.30-23.31.03/86baab41a3174d6bba13325a85d9a46b/bricks/172.27.20.50:-run-gluster-snaps-86baab41a3174d6bba13325a85d9a46b-brick2-apcfs

ubuntu at bwerthmann-743d2d98:~$ sudo find /var/lib/glusterd/ -type f -exec sed -i
"s/172.27.20.50/172.27.20.205/g" {} \;
ubuntu at bwerthmann-743d2d98:~$ sudo grep -iR 172 /var/lib/glusterd/* | grep -v
20\.205 | grep -v 18\.105 | grep -v 21\.136
ubuntu at bwerthmann-743d2d98:~$
ubuntu at bwerthmann-743d2d98:~$ sudo rm -rf /var/lib/glusterd/snaps/s*

Now glusterd starts

ubuntu at bwerthmann-743d2d98:~$ sudo service glusterfs-server start
glusterfs-server start/running, process 6637
ubuntu at bwerthmann-743d2d98:~$ sudo service glusterfs-server status
glusterfs-server start/running, process 6637

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=4yca1D4bob&a=cc_unsubscribe