[Gluster-users] issues replacing a failed node

[John Jolet]

I had a two-node replicated/distributed volume, spread across server1:/bricks/1 server2:/bricks/1 server1:/bricks/2 server2:/bricks/2.  I powered down server2 in order to re-rack it to make room for server3.  server2 fails to come up, for reasons having nothing to do with gluster.  So I decided to go ahead and bring up server3 and move server2's bricks to it.  I saw conflicting information on how to do that with a completely dead node and a new node of a different name.  Basically i did a peer probe server3, then volume replace-brick share name server2:/bricks/1 server3:/bricks/1.  then i did a volume replace-brick <blah> commit force.

this was probably a bad thing.  then i tried to do the replace-brick with the second set.  it fails to start saying replace-brick is already running on the volume.  now i'm stuck.  the data in brick/1 DOES appear on the new node, but i can't do anything with brick/2.  

if i try to do a commit, it says bricks/1 isn't on server2, and if i try to do anything else it says replace-brick is running.  i did a rebalance, hoping that would fix it, but it has not.  I attempted to stop the volume, but it said i couldn't until the replace-brick was committed or aborted.  I cannot abort, it says replace-brick abort failed.  Now what?  Mind, this is a temporary setup which has a complex directory structure, but no data as yet.  We are looking to use this for production VERY soon, and i'm not sure that (a) i have time to rebuild everything, and (and more importantly) (b) i need to be able to demonstrate to management that "look, a node failed and we replaced it with no data loss".

so, what's my next step to get this mess untangled, and the data safely on my new node...