[Gluster-users] Undeletable volume

[Brian Candler]

[glusterfs 3.2.5, Ubuntu 11.10 x86_64]

Due to a problem with a node which was taken away for a while, but in the
mean time other volumes were created, I now have a situation where the
volume info has become unsynchronised.

There are three nodes. I have pasted 'gluster volume info' below: nodes
storage1 and storage2 know the volume called 'scratch', but node storage3
doesn't.

Now when I try to delete the volume, I can't:

root at storage2:~# gluster volume delete scratch
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
Volume scratch has been started.Volume needs to be stopped before deletion.
root at storage2:~# gluster volume stop scratch
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
Volume scratch does not exist

Hmm. So try to synchronise the info:

root at storage2:~# gluster volume sync scratch
please delete all the volumes before full sync

But I can't delete this volume, that's the problem!

I could try 3.2.6, except strangely there is no build for Ubuntu 11.10,
only 11.04 / 10.10 / 10.04, see
http://download.gluster.com/pub/gluster/glusterfs/3.2/3.2.6/Ubuntu/

So for now I've hacked about with the files manually while glusterd was not running:

    cd /etc/glusterd
    rm -rf vols/scratch
    vi nfs/nfs-server.vol

Then I restarted glusterd. However things still aren't right:

root at storage3:/etc/glusterd# gluster volume stop data
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
Stopping volume data has been unsuccessful
root at storage3:/etc/glusterd# gluster volume stop scratch3
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
Stopping volume scratch3 has been successful

So it looks like I'm going to have to just zap the glusterd config and start
again, which is OK since this system isn't yet in production - but it
doesn't really inspire confidence in how to get out a problem in a
production environment :-(

This may be some sort of split-brain in the volume metadata, but there's no
split-brain in the volume contents, since these are just distributed
volumes, no replication is taking place.

Anyway, I just thought I'd forward it on in case it's of any use in
improving gluster.

Regards,

Brian.

===================================================================================
root at storage1:~# gluster volume info

Volume Name: scratch
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: storage2:/disk/scratch
Brick2: storage3:/disk/scratch

Volume Name: scratch3
Type: Distribute
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: storage3:/disk/scratch/scratch3

Volume Name: data
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: storage1:/disk/data1/data
Brick2: storage1:/disk/data2/data
root at storage1:~# 



root at storage2:~# gluster volume info

Volume Name: scratch
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: storage2:/disk/scratch
Brick2: storage3:/disk/scratch

Volume Name: scratch3
Type: Distribute
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: storage3:/disk/scratch/scratch3

Volume Name: data
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: storage1:/disk/data1/data
Brick2: storage1:/disk/data2/data
root at storage2:~# 


root at storage3:~# gluster volume info

Volume Name: scratch3
Type: Distribute
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: storage3:/disk/scratch/scratch3

Volume Name: data
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: storage1:/disk/data1/data
Brick2: storage1:/disk/data2/data
root at storage3:~#