[Gluster-users] Files lost after DHT expansion!

[mki-glusterfs at mozone.net]

Hi

I'm experimenting with a simple 5 storage node cluster with 2 clients,
using version 2.0.1 under linux.

The setup is a simple client side DHT across 4 of the 5 backend 
bricks.  Everything was working fine (performance aside).  Then I 
decided to grow the cluster to include the 5th brick.  After remounting
the filesystem on the clients, a bunch of the content in the cluster 
became unavailable!

Attempting to stat the missing files in question would give me an
ENOENT.  After reading some other thread post from a few days ago
someone mentioned adding "option lookup-unhashed on".  So I tried that
and then went through and manually stat()ed each file that was 
originally on the filesystem and only then did the files reappear.  I
can't imagine what one would do if there were several million files on
the filesystem prior to an expansion?

Am I missing something obvious in the setup and procedure for a
DHT expansion?

Please advise.

Thanks.

Mohan

The 5 storage nodes are setup as follows:

------------------------ BEGIN SERVER VOL FILE -----------------------
# base storage points
volume posix
        type storage/posix
        option background-unlink yes
        option directory /storage/export
end-volume

# required for async file replication
volume posix-locks
        type features/posix-locks
        subvolumes posix
end-volume

# performance translators
volume io-threads
        type performance/io-threads
        option thread-count 16
        subvolumes posix-locks
end-volume

#
volume server
        type protocol/server
        option transport-type tcp/server
        subvolumes io-threads
        option auth.addr.io-threads.allow 10.0.0.*
        option client-volume-filename /etc/glusterfs/glusterfs.vol
end-volume
------------------------ END SERVER VOL FILE -----------------------


The 2 client nodes are setup as follows (note that remote-brick-5
was added to the distribute volume for the expansion, it wasn't
there prior to):

------------------------ BEGIN CLIENT VOL FILE -----------------------
volume remote-brick-1
        type protocol/client
        option transport-type tcp
        option remote-host 10.0.0.161
        option remote-subvolume io-threads
end-volume

volume remote-brick-2
        type protocol/client
        option transport-type tcp
        option remote-host 10.0.0.162
        option remote-subvolume io-threads
end-volume

volume remote-brick-3
        type protocol/client
        option transport-type tcp
        option remote-host 10.0.0.163
        option remote-subvolume io-threads
end-volume

volume remote-brick-4
        type protocol/client
        option transport-type tcp
        option remote-host 10.0.0.164
        option remote-subvolume io-threads
end-volume

volume remote-brick-5
        type protocol/client
        option transport-type tcp
        option remote-host 10.0.0.160
        option remote-subvolume io-threads
end-volume

volume distribute
        type cluster/distribute
        subvolumes remote-brick-1 remote-brick-2 remote-brick-3 remote-brick-4 remote-brick-5
end-volume

volume io-threads
        type performance/io-threads
        option thread-count 16
        subvolumes distribute
end-volume

volume read-ahead
        type performance/read-ahead
        option page-count 16
        subvolumes io-threads
end-volume

volume io-cache
        type performance/io-cache
        option cache-size 512MB
        subvolumes read-ahead
end-volume

volume write-behind
        type performance/write-behind
        option cache-size 1MB
        subvolumes io-cache
end-volume
------------------------ END CLIENT VOL FILE -----------------------