[Gluster-users] No space left on device

[zander at ekp.uni-karlsruhe.de]

Dear all,

today, we started using GlusterFS at our institute at a large scale. A 
problem occurred for the first migrated users, that was already 
discussed on this mailing list, but with an older version of glusterFS 
and, as far as I could tell, without a solution.

When users wants to create or move a file, they sometimes get the error 
message:

mv: cannot create regular file `/storage/cluster/<etc...>': No space 
left on device

Doing df -h tells me, however:

glusterfs#192.168.101.247:/lemmy
                      104T   69T   36T  66% /storage/cluster

It may be of importance that one brick in the cluster is actually 100% 
used. Others are almost completely empty. I am using GlusterFS.3.1.1, 
the file servers are running debian lenny or ubuntu server 10.04, 
clients are SLC4, SLC5, CentOS and ubuntu server 10.04.

The migration of the users was done in the following way:

The following volume was created:

Volume Name: lemmy
Type: Distribute
Status: Started
Number of Bricks: 5
Transport-type: tcp
Bricks:
Brick1: 192.168.101.249:/storage/4/cluster
Brick2: 192.168.101.248:/storage/5/cluster
Brick3: 192.168.101.250:/storage/6/cluster
Brick4: 192.168.101.247:/storage/7/cluster
Brick5: 192.168.101.246:/storage/8/cluster

The user directories were moved into one of the above folders 
(whichever required the least amount of network traffic), using "find 
-print" a file list was created and an "ls" was done on each line. All 
the data can be read by the users, and for the most part, they can also 
write to their directories. Not always, however, as described above.

Any advice is very much appreciated,
Best Regards,
Daniel