[Gluster-users] No space left on device

[Daniel Zander]

Hi,

yes, we already checked that
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1            3055616   51143 3004473    2% /
none                 2057204     720 2056484    1% /dev
none                 2058311       1 2058310    1% /dev/shm
none                 2058311      22 2058289    1% /var/run
none                 2058311       2 2058309    1% /var/lock
none                 2058311       3 2058308    1% /lib/init/rw
/dev/sdb             4818066560 4277250 4813789310    1% /storage/6
/dev/sda6            1220608    2232 1218376    1% /var
/dev/sda7            2445984      12 2445972    1% /tmp

We managed to free some more space (now ~1TB free) and creating files 
seem to work again. However, during the migration of a user, some files 
don't show up in his directory, although we are able to find them via ls 
<filename>.

We decided to reverse the migration and try again, but then I was not 
able to delete his directory:

rm: FATAL: directory `delete_me/rootfiles/dplusdminus' changed dev/ino

Also, when we tried chown -R <user>:<group> user/ the following error 
popped up.

chown: fts_read failed: No such file or directory

Additionally, some directories show up in ls -l like that

?---------  ? ?    ?        ?            ? b_dd

This all is really confusing to us. Can someone shed some light on this 
mess?

Regards,
Daniel


On 01/19/2011 10:53 AM, Andrew Séguin wrote:
> Hi,
>
> Maybe a silly thought / maybe you have already thought of it, but is
> there still some free inodes on the underlying filesystem (df -i)?
>
> Regards,
> Andrew
>
> On Wed, Jan 19, 2011 at 10:44 AM, Daniel Zander
> <zander at ekp.uni-karlsruhe.de>  wrote:
>> Hi,
>>
>> we just made some space available on the affected brick (386 GB free), but
>> still the same problem remains. But I don't think that this is really a
>> glusterFS problem, as also root cannot create any directories directly on
>> the fileserver anymore.
>>
>> Thanks so far,
>> Daniel
>>
>>
>>
>> On 01/19/2011 10:12 AM, Mark "Naoki" Rogers wrote:
>>>
>>> I think you might want to look into re-balance:
>>>
>>> http://europe.gluster.org/community/documentation/index.php?title=Gluster_3.1:_Rebalancing_Volumes&redirect=no
>>>
>>> <http://europe.gluster.org/community/documentation/index.php?title=Gluster_3.1:_Rebalancing_Volumes&redirect=no>
>>>
>>>
>>> It's generally for adding/removing bricks but might re-distribute data
>>> in a way that solves your disk space issue.
>>>
>>>
>>> On 01/19/2011 04:43 PM, Daniel Zander wrote:
>>>>
>>>> Hi!
>>>>
>>>>> Assuming you are doing a straight distribute there(?), if the user in
>>>>
>>>> Yes's it's a distributed volume.
>>>>
>>>>> question is hashed onto the brick that is 100% full you'll get a space
>>>>
>>>> Is there a way around this other than moving files away from this one
>>>> brick by hand?
>>>>
>>>>> error. Not sure I followed your migration details though, when you say
>>>>> "user directories were moved into one of the above folders" do you mean
>>>>> copied directly onto the individual storage bricks?
>>>>
>>>> Yes, eg. user_a had the following directories
>>>> server5:/storage/5/user_a
>>>> server6:/storage/6/user_a
>>>>
>>>> Then we performed a move:
>>>> ssh server5 "mv /storage/5/user_a/ /storage/5/cluster/user_a"
>>>> ssh server6 "mv /storage/6/user_a/ /storage/6/cluster/user_a"
>>>>
>>>> This was done as it would not cause any network traffic. Then the volume
>>>> was created like that:
>>>> Brick1: 192.168.101.249:/storage/4/cluster
>>>> Brick2: 192.168.101.248:/storage/5/cluster
>>>> Brick3: 192.168.101.250:/storage/6/cluster
>>>> Brick4: 192.168.101.247:/storage/7/cluster
>>>> Brick5: 192.168.101.246:/storage/8/cluster
>>>>
>>>> Regards,
>>>> Daniel
>>>>
>>>>> On 01/19/2011 05:01 AM, zander at ekp.uni-karlsruhe.de wrote:
>>>>>>
>>>>>> mv: cannot create regular file `/storage/cluster/<etc...>': No space
>>>>>> left on device
>>>>>>
>>>>>> Doing df -h tells me, however:
>>>>>>
>>>>>> glusterfs#192.168.101.247:/lemmy
>>>>>> 104T 69T 36T 66% /storage/cluster
>>>>>>
>>>>>> It may be of importance that one brick in the cluster is actually 100%
>>>>>> used. Others are almost completely empty. I am using GlusterFS.3.1.1,
>>>>>> the file servers are running debian lenny or ubuntu server 10.04,
>>>>>> clients are SLC4, SLC5, CentOS and ubuntu server 10.04.
>>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>