[Gluster-users] Gluster not recognizing available space

Thu Jan 30 22:37:27 UTC 2014

Hi,

I now have a change in the behavior of the gluster issue.
This afternoon I deleted a file that I knew was located
on the gluster-0-1 disk (the deleting command was issued
from a client).  After that I was able to make a directoy
(empty) that appeared on all 3 bricks.  My next step was
to copy a directory with files to the /gdata directory.
That appeared to be successful, but I looked and nothing
was written to the gluster-0-1 brick, not even the
new directory.  That seems to be the new state, the client
still sees files on gluster-0-1 but when it writes to the
/gdata area it only writes to the gluster-data and gluster-0-0
bricks.

 From the bricks log on gluster-0-1 I have the following lines
from when I successfully created the test directory

[2014-01-30 14:26:55.176965] W [posix-handle.c:538:posix_handle_soft] 
0-gdata-posix: symlink 
../../c9/14/c9144c1f-01a2-41ba-80cc-deec9aac03c5/test -> 
/mseas-data-0-1/.glusterfs/d9/b8/d9b84fcb-a452-4f4e-bdd9-ba1a392a3844 
failed (No space left on device)
[2014-01-30 14:26:55.205253] E [posix.c:969:posix_mkdir] 0-gdata-posix: 
setting gfid on /mseas-data-0-1/projects/atl/test failed

Then when I tried copying a full directory

[2014-01-30 14:30:35.716046] E [posix.c:962:posix_mkdir] 0-gdata-posix: 
mkdir of 
/mseas-data-0-1/projects/atl/FV_Matlab_Framework_JingPJH/trunk/Save/mat/Bottom_Gravity_Current_28 
failed: No space left on device
[2014-01-30 14:30:35.716133] I [server3_1-fops.c:529:server_mkdir_cbk] 
0-gdata-server: 19027: MKDIR 
/projects/atl/FV_Matlab_Framework_JingPJH/trunk/Save/mat/Bottom_Gravity_Current_28 
(00000000-0000-0000-0000-000000000000) ==> -1 (No space left on device)
[2014-01-30 14:30:37.105372] I 
[server3_1-fops.c:1747:server_setattr_cbk] 0-gdata-server: 19029: 
SETATTR (null) (--) ==> -1 (No such file or directory)

    That last line is repeated many times, followed by

[2014-01-30 14:31:57.587539] I 
[server3_1-fops.c:1747:server_setattr_cbk] 0-gdata-server: 22077: 
SETATTR (null) (--) ==> -1 (No such file or directory)

Does this suggest anything?  Is there additional information
I should post?

Thanks,

Pat

> 
> Hi Joe,
> 
> Sorry to take so long in responding, but we had another emergency
> that took all my time...
> 
> The subsampled brick log file from gluster-0-1 is available at
> 
> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-0-1/bricks/mseas-data-0-1.log.1 
> 
> 
> The df results on gluster-0-1 are
> 
> [root at nas-0-1 glusterfs]# df /mseas-data-0-1/
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/sdb1            21484242304 20662833060 821409244  97% /mseas-data-0-1
> 
> [root at nas-0-1 glusterfs]# df -i /mseas-data-0-1/
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> /dev/sdb1            3290028528 4391552 3285636976    1% /mseas-data-0-1
> 
> I've also included the same time period log files from gluster-0-1 and
> the other bricks in
> 
> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-data/
> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-0-0/
> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-0-1/
> 
> Thanks for the help!
> 
> Pat
> 
>> 1st, I don't think we need log data from November. Let's try to share 
>> logs that are limited to an example of the error you're trying to get 
>> help with. A 40Mb client log is unnecessary when it's only the last 5k 
>> that's relevant.
>>
>> 2nd, your "server" logs are mostly more client logs, with the 
>> exception of etc-glusterfs-glusterd.vol.log. The brick logs that might 
>> actually show what's happening are absent. (/var/log/glusterfs/bricks/*)
>>
>> The last line of the client log does show that 
>> gluster-0-1:/mseas-data-0-1 is reporting that it's out of space. On 
>> gluster-0-1 please share the log /var/log/glusterfs/bricks/gluster-0-1 
>> but limit it to no more than an hour surrounding 2014-01-21 11:50:16. 
>> Also on that server, please share the result of "df /gluster-0-1 ; df 
>> -i /gluster-0-1".
>>
>> On 01/21/2014 10:58 PM, Pat Haley wrote:
>>>
>>> Hi Vijay,
>>>
>>> I've put the log files in
>>>
>>> http://mseas.mit.edu/download/phaley/GlusterUsers/server_logs/
>>> http://mseas.mit.edu/download/phaley/GlusterUsers/client_logs/
>>>
>>>> On 01/21/2014 10:24 PM, Pat Haley wrote:
>>>>>
>>>>> Hi Joe,
>>>>>
>>>>> The peer status on all 3 showed the
>>>>> proper connections.  Doing the killall
>>>>> and restart on all three bricks fixed
>>>>> the N in the Online column.  I then
>>>>> did have to remount the gluster
>>>>> filesystem on the client
>>>>>
>>>>> Unfortunately my original problem remains.
>>>>> I'm still getting "no space on left on device"
>>>>> when I try to write even a small file to
>>>>> the gluster filesystem.
>>>>>
>>>>> What should I look at next?
>>>>>
>>>>
>>>> Would it be possible to share the brick and client log files?
>>>>
>>>> Thanks,
>>>> Vijay
>>>>
>>>>
>>>>
>>>
>>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
> 

-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301