[Gluster-users] Gluster not recognizing available space

Pat Haley phaley at MIT.EDU
Tue Feb 4 01:28:40 UTC 2014


Quick update:

rebooting gluster-0-1 did not solve the problem.  The one
change I did notice is that now the only place I can write
is to one of the test directories I had previously created
which had not appeared on gluster-0-1.  Now, trying to create
a new directory elsewhere results in the "No space left on device"
error.

Pat


> 
> Hi,
> 
> I am writing to see if there are any suggestions
> as to what I should look at next to debug my
> problem or what I should try to correct it.\
> 
> Would rebooting the gluster-0-1 brick help?
> 
> A short recap:
>    In early January the gluster-0-1 brick filled up
>    (the gluster-data brick was close to being full
>     but the gluster-0-0 had 2T space left).
> 
>    I deleted files I knew to be on gluster-0-1 and
>    gluster-data (working through a client so as to
>    "go through" gluster).
> 
>    Now the gluster-0-1 brick is reporting to gluster
>    that it is full when it clearly is not.
> 
>    I have found that if I create a new directory,
>    I can write to it, but it only appears on the
>    other 2 bricks (gluster-data and gluster-0-0).
> 
> I appreciate any help in restoring the normal gluster
> function.  Thanks.
> 
> 
> Pat
> 
>>
>> Hi,
>>
>> I now have a change in the behavior of the gluster issue.
>> This afternoon I deleted a file that I knew was located
>> on the gluster-0-1 disk (the deleting command was issued
>> from a client).  After that I was able to make a directoy
>> (empty) that appeared on all 3 bricks.  My next step was
>> to copy a directory with files to the /gdata directory.
>> That appeared to be successful, but I looked and nothing
>> was written to the gluster-0-1 brick, not even the
>> new directory.  That seems to be the new state, the client
>> still sees files on gluster-0-1 but when it writes to the
>> /gdata area it only writes to the gluster-data and gluster-0-0
>> bricks.
>>
>>  From the bricks log on gluster-0-1 I have the following lines
>> from when I successfully created the test directory
>>
>> [2014-01-30 14:26:55.176965] W [posix-handle.c:538:posix_handle_soft] 
>> 0-gdata-posix: symlink 
>> ../../c9/14/c9144c1f-01a2-41ba-80cc-deec9aac03c5/test -> 
>> /mseas-data-0-1/.glusterfs/d9/b8/d9b84fcb-a452-4f4e-bdd9-ba1a392a3844 
>> failed (No space left on device)
>> [2014-01-30 14:26:55.205253] E [posix.c:969:posix_mkdir] 
>> 0-gdata-posix: setting gfid on /mseas-data-0-1/projects/atl/test failed
>>
>> Then when I tried copying a full directory
>>
>> [2014-01-30 14:30:35.716046] E [posix.c:962:posix_mkdir] 
>> 0-gdata-posix: mkdir of 
>> /mseas-data-0-1/projects/atl/FV_Matlab_Framework_JingPJH/trunk/Save/mat/Bottom_Gravity_Current_28 
>> failed: No space left on device
>> [2014-01-30 14:30:35.716133] I [server3_1-fops.c:529:server_mkdir_cbk] 
>> 0-gdata-server: 19027: MKDIR 
>> /projects/atl/FV_Matlab_Framework_JingPJH/trunk/Save/mat/Bottom_Gravity_Current_28 
>> (00000000-0000-0000-0000-000000000000) ==> -1 (No space left on device)
>> [2014-01-30 14:30:37.105372] I 
>> [server3_1-fops.c:1747:server_setattr_cbk] 0-gdata-server: 19029: 
>> SETATTR (null) (--) ==> -1 (No such file or directory)
>>
>>    That last line is repeated many times, followed by
>>
>> [2014-01-30 14:31:57.587539] I 
>> [server3_1-fops.c:1747:server_setattr_cbk] 0-gdata-server: 22077: 
>> SETATTR (null) (--) ==> -1 (No such file or directory)
>>
>> Does this suggest anything?  Is there additional information
>> I should post?
>>
>> Thanks,
>>
>> Pat
>>
>>
>>>
>>> Hi Joe,
>>>
>>> Sorry to take so long in responding, but we had another emergency
>>> that took all my time...
>>>
>>> The subsampled brick log file from gluster-0-1 is available at
>>>
>>> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-0-1/bricks/mseas-data-0-1.log.1 
>>>
>>>
>>> The df results on gluster-0-1 are
>>>
>>> [root at nas-0-1 glusterfs]# df /mseas-data-0-1/
>>> Filesystem           1K-blocks      Used Available Use% Mounted on
>>> /dev/sdb1            21484242304 20662833060 821409244  97% 
>>> /mseas-data-0-1
>>>
>>> [root at nas-0-1 glusterfs]# df -i /mseas-data-0-1/
>>> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
>>> /dev/sdb1            3290028528 4391552 3285636976    1% /mseas-data-0-1
>>>
>>> I've also included the same time period log files from gluster-0-1 and
>>> the other bricks in
>>>
>>> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-data/
>>> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-0-0/
>>> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-0-1/
>>>
>>> Thanks for the help!
>>>
>>> Pat
>>>
>>>> 1st, I don't think we need log data from November. Let's try to 
>>>> share logs that are limited to an example of the error you're trying 
>>>> to get help with. A 40Mb client log is unnecessary when it's only 
>>>> the last 5k that's relevant.
>>>>
>>>> 2nd, your "server" logs are mostly more client logs, with the 
>>>> exception of etc-glusterfs-glusterd.vol.log. The brick logs that 
>>>> might actually show what's happening are absent. 
>>>> (/var/log/glusterfs/bricks/*)
>>>>
>>>> The last line of the client log does show that 
>>>> gluster-0-1:/mseas-data-0-1 is reporting that it's out of space. On 
>>>> gluster-0-1 please share the log 
>>>> /var/log/glusterfs/bricks/gluster-0-1 but limit it to no more than 
>>>> an hour surrounding 2014-01-21 11:50:16. Also on that server, please 
>>>> share the result of "df /gluster-0-1 ; df -i /gluster-0-1".
>>>>
>>>> On 01/21/2014 10:58 PM, Pat Haley wrote:
>>>>>
>>>>> Hi Vijay,
>>>>>
>>>>> I've put the log files in
>>>>>
>>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/server_logs/
>>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/client_logs/
>>>>>
>>>>>> On 01/21/2014 10:24 PM, Pat Haley wrote:
>>>>>>>
>>>>>>> Hi Joe,
>>>>>>>
>>>>>>> The peer status on all 3 showed the
>>>>>>> proper connections.  Doing the killall
>>>>>>> and restart on all three bricks fixed
>>>>>>> the N in the Online column.  I then
>>>>>>> did have to remount the gluster
>>>>>>> filesystem on the client
>>>>>>>
>>>>>>> Unfortunately my original problem remains.
>>>>>>> I'm still getting "no space on left on device"
>>>>>>> when I try to write even a small file to
>>>>>>> the gluster filesystem.
>>>>>>>
>>>>>>> What should I look at next?
>>>>>>>
>>>>>>
>>>>>> Would it be possible to share the brick and client log files?
>>>>>>
>>>>>> Thanks,
>>>>>> Vijay
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>
>>
> 
> 


-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301



More information about the Gluster-users mailing list