[Gluster-users] Gluster not recognizing available space
Pat Haley
phaley at MIT.EDU
Tue Feb 4 01:28:40 UTC 2014
Quick update:
rebooting gluster-0-1 did not solve the problem. The one
change I did notice is that now the only place I can write
is to one of the test directories I had previously created
which had not appeared on gluster-0-1. Now, trying to create
a new directory elsewhere results in the "No space left on device"
error.
Pat
>
> Hi,
>
> I am writing to see if there are any suggestions
> as to what I should look at next to debug my
> problem or what I should try to correct it.\
>
> Would rebooting the gluster-0-1 brick help?
>
> A short recap:
> In early January the gluster-0-1 brick filled up
> (the gluster-data brick was close to being full
> but the gluster-0-0 had 2T space left).
>
> I deleted files I knew to be on gluster-0-1 and
> gluster-data (working through a client so as to
> "go through" gluster).
>
> Now the gluster-0-1 brick is reporting to gluster
> that it is full when it clearly is not.
>
> I have found that if I create a new directory,
> I can write to it, but it only appears on the
> other 2 bricks (gluster-data and gluster-0-0).
>
> I appreciate any help in restoring the normal gluster
> function. Thanks.
>
>
> Pat
>
>>
>> Hi,
>>
>> I now have a change in the behavior of the gluster issue.
>> This afternoon I deleted a file that I knew was located
>> on the gluster-0-1 disk (the deleting command was issued
>> from a client). After that I was able to make a directoy
>> (empty) that appeared on all 3 bricks. My next step was
>> to copy a directory with files to the /gdata directory.
>> That appeared to be successful, but I looked and nothing
>> was written to the gluster-0-1 brick, not even the
>> new directory. That seems to be the new state, the client
>> still sees files on gluster-0-1 but when it writes to the
>> /gdata area it only writes to the gluster-data and gluster-0-0
>> bricks.
>>
>> From the bricks log on gluster-0-1 I have the following lines
>> from when I successfully created the test directory
>>
>> [2014-01-30 14:26:55.176965] W [posix-handle.c:538:posix_handle_soft]
>> 0-gdata-posix: symlink
>> ../../c9/14/c9144c1f-01a2-41ba-80cc-deec9aac03c5/test ->
>> /mseas-data-0-1/.glusterfs/d9/b8/d9b84fcb-a452-4f4e-bdd9-ba1a392a3844
>> failed (No space left on device)
>> [2014-01-30 14:26:55.205253] E [posix.c:969:posix_mkdir]
>> 0-gdata-posix: setting gfid on /mseas-data-0-1/projects/atl/test failed
>>
>> Then when I tried copying a full directory
>>
>> [2014-01-30 14:30:35.716046] E [posix.c:962:posix_mkdir]
>> 0-gdata-posix: mkdir of
>> /mseas-data-0-1/projects/atl/FV_Matlab_Framework_JingPJH/trunk/Save/mat/Bottom_Gravity_Current_28
>> failed: No space left on device
>> [2014-01-30 14:30:35.716133] I [server3_1-fops.c:529:server_mkdir_cbk]
>> 0-gdata-server: 19027: MKDIR
>> /projects/atl/FV_Matlab_Framework_JingPJH/trunk/Save/mat/Bottom_Gravity_Current_28
>> (00000000-0000-0000-0000-000000000000) ==> -1 (No space left on device)
>> [2014-01-30 14:30:37.105372] I
>> [server3_1-fops.c:1747:server_setattr_cbk] 0-gdata-server: 19029:
>> SETATTR (null) (--) ==> -1 (No such file or directory)
>>
>> That last line is repeated many times, followed by
>>
>> [2014-01-30 14:31:57.587539] I
>> [server3_1-fops.c:1747:server_setattr_cbk] 0-gdata-server: 22077:
>> SETATTR (null) (--) ==> -1 (No such file or directory)
>>
>> Does this suggest anything? Is there additional information
>> I should post?
>>
>> Thanks,
>>
>> Pat
>>
>>
>>>
>>> Hi Joe,
>>>
>>> Sorry to take so long in responding, but we had another emergency
>>> that took all my time...
>>>
>>> The subsampled brick log file from gluster-0-1 is available at
>>>
>>> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-0-1/bricks/mseas-data-0-1.log.1
>>>
>>>
>>> The df results on gluster-0-1 are
>>>
>>> [root at nas-0-1 glusterfs]# df /mseas-data-0-1/
>>> Filesystem 1K-blocks Used Available Use% Mounted on
>>> /dev/sdb1 21484242304 20662833060 821409244 97%
>>> /mseas-data-0-1
>>>
>>> [root at nas-0-1 glusterfs]# df -i /mseas-data-0-1/
>>> Filesystem Inodes IUsed IFree IUse% Mounted on
>>> /dev/sdb1 3290028528 4391552 3285636976 1% /mseas-data-0-1
>>>
>>> I've also included the same time period log files from gluster-0-1 and
>>> the other bricks in
>>>
>>> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-data/
>>> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-0-0/
>>> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-0-1/
>>>
>>> Thanks for the help!
>>>
>>> Pat
>>>
>>>> 1st, I don't think we need log data from November. Let's try to
>>>> share logs that are limited to an example of the error you're trying
>>>> to get help with. A 40Mb client log is unnecessary when it's only
>>>> the last 5k that's relevant.
>>>>
>>>> 2nd, your "server" logs are mostly more client logs, with the
>>>> exception of etc-glusterfs-glusterd.vol.log. The brick logs that
>>>> might actually show what's happening are absent.
>>>> (/var/log/glusterfs/bricks/*)
>>>>
>>>> The last line of the client log does show that
>>>> gluster-0-1:/mseas-data-0-1 is reporting that it's out of space. On
>>>> gluster-0-1 please share the log
>>>> /var/log/glusterfs/bricks/gluster-0-1 but limit it to no more than
>>>> an hour surrounding 2014-01-21 11:50:16. Also on that server, please
>>>> share the result of "df /gluster-0-1 ; df -i /gluster-0-1".
>>>>
>>>> On 01/21/2014 10:58 PM, Pat Haley wrote:
>>>>>
>>>>> Hi Vijay,
>>>>>
>>>>> I've put the log files in
>>>>>
>>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/server_logs/
>>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/client_logs/
>>>>>
>>>>>> On 01/21/2014 10:24 PM, Pat Haley wrote:
>>>>>>>
>>>>>>> Hi Joe,
>>>>>>>
>>>>>>> The peer status on all 3 showed the
>>>>>>> proper connections. Doing the killall
>>>>>>> and restart on all three bricks fixed
>>>>>>> the N in the Online column. I then
>>>>>>> did have to remount the gluster
>>>>>>> filesystem on the client
>>>>>>>
>>>>>>> Unfortunately my original problem remains.
>>>>>>> I'm still getting "no space on left on device"
>>>>>>> when I try to write even a small file to
>>>>>>> the gluster filesystem.
>>>>>>>
>>>>>>> What should I look at next?
>>>>>>>
>>>>>>
>>>>>> Would it be possible to share the brick and client log files?
>>>>>>
>>>>>> Thanks,
>>>>>> Vijay
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>
>>
>
>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: phaley at mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
More information about the Gluster-users
mailing list