[Gluster-users] Gluster not recognizing available space
Pat Haley
phaley at MIT.EDU
Mon Feb 3 19:30:42 UTC 2014
Hi,
I am writing to see if there are any suggestions
as to what I should look at next to debug my
problem or what I should try to correct it.\
Would rebooting the gluster-0-1 brick help?
A short recap:
In early January the gluster-0-1 brick filled up
(the gluster-data brick was close to being full
but the gluster-0-0 had 2T space left).
I deleted files I knew to be on gluster-0-1 and
gluster-data (working through a client so as to
"go through" gluster).
Now the gluster-0-1 brick is reporting to gluster
that it is full when it clearly is not.
I have found that if I create a new directory,
I can write to it, but it only appears on the
other 2 bricks (gluster-data and gluster-0-0).
I appreciate any help in restoring the normal gluster
function. Thanks.
Pat
>
> Hi,
>
> I now have a change in the behavior of the gluster issue.
> This afternoon I deleted a file that I knew was located
> on the gluster-0-1 disk (the deleting command was issued
> from a client). After that I was able to make a directoy
> (empty) that appeared on all 3 bricks. My next step was
> to copy a directory with files to the /gdata directory.
> That appeared to be successful, but I looked and nothing
> was written to the gluster-0-1 brick, not even the
> new directory. That seems to be the new state, the client
> still sees files on gluster-0-1 but when it writes to the
> /gdata area it only writes to the gluster-data and gluster-0-0
> bricks.
>
> From the bricks log on gluster-0-1 I have the following lines
> from when I successfully created the test directory
>
> [2014-01-30 14:26:55.176965] W [posix-handle.c:538:posix_handle_soft]
> 0-gdata-posix: symlink
> ../../c9/14/c9144c1f-01a2-41ba-80cc-deec9aac03c5/test ->
> /mseas-data-0-1/.glusterfs/d9/b8/d9b84fcb-a452-4f4e-bdd9-ba1a392a3844
> failed (No space left on device)
> [2014-01-30 14:26:55.205253] E [posix.c:969:posix_mkdir] 0-gdata-posix:
> setting gfid on /mseas-data-0-1/projects/atl/test failed
>
> Then when I tried copying a full directory
>
> [2014-01-30 14:30:35.716046] E [posix.c:962:posix_mkdir] 0-gdata-posix:
> mkdir of
> /mseas-data-0-1/projects/atl/FV_Matlab_Framework_JingPJH/trunk/Save/mat/Bottom_Gravity_Current_28
> failed: No space left on device
> [2014-01-30 14:30:35.716133] I [server3_1-fops.c:529:server_mkdir_cbk]
> 0-gdata-server: 19027: MKDIR
> /projects/atl/FV_Matlab_Framework_JingPJH/trunk/Save/mat/Bottom_Gravity_Current_28
> (00000000-0000-0000-0000-000000000000) ==> -1 (No space left on device)
> [2014-01-30 14:30:37.105372] I
> [server3_1-fops.c:1747:server_setattr_cbk] 0-gdata-server: 19029:
> SETATTR (null) (--) ==> -1 (No such file or directory)
>
> That last line is repeated many times, followed by
>
> [2014-01-30 14:31:57.587539] I
> [server3_1-fops.c:1747:server_setattr_cbk] 0-gdata-server: 22077:
> SETATTR (null) (--) ==> -1 (No such file or directory)
>
> Does this suggest anything? Is there additional information
> I should post?
>
> Thanks,
>
> Pat
>
>
>>
>> Hi Joe,
>>
>> Sorry to take so long in responding, but we had another emergency
>> that took all my time...
>>
>> The subsampled brick log file from gluster-0-1 is available at
>>
>> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-0-1/bricks/mseas-data-0-1.log.1
>>
>>
>> The df results on gluster-0-1 are
>>
>> [root at nas-0-1 glusterfs]# df /mseas-data-0-1/
>> Filesystem 1K-blocks Used Available Use% Mounted on
>> /dev/sdb1 21484242304 20662833060 821409244 97%
>> /mseas-data-0-1
>>
>> [root at nas-0-1 glusterfs]# df -i /mseas-data-0-1/
>> Filesystem Inodes IUsed IFree IUse% Mounted on
>> /dev/sdb1 3290028528 4391552 3285636976 1% /mseas-data-0-1
>>
>> I've also included the same time period log files from gluster-0-1 and
>> the other bricks in
>>
>> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-data/
>> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-0-0/
>> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-0-1/
>>
>> Thanks for the help!
>>
>> Pat
>>
>>> 1st, I don't think we need log data from November. Let's try to share
>>> logs that are limited to an example of the error you're trying to get
>>> help with. A 40Mb client log is unnecessary when it's only the last
>>> 5k that's relevant.
>>>
>>> 2nd, your "server" logs are mostly more client logs, with the
>>> exception of etc-glusterfs-glusterd.vol.log. The brick logs that
>>> might actually show what's happening are absent.
>>> (/var/log/glusterfs/bricks/*)
>>>
>>> The last line of the client log does show that
>>> gluster-0-1:/mseas-data-0-1 is reporting that it's out of space. On
>>> gluster-0-1 please share the log
>>> /var/log/glusterfs/bricks/gluster-0-1 but limit it to no more than an
>>> hour surrounding 2014-01-21 11:50:16. Also on that server, please
>>> share the result of "df /gluster-0-1 ; df -i /gluster-0-1".
>>>
>>> On 01/21/2014 10:58 PM, Pat Haley wrote:
>>>>
>>>> Hi Vijay,
>>>>
>>>> I've put the log files in
>>>>
>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/server_logs/
>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/client_logs/
>>>>
>>>>> On 01/21/2014 10:24 PM, Pat Haley wrote:
>>>>>>
>>>>>> Hi Joe,
>>>>>>
>>>>>> The peer status on all 3 showed the
>>>>>> proper connections. Doing the killall
>>>>>> and restart on all three bricks fixed
>>>>>> the N in the Online column. I then
>>>>>> did have to remount the gluster
>>>>>> filesystem on the client
>>>>>>
>>>>>> Unfortunately my original problem remains.
>>>>>> I'm still getting "no space on left on device"
>>>>>> when I try to write even a small file to
>>>>>> the gluster filesystem.
>>>>>>
>>>>>> What should I look at next?
>>>>>>
>>>>>
>>>>> Would it be possible to share the brick and client log files?
>>>>>
>>>>> Thanks,
>>>>> Vijay
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>>
>
>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: phaley at mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
More information about the Gluster-users
mailing list