[Gluster-users] Gluster not recognizing available space

Pat Haley phaley at MIT.EDU
Fri Jan 31 15:43:04 UTC 2014


Hi,

A tiny bit more data:

I tried restarting nfs on gluster-0-1.  Following
that I was back to the original behavior of getting
"No space left on device" error messages when I tried
to write from a client.  I then deleted the dummy test
directories I had made yesterday, followed by trying
to recreate them.  As yesterday the first directory
is created on all bricks, although the log file on
gluster-0-1 has the same error messages as yesterday.
Again as yesterday from the client side the second
directory seems to be created successfully but no
such directory appears on gluster-0-1.

Does any of this suggest where I should look next?

Thanks.

Pat


> 
> Hi,
> 
> I now have a change in the behavior of the gluster issue.
> This afternoon I deleted a file that I knew was located
> on the gluster-0-1 disk (the deleting command was issued
> from a client).  After that I was able to make a directoy
> (empty) that appeared on all 3 bricks.  My next step was
> to copy a directory with files to the /gdata directory.
> That appeared to be successful, but I looked and nothing
> was written to the gluster-0-1 brick, not even the
> new directory.  That seems to be the new state, the client
> still sees files on gluster-0-1 but when it writes to the
> /gdata area it only writes to the gluster-data and gluster-0-0
> bricks.
> 
>  From the bricks log on gluster-0-1 I have the following lines
> from when I successfully created the test directory
> 
> [2014-01-30 14:26:55.176965] W [posix-handle.c:538:posix_handle_soft] 
> 0-gdata-posix: symlink 
> ../../c9/14/c9144c1f-01a2-41ba-80cc-deec9aac03c5/test -> 
> /mseas-data-0-1/.glusterfs/d9/b8/d9b84fcb-a452-4f4e-bdd9-ba1a392a3844 
> failed (No space left on device)
> [2014-01-30 14:26:55.205253] E [posix.c:969:posix_mkdir] 0-gdata-posix: 
> setting gfid on /mseas-data-0-1/projects/atl/test failed
> 
> Then when I tried copying a full directory
> 
> [2014-01-30 14:30:35.716046] E [posix.c:962:posix_mkdir] 0-gdata-posix: 
> mkdir of 
> /mseas-data-0-1/projects/atl/FV_Matlab_Framework_JingPJH/trunk/Save/mat/Bottom_Gravity_Current_28 
> failed: No space left on device
> [2014-01-30 14:30:35.716133] I [server3_1-fops.c:529:server_mkdir_cbk] 
> 0-gdata-server: 19027: MKDIR 
> /projects/atl/FV_Matlab_Framework_JingPJH/trunk/Save/mat/Bottom_Gravity_Current_28 
> (00000000-0000-0000-0000-000000000000) ==> -1 (No space left on device)
> [2014-01-30 14:30:37.105372] I 
> [server3_1-fops.c:1747:server_setattr_cbk] 0-gdata-server: 19029: 
> SETATTR (null) (--) ==> -1 (No such file or directory)
> 
>    That last line is repeated many times, followed by
> 
> [2014-01-30 14:31:57.587539] I 
> [server3_1-fops.c:1747:server_setattr_cbk] 0-gdata-server: 22077: 
> SETATTR (null) (--) ==> -1 (No such file or directory)
> 
> Does this suggest anything?  Is there additional information
> I should post?
> 
> Thanks,
> 
> Pat
> 
> 
>>
>> Hi Joe,
>>
>> Sorry to take so long in responding, but we had another emergency
>> that took all my time...
>>
>> The subsampled brick log file from gluster-0-1 is available at
>>
>> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-0-1/bricks/mseas-data-0-1.log.1 
>>
>>
>> The df results on gluster-0-1 are
>>
>> [root at nas-0-1 glusterfs]# df /mseas-data-0-1/
>> Filesystem           1K-blocks      Used Available Use% Mounted on
>> /dev/sdb1            21484242304 20662833060 821409244  97% 
>> /mseas-data-0-1
>>
>> [root at nas-0-1 glusterfs]# df -i /mseas-data-0-1/
>> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
>> /dev/sdb1            3290028528 4391552 3285636976    1% /mseas-data-0-1
>>
>> I've also included the same time period log files from gluster-0-1 and
>> the other bricks in
>>
>> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-data/
>> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-0-0/
>> http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-0-1/
>>
>> Thanks for the help!
>>
>> Pat
>>
>>> 1st, I don't think we need log data from November. Let's try to share 
>>> logs that are limited to an example of the error you're trying to get 
>>> help with. A 40Mb client log is unnecessary when it's only the last 
>>> 5k that's relevant.
>>>
>>> 2nd, your "server" logs are mostly more client logs, with the 
>>> exception of etc-glusterfs-glusterd.vol.log. The brick logs that 
>>> might actually show what's happening are absent. 
>>> (/var/log/glusterfs/bricks/*)
>>>
>>> The last line of the client log does show that 
>>> gluster-0-1:/mseas-data-0-1 is reporting that it's out of space. On 
>>> gluster-0-1 please share the log 
>>> /var/log/glusterfs/bricks/gluster-0-1 but limit it to no more than an 
>>> hour surrounding 2014-01-21 11:50:16. Also on that server, please 
>>> share the result of "df /gluster-0-1 ; df -i /gluster-0-1".
>>>
>>> On 01/21/2014 10:58 PM, Pat Haley wrote:
>>>>
>>>> Hi Vijay,
>>>>
>>>> I've put the log files in
>>>>
>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/server_logs/
>>>> http://mseas.mit.edu/download/phaley/GlusterUsers/client_logs/
>>>>
>>>>> On 01/21/2014 10:24 PM, Pat Haley wrote:
>>>>>>
>>>>>> Hi Joe,
>>>>>>
>>>>>> The peer status on all 3 showed the
>>>>>> proper connections.  Doing the killall
>>>>>> and restart on all three bricks fixed
>>>>>> the N in the Online column.  I then
>>>>>> did have to remount the gluster
>>>>>> filesystem on the client
>>>>>>
>>>>>> Unfortunately my original problem remains.
>>>>>> I'm still getting "no space on left on device"
>>>>>> when I try to write even a small file to
>>>>>> the gluster filesystem.
>>>>>>
>>>>>> What should I look at next?
>>>>>>
>>>>>
>>>>> Would it be possible to share the brick and client log files?
>>>>>
>>>>> Thanks,
>>>>> Vijay
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>>
> 
> 


-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301



More information about the Gluster-users mailing list