[Gluster-users] Gluster not recognizing available space

Brian Foster bfoster at redhat.com
Sat Jan 18 15:10:11 UTC 2014


On 01/17/2014 07:48 PM, Pat Haley wrote:
> 
> Hi Franco,
> 
> I checked using df -i on all 3 bricks.  No brick is over
> 1% inode usage.
> 

It might be worth a quick inode allocation test on the fs for each
brick, regardless. There are other non-obvious scenarios that can cause
inode allocation to fail, at least on xfs (i.e., contiguous block
allocation). Ideally, you'll have the ability to do this in a
subdirectory outside the actual glusterfs brick.

Brian

> Thanks.
> 
> Pat
> 
>> Have you run out of inodes on the underlying  filesystems?
>>
>> On 18 Jan 2014 05:41, Pat Haley <phaley at MIT.EDU> wrote:
>>
>> Latest updates:
>>
>> no error messages were found on the log files of the bricks.
>>
>> The error messages appear on the client log files.  Writing
>> from a second client also has the same errors.
>>
>> Note that if I try to write a directory with 480 6Mb files
>> to /projects, over 3/4 of the files are written.  It is
>> random which files are not written (i.e. it is not the
>> last 1/4 of the files which fail)
>>
>>>
>>>  Hi,
>>>
>>>  Some additional data
>>>
>>>
>>>  [root at mseas-data save]# gluster volume info
>>>
>>>  Volume Name: gdata
>>>  Type: Distribute
>>>  Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d
>>>  Status: Started
>>>  Number of Bricks: 3
>>>  Transport-type: tcp
>>>  Bricks:
>>>  Brick1: gluster-0-0:/mseas-data-0-0
>>>  Brick2: gluster-0-1:/mseas-data-0-1
>>>  Brick3: gluster-data:/data
>>>
>>>
>>>
>>>  [root at mseas-data save]# gluster volume status
>>>  Status of volume: gdata
>>>  Gluster process                                         Port    
>> Online  Pid
>>>
>> ------------------------------------------------------------------------------
>>
>>>
>>>  Brick gluster-0-0:/mseas-data-0-0                       24009   Y 27006
>>>  Brick gluster-0-1:/mseas-data-0-1                       24009  
>>> Y       7063
>>>  Brick gluster-data:/data                                24010  
>>> N       8007
>>>  NFS Server on localhost                                 38467  
>>> Y       8013
>>>  NFS Server on gluster-0-1                               38467   Y 10228
>>>  NFS Server on 10.1.1.10                                 38467  
>>> Y       3867
>>>
>>>
>>>  Noticing that the brick gluster-data:/data was appearing as N
>>>  in the "online" column, I tried (1) detaching gluster-data (using
>>>  gluster peer detach gluster-data issued from gluster-0-0), (2) removing
>>>  /var/lib/glusterd, (3) restarting glusterd on gluster-data,
>>>  (4) reattaching /gluster-data (using gluster peer probe gluster-data
>>>  issued from gluster-0-0) then (5) restart glusterd one more time on all
>>>  3 bricks.  The brick gluster-data:/data still appears as N in the
>>> Online
>>>  column.
>>>
>>>  [root at mseas-data save]# gluster peer status
>>>  Number of Peers: 2
>>>
>>>  Hostname: gluster-0-1
>>>  Uuid: 393fc4a6-1573-4564-971e-1b1aec434167
>>>  State: Peer in Cluster (Connected)
>>>
>>>  Hostname: 10.1.1.10
>>>  Uuid: 3619440a-4ca3-4151-b62e-d4d6bf2e0c03
>>>  State: Peer in Cluster (Connected)
>>>
>>>  (similarly from the other bricks)
>>>
>>>  Ping works between all bricks too.
>>>
>>> >
>>> > Hi,
>>> >
>>> > We are using gluster to present 3 bricks as a single name space.
>>> > We appear to have a situation in which gluster thinks there
>>> > is no disk space when there is actually plenty. I have restarted
>>> > the glusterd deamons on all three bricks and I still get the
>>> > following message
>>> >
>>> > /bin/cp: cannot create regular file
>>> > `./Bottom_Gravity_Current_25/344.mat': No space left on device
>>> >
>>> > This is a 6Mbyte file.  The total space available on
>>> > gluster is 3.6T
>>> >
>>> > Filesystem            Size  Used Avail Use% Mounted on
>>> > mseas-data:/gdata      55T   51T  3.6T  94% /gdata
>>> >
>>> > Also, no single brick is full:
>>> >
>>> > Filesystem            Size  Used Avail Use% Mounted on
>>> > /dev/mapper/the_raid-lv_data
>>> >                        15T   14T  804G  95% /data
>>> >
>>> > Filesystem            Size  Used Avail Use% Mounted on
>>> > /dev/sdb1              21T   18T  2.1T  90% /mseas-data-0-0
>>> >
>>> > Filesystem            Size  Used Avail Use% Mounted on
>>> > /dev/sdb1              21T   20T  784G  97% /mseas-data-0-1
>>> >
>>> > What should we do to fix this problem or look at to diagnose
>>> > this problem?
>>> >
>>> > Thanks.
>>> >
>>> > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> > Pat Haley                          Email:  phaley at mit.edu
>>> > Center for Ocean Engineering       Phone:  (617) 253-6824
>>> > Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>> > MIT, Room 5-213                    http://web.mit.edu/phaley/www/
>>> > 77 Massachusetts Avenue
>>> > Cambridge, MA  02139-4301
>>> > _______________________________________________
>>> > Gluster-users mailing list
>>> > Gluster-users at gluster.org
>>> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>
>>
>> -- 
>>
>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> Pat Haley                          Email:  phaley at mit.edu
>> Center for Ocean Engineering       Phone:  (617) 253-6824
>> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>> MIT, Room 5-213                    http://web.mit.edu/phaley/www/
>> 77 Massachusetts Avenue
>> Cambridge, MA  02139-4301
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>> ------------------------------------------------------------------------
>>
>>
>> This email and any files transmitted with it are confidential and are
>> intended solely for the use of the individual or entity to whom they
>> are addressed. If you are not the original recipient or the person
>> responsible for delivering the email to the intended recipient, be
>> advised that you have received this email in error, and that any use,
>> dissemination, forwarding, printing, or copying of this email is
>> strictly prohibited. If you received this email in error, please
>> immediately notify the sender and delete the original.
>>
> 
> 




More information about the Gluster-users mailing list