[Gluster-users] Gluster not recognizing available space
Brian Foster
bfoster at redhat.com
Sat Jan 18 15:10:11 UTC 2014
On 01/17/2014 07:48 PM, Pat Haley wrote:
>
> Hi Franco,
>
> I checked using df -i on all 3 bricks. No brick is over
> 1% inode usage.
>
It might be worth a quick inode allocation test on the fs for each
brick, regardless. There are other non-obvious scenarios that can cause
inode allocation to fail, at least on xfs (i.e., contiguous block
allocation). Ideally, you'll have the ability to do this in a
subdirectory outside the actual glusterfs brick.
Brian
> Thanks.
>
> Pat
>
>> Have you run out of inodes on the underlying filesystems?
>>
>> On 18 Jan 2014 05:41, Pat Haley <phaley at MIT.EDU> wrote:
>>
>> Latest updates:
>>
>> no error messages were found on the log files of the bricks.
>>
>> The error messages appear on the client log files. Writing
>> from a second client also has the same errors.
>>
>> Note that if I try to write a directory with 480 6Mb files
>> to /projects, over 3/4 of the files are written. It is
>> random which files are not written (i.e. it is not the
>> last 1/4 of the files which fail)
>>
>>>
>>> Hi,
>>>
>>> Some additional data
>>>
>>>
>>> [root at mseas-data save]# gluster volume info
>>>
>>> Volume Name: gdata
>>> Type: Distribute
>>> Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d
>>> Status: Started
>>> Number of Bricks: 3
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: gluster-0-0:/mseas-data-0-0
>>> Brick2: gluster-0-1:/mseas-data-0-1
>>> Brick3: gluster-data:/data
>>>
>>>
>>>
>>> [root at mseas-data save]# gluster volume status
>>> Status of volume: gdata
>>> Gluster process Port
>> Online Pid
>>>
>> ------------------------------------------------------------------------------
>>
>>>
>>> Brick gluster-0-0:/mseas-data-0-0 24009 Y 27006
>>> Brick gluster-0-1:/mseas-data-0-1 24009
>>> Y 7063
>>> Brick gluster-data:/data 24010
>>> N 8007
>>> NFS Server on localhost 38467
>>> Y 8013
>>> NFS Server on gluster-0-1 38467 Y 10228
>>> NFS Server on 10.1.1.10 38467
>>> Y 3867
>>>
>>>
>>> Noticing that the brick gluster-data:/data was appearing as N
>>> in the "online" column, I tried (1) detaching gluster-data (using
>>> gluster peer detach gluster-data issued from gluster-0-0), (2) removing
>>> /var/lib/glusterd, (3) restarting glusterd on gluster-data,
>>> (4) reattaching /gluster-data (using gluster peer probe gluster-data
>>> issued from gluster-0-0) then (5) restart glusterd one more time on all
>>> 3 bricks. The brick gluster-data:/data still appears as N in the
>>> Online
>>> column.
>>>
>>> [root at mseas-data save]# gluster peer status
>>> Number of Peers: 2
>>>
>>> Hostname: gluster-0-1
>>> Uuid: 393fc4a6-1573-4564-971e-1b1aec434167
>>> State: Peer in Cluster (Connected)
>>>
>>> Hostname: 10.1.1.10
>>> Uuid: 3619440a-4ca3-4151-b62e-d4d6bf2e0c03
>>> State: Peer in Cluster (Connected)
>>>
>>> (similarly from the other bricks)
>>>
>>> Ping works between all bricks too.
>>>
>>> >
>>> > Hi,
>>> >
>>> > We are using gluster to present 3 bricks as a single name space.
>>> > We appear to have a situation in which gluster thinks there
>>> > is no disk space when there is actually plenty. I have restarted
>>> > the glusterd deamons on all three bricks and I still get the
>>> > following message
>>> >
>>> > /bin/cp: cannot create regular file
>>> > `./Bottom_Gravity_Current_25/344.mat': No space left on device
>>> >
>>> > This is a 6Mbyte file. The total space available on
>>> > gluster is 3.6T
>>> >
>>> > Filesystem Size Used Avail Use% Mounted on
>>> > mseas-data:/gdata 55T 51T 3.6T 94% /gdata
>>> >
>>> > Also, no single brick is full:
>>> >
>>> > Filesystem Size Used Avail Use% Mounted on
>>> > /dev/mapper/the_raid-lv_data
>>> > 15T 14T 804G 95% /data
>>> >
>>> > Filesystem Size Used Avail Use% Mounted on
>>> > /dev/sdb1 21T 18T 2.1T 90% /mseas-data-0-0
>>> >
>>> > Filesystem Size Used Avail Use% Mounted on
>>> > /dev/sdb1 21T 20T 784G 97% /mseas-data-0-1
>>> >
>>> > What should we do to fix this problem or look at to diagnose
>>> > this problem?
>>> >
>>> > Thanks.
>>> >
>>> > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> > Pat Haley Email: phaley at mit.edu
>>> > Center for Ocean Engineering Phone: (617) 253-6824
>>> > Dept. of Mechanical Engineering Fax: (617) 253-8125
>>> > MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>> > 77 Massachusetts Avenue
>>> > Cambridge, MA 02139-4301
>>> > _______________________________________________
>>> > Gluster-users mailing list
>>> > Gluster-users at gluster.org
>>> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>
>>
>> --
>>
>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> Pat Haley Email: phaley at mit.edu
>> Center for Ocean Engineering Phone: (617) 253-6824
>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>> 77 Massachusetts Avenue
>> Cambridge, MA 02139-4301
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>> ------------------------------------------------------------------------
>>
>>
>> This email and any files transmitted with it are confidential and are
>> intended solely for the use of the individual or entity to whom they
>> are addressed. If you are not the original recipient or the person
>> responsible for delivering the email to the intended recipient, be
>> advised that you have received this email in error, and that any use,
>> dissemination, forwarding, printing, or copying of this email is
>> strictly prohibited. If you received this email in error, please
>> immediately notify the sender and delete the original.
>>
>
>
More information about the Gluster-users
mailing list