[Gluster-users] Gluster not recognizing available space
Pat Haley
phaley at MIT.EDU
Tue Jan 21 15:04:49 UTC 2014
Also, going back to an earlier Email,
should I be concerned that in the output
from "gluster volume status" the
brick "gluster-data:/data" has an "N"
in the "Online" column? Does this suggest
an additional debugging route?
gluster volume status
Status of volume: gdata
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick gluster-0-0:/mseas-data-0-0 24009 Y
27006
Brick gluster-0-1:/mseas-data-0-1 24009 Y 7063
Brick gluster-data:/data 24010 N
15772
NFS Server on localhost 38467 Y
14936
NFS Server on gluster-data 38467 Y
15778
NFS Server on gluster-0-1 38467 Y
21083
>
> First, another update on my test of writing
> a directory with 480 6Mb files. Not only do
> over 3/4 of the files appear, but the are
> written on all 3 bricks. Again, it is random
> which files are not written but what I seem
> to see is that files are written to each brick
> even after the failures. Does this suggest
> anything else I should be looking at?
>
> As to Brian's suggestion, how exactly do I perform
> a "quick inode allocation test"?
>
> Thanks
>
> Pat
>
>> On 01/17/2014 07:48 PM, Pat Haley wrote:
>>> Hi Franco,
>>>
>>> I checked using df -i on all 3 bricks. No brick is over
>>> 1% inode usage.
>>>
>>
>> It might be worth a quick inode allocation test on the fs for each
>> brick, regardless. There are other non-obvious scenarios that can cause
>> inode allocation to fail, at least on xfs (i.e., contiguous block
>> allocation). Ideally, you'll have the ability to do this in a
>> subdirectory outside the actual glusterfs brick.
>>
>> Brian
>>
>>> Thanks.
>>>
>>> Pat
>>>
>>>> Have you run out of inodes on the underlying filesystems?
>>>>
>>>> On 18 Jan 2014 05:41, Pat Haley <phaley at MIT.EDU> wrote:
>>>>
>>>> Latest updates:
>>>>
>>>> no error messages were found on the log files of the bricks.
>>>>
>>>> The error messages appear on the client log files. Writing
>>>> from a second client also has the same errors.
>>>>
>>>> Note that if I try to write a directory with 480 6Mb files
>>>> to /projects, over 3/4 of the files are written. It is
>>>> random which files are not written (i.e. it is not the
>>>> last 1/4 of the files which fail)
>>>>
>>>>> Hi,
>>>>>
>>>>> Some additional data
>>>>>
>>>>>
>>>>> [root at mseas-data save]# gluster volume info
>>>>>
>>>>> Volume Name: gdata
>>>>> Type: Distribute
>>>>> Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d
>>>>> Status: Started
>>>>> Number of Bricks: 3
>>>>> Transport-type: tcp
>>>>> Bricks:
>>>>> Brick1: gluster-0-0:/mseas-data-0-0
>>>>> Brick2: gluster-0-1:/mseas-data-0-1
>>>>> Brick3: gluster-data:/data
>>>>>
>>>>>
>>>>>
>>>>> [root at mseas-data save]# gluster volume status
>>>>> Status of volume: gdata
>>>>> Gluster process Port
>>>> Online Pid
>>>> ------------------------------------------------------------------------------
>>>>
>>>>
>>>>> Brick gluster-0-0:/mseas-data-0-0 24009 Y
>>>>> 27006
>>>>> Brick gluster-0-1:/mseas-data-0-1 24009
>>>>> Y 7063
>>>>> Brick gluster-data:/data 24010
>>>>> N 8007
>>>>> NFS Server on localhost 38467
>>>>> Y 8013
>>>>> NFS Server on gluster-0-1 38467 Y
>>>>> 10228
>>>>> NFS Server on 10.1.1.10 38467
>>>>> Y 3867
>>>>>
>>>>>
>>>>> Noticing that the brick gluster-data:/data was appearing as N
>>>>> in the "online" column, I tried (1) detaching gluster-data (using
>>>>> gluster peer detach gluster-data issued from gluster-0-0), (2)
>>>>> removing
>>>>> /var/lib/glusterd, (3) restarting glusterd on gluster-data,
>>>>> (4) reattaching /gluster-data (using gluster peer probe gluster-data
>>>>> issued from gluster-0-0) then (5) restart glusterd one more time
>>>>> on all
>>>>> 3 bricks. The brick gluster-data:/data still appears as N in the
>>>>> Online
>>>>> column.
>>>>>
>>>>> [root at mseas-data save]# gluster peer status
>>>>> Number of Peers: 2
>>>>>
>>>>> Hostname: gluster-0-1
>>>>> Uuid: 393fc4a6-1573-4564-971e-1b1aec434167
>>>>> State: Peer in Cluster (Connected)
>>>>>
>>>>> Hostname: 10.1.1.10
>>>>> Uuid: 3619440a-4ca3-4151-b62e-d4d6bf2e0c03
>>>>> State: Peer in Cluster (Connected)
>>>>>
>>>>> (similarly from the other bricks)
>>>>>
>>>>> Ping works between all bricks too.
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We are using gluster to present 3 bricks as a single name space.
>>>>>> We appear to have a situation in which gluster thinks there
>>>>>> is no disk space when there is actually plenty. I have restarted
>>>>>> the glusterd deamons on all three bricks and I still get the
>>>>>> following message
>>>>>>
>>>>>> /bin/cp: cannot create regular file
>>>>>> `./Bottom_Gravity_Current_25/344.mat': No space left on device
>>>>>>
>>>>>> This is a 6Mbyte file. The total space available on
>>>>>> gluster is 3.6T
>>>>>>
>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>> mseas-data:/gdata 55T 51T 3.6T 94% /gdata
>>>>>>
>>>>>> Also, no single brick is full:
>>>>>>
>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>> /dev/mapper/the_raid-lv_data
>>>>>> 15T 14T 804G 95% /data
>>>>>>
>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>> /dev/sdb1 21T 18T 2.1T 90% /mseas-data-0-0
>>>>>>
>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>> /dev/sdb1 21T 20T 784G 97% /mseas-data-0-1
>>>>>>
>>>>>> What should we do to fix this problem or look at to diagnose
>>>>>> this problem?
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>> Pat Haley Email: phaley at mit.edu
>>>>>> Center for Ocean Engineering Phone: (617) 253-6824
>>>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>>>>> 77 Massachusetts Avenue
>>>>>> Cambridge, MA 02139-4301
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>
>>>> --
>>>>
>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>> Pat Haley Email: phaley at mit.edu
>>>> Center for Ocean Engineering Phone: (617) 253-6824
>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>>> 77 Massachusetts Avenue
>>>> Cambridge, MA 02139-4301
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>>
>>>>
>>>> This email and any files transmitted with it are confidential and are
>>>> intended solely for the use of the individual or entity to whom they
>>>> are addressed. If you are not the original recipient or the person
>>>> responsible for delivering the email to the intended recipient, be
>>>> advised that you have received this email in error, and that any use,
>>>> dissemination, forwarding, printing, or copying of this email is
>>>> strictly prohibited. If you received this email in error, please
>>>> immediately notify the sender and delete the original.
>>>>
>>>
>>
>
>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: phaley at mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
More information about the Gluster-users
mailing list