[Gluster-users] Gluster not recognizing available space

Pat Haley phaley at MIT.EDU
Tue Jan 21 14:37:58 UTC 2014


First, another update on my test of writing
a directory with 480 6Mb files.  Not only do
over 3/4 of the files appear, but the are
written on all 3 bricks.  Again, it is random
which files are not written but what I seem
to see is that files are written to each brick
even after the failures.  Does this suggest
anything else I should be looking at?

As to Brian's suggestion, how exactly do I perform
a "quick inode allocation test"?

Thanks

Pat

> On 01/17/2014 07:48 PM, Pat Haley wrote:
>> Hi Franco,
>>
>> I checked using df -i on all 3 bricks.  No brick is over
>> 1% inode usage.
>>
> 
> It might be worth a quick inode allocation test on the fs for each
> brick, regardless. There are other non-obvious scenarios that can cause
> inode allocation to fail, at least on xfs (i.e., contiguous block
> allocation). Ideally, you'll have the ability to do this in a
> subdirectory outside the actual glusterfs brick.
> 
> Brian
> 
>> Thanks.
>>
>> Pat
>>
>>> Have you run out of inodes on the underlying  filesystems?
>>>
>>> On 18 Jan 2014 05:41, Pat Haley <phaley at MIT.EDU> wrote:
>>>
>>> Latest updates:
>>>
>>> no error messages were found on the log files of the bricks.
>>>
>>> The error messages appear on the client log files.  Writing
>>> from a second client also has the same errors.
>>>
>>> Note that if I try to write a directory with 480 6Mb files
>>> to /projects, over 3/4 of the files are written.  It is
>>> random which files are not written (i.e. it is not the
>>> last 1/4 of the files which fail)
>>>
>>>>  Hi,
>>>>
>>>>  Some additional data
>>>>
>>>>
>>>>  [root at mseas-data save]# gluster volume info
>>>>
>>>>  Volume Name: gdata
>>>>  Type: Distribute
>>>>  Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d
>>>>  Status: Started
>>>>  Number of Bricks: 3
>>>>  Transport-type: tcp
>>>>  Bricks:
>>>>  Brick1: gluster-0-0:/mseas-data-0-0
>>>>  Brick2: gluster-0-1:/mseas-data-0-1
>>>>  Brick3: gluster-data:/data
>>>>
>>>>
>>>>
>>>>  [root at mseas-data save]# gluster volume status
>>>>  Status of volume: gdata
>>>>  Gluster process                                         Port    
>>> Online  Pid
>>> ------------------------------------------------------------------------------
>>>
>>>>  Brick gluster-0-0:/mseas-data-0-0                       24009   Y 27006
>>>>  Brick gluster-0-1:/mseas-data-0-1                       24009  
>>>> Y       7063
>>>>  Brick gluster-data:/data                                24010  
>>>> N       8007
>>>>  NFS Server on localhost                                 38467  
>>>> Y       8013
>>>>  NFS Server on gluster-0-1                               38467   Y 10228
>>>>  NFS Server on 10.1.1.10                                 38467  
>>>> Y       3867
>>>>
>>>>
>>>>  Noticing that the brick gluster-data:/data was appearing as N
>>>>  in the "online" column, I tried (1) detaching gluster-data (using
>>>>  gluster peer detach gluster-data issued from gluster-0-0), (2) removing
>>>>  /var/lib/glusterd, (3) restarting glusterd on gluster-data,
>>>>  (4) reattaching /gluster-data (using gluster peer probe gluster-data
>>>>  issued from gluster-0-0) then (5) restart glusterd one more time on all
>>>>  3 bricks.  The brick gluster-data:/data still appears as N in the
>>>> Online
>>>>  column.
>>>>
>>>>  [root at mseas-data save]# gluster peer status
>>>>  Number of Peers: 2
>>>>
>>>>  Hostname: gluster-0-1
>>>>  Uuid: 393fc4a6-1573-4564-971e-1b1aec434167
>>>>  State: Peer in Cluster (Connected)
>>>>
>>>>  Hostname: 10.1.1.10
>>>>  Uuid: 3619440a-4ca3-4151-b62e-d4d6bf2e0c03
>>>>  State: Peer in Cluster (Connected)
>>>>
>>>>  (similarly from the other bricks)
>>>>
>>>>  Ping works between all bricks too.
>>>>
>>>>> Hi,
>>>>>
>>>>> We are using gluster to present 3 bricks as a single name space.
>>>>> We appear to have a situation in which gluster thinks there
>>>>> is no disk space when there is actually plenty. I have restarted
>>>>> the glusterd deamons on all three bricks and I still get the
>>>>> following message
>>>>>
>>>>> /bin/cp: cannot create regular file
>>>>> `./Bottom_Gravity_Current_25/344.mat': No space left on device
>>>>>
>>>>> This is a 6Mbyte file.  The total space available on
>>>>> gluster is 3.6T
>>>>>
>>>>> Filesystem            Size  Used Avail Use% Mounted on
>>>>> mseas-data:/gdata      55T   51T  3.6T  94% /gdata
>>>>>
>>>>> Also, no single brick is full:
>>>>>
>>>>> Filesystem            Size  Used Avail Use% Mounted on
>>>>> /dev/mapper/the_raid-lv_data
>>>>>                        15T   14T  804G  95% /data
>>>>>
>>>>> Filesystem            Size  Used Avail Use% Mounted on
>>>>> /dev/sdb1              21T   18T  2.1T  90% /mseas-data-0-0
>>>>>
>>>>> Filesystem            Size  Used Avail Use% Mounted on
>>>>> /dev/sdb1              21T   20T  784G  97% /mseas-data-0-1
>>>>>
>>>>> What should we do to fix this problem or look at to diagnose
>>>>> this problem?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>> Pat Haley                          Email:  phaley at mit.edu
>>>>> Center for Ocean Engineering       Phone:  (617) 253-6824
>>>>> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>>>> MIT, Room 5-213                    http://web.mit.edu/phaley/www/
>>>>> 77 Massachusetts Avenue
>>>>> Cambridge, MA  02139-4301
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>
>>> -- 
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> Pat Haley                          Email:  phaley at mit.edu
>>> Center for Ocean Engineering       Phone:  (617) 253-6824
>>> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>> MIT, Room 5-213                    http://web.mit.edu/phaley/www/
>>> 77 Massachusetts Avenue
>>> Cambridge, MA  02139-4301
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>> This email and any files transmitted with it are confidential and are
>>> intended solely for the use of the individual or entity to whom they
>>> are addressed. If you are not the original recipient or the person
>>> responsible for delivering the email to the intended recipient, be
>>> advised that you have received this email in error, and that any use,
>>> dissemination, forwarding, printing, or copying of this email is
>>> strictly prohibited. If you received this email in error, please
>>> immediately notify the sender and delete the original.
>>>
>>
> 


-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301



More information about the Gluster-users mailing list