[Gluster-users] Gluster not recognizing available space

Lalatendu Mohanty lmohanty at redhat.com
Tue Jan 21 15:42:50 UTC 2014


On 01/21/2014 08:34 PM, Pat Haley wrote:
>
> Also, going back to an earlier Email,
> should I be concerned that in the output
> from "gluster volume status" the
> brick "gluster-data:/data" has an "N"
> in the "Online" column?  Does this suggest
> an additional debugging route?
>

Yes, it means  brick i.e. gluster-data:/data is not reachable. Check the 
glusterfsd process in "gluster-data", it should be running .  An 
unreachable brick can be a potential issue, specially when you are using 
a distributed volume. I hope you have disabled selinux and have iptables 
rules in favor of glusterfs processes.

-Lala
> gluster volume status
> Status of volume: gdata
> Gluster process                                         Port Online  Pid
> ------------------------------------------------------------------------------ 
>
> Brick gluster-0-0:/mseas-data-0-0                       24009   Y 27006
> Brick gluster-0-1:/mseas-data-0-1                       24009 Y       
> 7063
> Brick gluster-data:/data                                24010   N 15772
> NFS Server on localhost                                 38467   Y 14936
> NFS Server on gluster-data                              38467   Y 15778
> NFS Server on gluster-0-1                               38467   Y 21083
>
>>
>> First, another update on my test of writing
>> a directory with 480 6Mb files.  Not only do
>> over 3/4 of the files appear, but the are
>> written on all 3 bricks.  Again, it is random
>> which files are not written but what I seem
>> to see is that files are written to each brick
>> even after the failures.  Does this suggest
>> anything else I should be looking at?
>>
>> As to Brian's suggestion, how exactly do I perform
>> a "quick inode allocation test"?
>>
>> Thanks
>>
>> Pat
>>
>>> On 01/17/2014 07:48 PM, Pat Haley wrote:
>>>> Hi Franco,
>>>>
>>>> I checked using df -i on all 3 bricks.  No brick is over
>>>> 1% inode usage.
>>>>
>>>
>>> It might be worth a quick inode allocation test on the fs for each
>>> brick, regardless. There are other non-obvious scenarios that can cause
>>> inode allocation to fail, at least on xfs (i.e., contiguous block
>>> allocation). Ideally, you'll have the ability to do this in a
>>> subdirectory outside the actual glusterfs brick.
>>>
>>> Brian
>>>
>>>> Thanks.
>>>>
>>>> Pat
>>>>
>>>>> Have you run out of inodes on the underlying  filesystems?
>>>>>
>>>>> On 18 Jan 2014 05:41, Pat Haley <phaley at MIT.EDU> wrote:
>>>>>
>>>>> Latest updates:
>>>>>
>>>>> no error messages were found on the log files of the bricks.
>>>>>
>>>>> The error messages appear on the client log files. Writing
>>>>> from a second client also has the same errors.
>>>>>
>>>>> Note that if I try to write a directory with 480 6Mb files
>>>>> to /projects, over 3/4 of the files are written.  It is
>>>>> random which files are not written (i.e. it is not the
>>>>> last 1/4 of the files which fail)
>>>>>
>>>>>>  Hi,
>>>>>>
>>>>>>  Some additional data
>>>>>>
>>>>>>
>>>>>>  [root at mseas-data save]# gluster volume info
>>>>>>
>>>>>>  Volume Name: gdata
>>>>>>  Type: Distribute
>>>>>>  Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d
>>>>>>  Status: Started
>>>>>>  Number of Bricks: 3
>>>>>>  Transport-type: tcp
>>>>>>  Bricks:
>>>>>>  Brick1: gluster-0-0:/mseas-data-0-0
>>>>>>  Brick2: gluster-0-1:/mseas-data-0-1
>>>>>>  Brick3: gluster-data:/data
>>>>>>
>>>>>>
>>>>>>
>>>>>>  [root at mseas-data save]# gluster volume status
>>>>>>  Status of volume: gdata
>>>>>>  Gluster process Port 
>>>>> Online  Pid
>>>>> ------------------------------------------------------------------------------ 
>>>>>
>>>>>
>>>>>>  Brick gluster-0-0:/mseas-data-0-0 24009   Y 27006
>>>>>>  Brick gluster-0-1:/mseas-data-0-1 24009  Y       7063
>>>>>>  Brick gluster-data:/data 24010  N       8007
>>>>>>  NFS Server on localhost 38467  Y       8013
>>>>>>  NFS Server on gluster-0-1 38467   Y 10228
>>>>>>  NFS Server on 10.1.1.10 38467  Y       3867
>>>>>>
>>>>>>
>>>>>>  Noticing that the brick gluster-data:/data was appearing as N
>>>>>>  in the "online" column, I tried (1) detaching gluster-data (using
>>>>>>  gluster peer detach gluster-data issued from gluster-0-0), (2) 
>>>>>> removing
>>>>>>  /var/lib/glusterd, (3) restarting glusterd on gluster-data,
>>>>>>  (4) reattaching /gluster-data (using gluster peer probe 
>>>>>> gluster-data
>>>>>>  issued from gluster-0-0) then (5) restart glusterd one more time 
>>>>>> on all
>>>>>>  3 bricks.  The brick gluster-data:/data still appears as N in the
>>>>>> Online
>>>>>>  column.
>>>>>>
>>>>>>  [root at mseas-data save]# gluster peer status
>>>>>>  Number of Peers: 2
>>>>>>
>>>>>>  Hostname: gluster-0-1
>>>>>>  Uuid: 393fc4a6-1573-4564-971e-1b1aec434167
>>>>>>  State: Peer in Cluster (Connected)
>>>>>>
>>>>>>  Hostname: 10.1.1.10
>>>>>>  Uuid: 3619440a-4ca3-4151-b62e-d4d6bf2e0c03
>>>>>>  State: Peer in Cluster (Connected)
>>>>>>
>>>>>>  (similarly from the other bricks)
>>>>>>
>>>>>>  Ping works between all bricks too.
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> We are using gluster to present 3 bricks as a single name space.
>>>>>>> We appear to have a situation in which gluster thinks there
>>>>>>> is no disk space when there is actually plenty. I have restarted
>>>>>>> the glusterd deamons on all three bricks and I still get the
>>>>>>> following message
>>>>>>>
>>>>>>> /bin/cp: cannot create regular file
>>>>>>> `./Bottom_Gravity_Current_25/344.mat': No space left on device
>>>>>>>
>>>>>>> This is a 6Mbyte file.  The total space available on
>>>>>>> gluster is 3.6T
>>>>>>>
>>>>>>> Filesystem            Size  Used Avail Use% Mounted on
>>>>>>> mseas-data:/gdata      55T   51T  3.6T  94% /gdata
>>>>>>>
>>>>>>> Also, no single brick is full:
>>>>>>>
>>>>>>> Filesystem            Size  Used Avail Use% Mounted on
>>>>>>> /dev/mapper/the_raid-lv_data
>>>>>>>                        15T   14T  804G  95% /data
>>>>>>>
>>>>>>> Filesystem            Size  Used Avail Use% Mounted on
>>>>>>> /dev/sdb1              21T   18T  2.1T  90% /mseas-data-0-0
>>>>>>>
>>>>>>> Filesystem            Size  Used Avail Use% Mounted on
>>>>>>> /dev/sdb1              21T   20T  784G  97% /mseas-data-0-1
>>>>>>>
>>>>>>> What should we do to fix this problem or look at to diagnose
>>>>>>> this problem?
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>> Pat Haley                          Email: phaley at mit.edu
>>>>>>> Center for Ocean Engineering       Phone:  (617) 253-6824
>>>>>>> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>>>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>>>>>> 77 Massachusetts Avenue
>>>>>>> Cambridge, MA  02139-4301
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>>
>>>>>
>>>>> -- 
>>>>>
>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>> Pat Haley                          Email:  phaley at mit.edu
>>>>> Center for Ocean Engineering       Phone:  (617) 253-6824
>>>>> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>>>> 77 Massachusetts Avenue
>>>>> Cambridge, MA  02139-4301
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>> ------------------------------------------------------------------------ 
>>>>>
>>>>>
>>>>>
>>>>> This email and any files transmitted with it are confidential and are
>>>>> intended solely for the use of the individual or entity to whom they
>>>>> are addressed. If you are not the original recipient or the person
>>>>> responsible for delivering the email to the intended recipient, be
>>>>> advised that you have received this email in error, and that any use,
>>>>> dissemination, forwarding, printing, or copying of this email is
>>>>> strictly prohibited. If you received this email in error, please
>>>>> immediately notify the sender and delete the original.
>>>>>
>>>>
>>>
>>
>>
>
>




More information about the Gluster-users mailing list