[Gluster-users] Gluster not recognizing available space

[Pat Haley]

Latest updates:

no error messages were found on the log files of the bricks.

The error messages appear on the client log files.  Writing
from a second client also has the same errors.

Note that if I try to write a directory with 480 6Mb files
to /projects, over 3/4 of the files are written.  It is
random which files are not written (i.e. it is not the
last 1/4 of the files which fail)

> 
> Hi,
> 
> Some additional data
> 
> 
> [root at mseas-data save]# gluster volume info
> 
> Volume Name: gdata
> Type: Distribute
> Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d
> Status: Started
> Number of Bricks: 3
> Transport-type: tcp
> Bricks:
> Brick1: gluster-0-0:/mseas-data-0-0
> Brick2: gluster-0-1:/mseas-data-0-1
> Brick3: gluster-data:/data
> 
> 
> 
> [root at mseas-data save]# gluster volume status
> Status of volume: gdata
> Gluster process                                         Port    Online  Pid
> ------------------------------------------------------------------------------ 
> 
> Brick gluster-0-0:/mseas-data-0-0                       24009   Y 27006
> Brick gluster-0-1:/mseas-data-0-1                       24009   Y       
> 7063
> Brick gluster-data:/data                                24010   N       
> 8007
> NFS Server on localhost                                 38467   Y       
> 8013
> NFS Server on gluster-0-1                               38467   Y 10228
> NFS Server on 10.1.1.10                                 38467   Y       
> 3867
> 
> 
> Noticing that the brick gluster-data:/data was appearing as N
> in the "online" column, I tried (1) detaching gluster-data (using
> gluster peer detach gluster-data issued from gluster-0-0), (2) removing
> /var/lib/glusterd, (3) restarting glusterd on gluster-data,
> (4) reattaching /gluster-data (using gluster peer probe gluster-data
> issued from gluster-0-0) then (5) restart glusterd one more time on all
> 3 bricks.  The brick gluster-data:/data still appears as N in the Online
> column.
> 
> [root at mseas-data save]# gluster peer status
> Number of Peers: 2
> 
> Hostname: gluster-0-1
> Uuid: 393fc4a6-1573-4564-971e-1b1aec434167
> State: Peer in Cluster (Connected)
> 
> Hostname: 10.1.1.10
> Uuid: 3619440a-4ca3-4151-b62e-d4d6bf2e0c03
> State: Peer in Cluster (Connected)
> 
> (similarly from the other bricks)
> 
> Ping works between all bricks too.
> 
>>
>> Hi,
>>
>> We are using gluster to present 3 bricks as a single name space.
>> We appear to have a situation in which gluster thinks there
>> is no disk space when there is actually plenty. I have restarted
>> the glusterd deamons on all three bricks and I still get the
>> following message
>>
>> /bin/cp: cannot create regular file 
>> `./Bottom_Gravity_Current_25/344.mat': No space left on device
>>
>> This is a 6Mbyte file.  The total space available on
>> gluster is 3.6T
>>
>> Filesystem            Size  Used Avail Use% Mounted on
>> mseas-data:/gdata      55T   51T  3.6T  94% /gdata
>>
>> Also, no single brick is full:
>>
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/mapper/the_raid-lv_data
>>                        15T   14T  804G  95% /data
>>
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/sdb1              21T   18T  2.1T  90% /mseas-data-0-0
>>
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/sdb1              21T   20T  784G  97% /mseas-data-0-1
>>
>> What should we do to fix this problem or look at to diagnose
>> this problem?
>>
>> Thanks.
>>
>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> Pat Haley                          Email:  phaley at mit.edu
>> Center for Ocean Engineering       Phone:  (617) 253-6824
>> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>> MIT, Room 5-213                    http://web.mit.edu/phaley/www/
>> 77 Massachusetts Avenue
>> Cambridge, MA  02139-4301
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
> 


-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301