[Gluster-users] Gluster not recognizing available space
Pat Haley
phaley at MIT.EDU
Sat Jan 18 00:48:29 UTC 2014
Hi Franco,
I checked using df -i on all 3 bricks. No brick is over
1% inode usage.
Thanks.
Pat
> Have you run out of inodes on the underlying filesystems?
>
> On 18 Jan 2014 05:41, Pat Haley <phaley at MIT.EDU> wrote:
>
> Latest updates:
>
> no error messages were found on the log files of the bricks.
>
> The error messages appear on the client log files. Writing
> from a second client also has the same errors.
>
> Note that if I try to write a directory with 480 6Mb files
> to /projects, over 3/4 of the files are written. It is
> random which files are not written (i.e. it is not the
> last 1/4 of the files which fail)
>
>>
>> Hi,
>>
>> Some additional data
>>
>>
>> [root at mseas-data save]# gluster volume info
>>
>> Volume Name: gdata
>> Type: Distribute
>> Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d
>> Status: Started
>> Number of Bricks: 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: gluster-0-0:/mseas-data-0-0
>> Brick2: gluster-0-1:/mseas-data-0-1
>> Brick3: gluster-data:/data
>>
>>
>>
>> [root at mseas-data save]# gluster volume status
>> Status of volume: gdata
>> Gluster process Port
> Online Pid
>>
> ------------------------------------------------------------------------------
>
>>
>> Brick gluster-0-0:/mseas-data-0-0 24009 Y 27006
>> Brick gluster-0-1:/mseas-data-0-1 24009 Y
>> 7063
>> Brick gluster-data:/data 24010 N
>> 8007
>> NFS Server on localhost 38467 Y
>> 8013
>> NFS Server on gluster-0-1 38467 Y 10228
>> NFS Server on 10.1.1.10 38467 Y
>> 3867
>>
>>
>> Noticing that the brick gluster-data:/data was appearing as N
>> in the "online" column, I tried (1) detaching gluster-data (using
>> gluster peer detach gluster-data issued from gluster-0-0), (2) removing
>> /var/lib/glusterd, (3) restarting glusterd on gluster-data,
>> (4) reattaching /gluster-data (using gluster peer probe gluster-data
>> issued from gluster-0-0) then (5) restart glusterd one more time on all
>> 3 bricks. The brick gluster-data:/data still appears as N in the Online
>> column.
>>
>> [root at mseas-data save]# gluster peer status
>> Number of Peers: 2
>>
>> Hostname: gluster-0-1
>> Uuid: 393fc4a6-1573-4564-971e-1b1aec434167
>> State: Peer in Cluster (Connected)
>>
>> Hostname: 10.1.1.10
>> Uuid: 3619440a-4ca3-4151-b62e-d4d6bf2e0c03
>> State: Peer in Cluster (Connected)
>>
>> (similarly from the other bricks)
>>
>> Ping works between all bricks too.
>>
>> >
>> > Hi,
>> >
>> > We are using gluster to present 3 bricks as a single name space.
>> > We appear to have a situation in which gluster thinks there
>> > is no disk space when there is actually plenty. I have restarted
>> > the glusterd deamons on all three bricks and I still get the
>> > following message
>> >
>> > /bin/cp: cannot create regular file
>> > `./Bottom_Gravity_Current_25/344.mat': No space left on device
>> >
>> > This is a 6Mbyte file. The total space available on
>> > gluster is 3.6T
>> >
>> > Filesystem Size Used Avail Use% Mounted on
>> > mseas-data:/gdata 55T 51T 3.6T 94% /gdata
>> >
>> > Also, no single brick is full:
>> >
>> > Filesystem Size Used Avail Use% Mounted on
>> > /dev/mapper/the_raid-lv_data
>> > 15T 14T 804G 95% /data
>> >
>> > Filesystem Size Used Avail Use% Mounted on
>> > /dev/sdb1 21T 18T 2.1T 90% /mseas-data-0-0
>> >
>> > Filesystem Size Used Avail Use% Mounted on
>> > /dev/sdb1 21T 20T 784G 97% /mseas-data-0-1
>> >
>> > What should we do to fix this problem or look at to diagnose
>> > this problem?
>> >
>> > Thanks.
>> >
>> > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> > Pat Haley Email: phaley at mit.edu
>> > Center for Ocean Engineering Phone: (617) 253-6824
>> > Dept. of Mechanical Engineering Fax: (617) 253-8125
>> > MIT, Room 5-213 http://web.mit.edu/phaley/www/
>> > 77 Massachusetts Avenue
>> > Cambridge, MA 02139-4301
>> > _______________________________________________
>> > Gluster-users mailing list
>> > Gluster-users at gluster.org
>> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>>
>
>
> --
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley Email: phaley at mit.edu
> Center for Ocean Engineering Phone: (617) 253-6824
> Dept. of Mechanical Engineering Fax: (617) 253-8125
> MIT, Room 5-213 http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA 02139-4301
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
> ------------------------------------------------------------------------
>
>
> This email and any files transmitted with it are confidential and are
> intended solely for the use of the individual or entity to whom they are
> addressed. If you are not the original recipient or the person
> responsible for delivering the email to the intended recipient, be
> advised that you have received this email in error, and that any use,
> dissemination, forwarding, printing, or copying of this email is
> strictly prohibited. If you received this email in error, please
> immediately notify the sender and delete the original.
>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: phaley at mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
More information about the Gluster-users
mailing list