[Gluster-users] Gluster not recognizing available space
Pat Haley
phaley at MIT.EDU
Tue Jan 21 15:52:01 UTC 2014
Hi Lala,
The glusterfsd process is running (see below). I also tried
"service iptables stop" (followed by restarting
gluserd) but still have the N in the OnLine column.
What should I look at next?
Thanks
# ps aux | grep gluster
root 2916 1.4 1.2 4442964 105740 ? Ssl 2013 870:40
/usr/sbin/glusterfsd -s localhost --volfile-id gdata.gluster-data.data
-p /var/lib/glusterd/vols/gdata/run/gluster-data-data.pid -S
/tmp/e3eac7ce95e786a3d909b8fc65ed2059.socket --brick-name /data -l
/var/log/glusterfs/bricks/data.log --xlator-option
*-posix.glusterd-uuid=e1bd42a5-9061-4930-8fe3-17e8e11e3e52 --brick-port
24009 --xlator-option gdata-server.listen-port=24009
root 3882 0.0 0.4 201724 38220 ? Ssl 2013 0:00
/usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p
/var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S
/tmp/1ed43f585eab7d47ff1383c7dbf3e9bb.socket
root 7623 0.0 0.4 136192 38180 ? Ssl Jan17 0:00
/usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p
/var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S
/tmp/ef02475524de51574a2984c0b399bfd6.socket
root 15778 0.0 0.4 136180 38180 ? Ssl 09:54 0:00
/usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p
/var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S
/tmp/a981bce11177b462b2335098e6498145.socket
root 17119 0.6 0.1 110640 10820 ? Ssl 10:48 0:00
/usr/sbin/glusterd -p /var/run/glusterd.pid
root 17145 0.0 0.4 136192 38168 ? Ssl 10:48 0:00
/usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p
/var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S
/tmp/43428f984ba9bca411868252631f17bc.socket
root 17206 0.0 0.0 61192 752 pts/1 S+ 10:49 0:00 grep
gluster
> On 01/21/2014 08:34 PM, Pat Haley wrote:
>>
>> Also, going back to an earlier Email,
>> should I be concerned that in the output
>> from "gluster volume status" the
>> brick "gluster-data:/data" has an "N"
>> in the "Online" column? Does this suggest
>> an additional debugging route?
>>
>
> Yes, it means brick i.e. gluster-data:/data is not reachable. Check the
> glusterfsd process in "gluster-data", it should be running . An
> unreachable brick can be a potential issue, specially when you are using
> a distributed volume. I hope you have disabled selinux and have iptables
> rules in favor of glusterfs processes.
>
> -Lala
>> gluster volume status
>> Status of volume: gdata
>> Gluster process Port Online Pid
>> ------------------------------------------------------------------------------
>>
>> Brick gluster-0-0:/mseas-data-0-0 24009 Y 27006
>> Brick gluster-0-1:/mseas-data-0-1 24009 Y
>> 7063
>> Brick gluster-data:/data 24010 N 15772
>> NFS Server on localhost 38467 Y 14936
>> NFS Server on gluster-data 38467 Y 15778
>> NFS Server on gluster-0-1 38467 Y 21083
>>
>>>
>>> First, another update on my test of writing
>>> a directory with 480 6Mb files. Not only do
>>> over 3/4 of the files appear, but the are
>>> written on all 3 bricks. Again, it is random
>>> which files are not written but what I seem
>>> to see is that files are written to each brick
>>> even after the failures. Does this suggest
>>> anything else I should be looking at?
>>>
>>> As to Brian's suggestion, how exactly do I perform
>>> a "quick inode allocation test"?
>>>
>>> Thanks
>>>
>>> Pat
>>>
>>>> On 01/17/2014 07:48 PM, Pat Haley wrote:
>>>>> Hi Franco,
>>>>>
>>>>> I checked using df -i on all 3 bricks. No brick is over
>>>>> 1% inode usage.
>>>>>
>>>>
>>>> It might be worth a quick inode allocation test on the fs for each
>>>> brick, regardless. There are other non-obvious scenarios that can cause
>>>> inode allocation to fail, at least on xfs (i.e., contiguous block
>>>> allocation). Ideally, you'll have the ability to do this in a
>>>> subdirectory outside the actual glusterfs brick.
>>>>
>>>> Brian
>>>>
>>>>> Thanks.
>>>>>
>>>>> Pat
>>>>>
>>>>>> Have you run out of inodes on the underlying filesystems?
>>>>>>
>>>>>> On 18 Jan 2014 05:41, Pat Haley <phaley at MIT.EDU> wrote:
>>>>>>
>>>>>> Latest updates:
>>>>>>
>>>>>> no error messages were found on the log files of the bricks.
>>>>>>
>>>>>> The error messages appear on the client log files. Writing
>>>>>> from a second client also has the same errors.
>>>>>>
>>>>>> Note that if I try to write a directory with 480 6Mb files
>>>>>> to /projects, over 3/4 of the files are written. It is
>>>>>> random which files are not written (i.e. it is not the
>>>>>> last 1/4 of the files which fail)
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Some additional data
>>>>>>>
>>>>>>>
>>>>>>> [root at mseas-data save]# gluster volume info
>>>>>>>
>>>>>>> Volume Name: gdata
>>>>>>> Type: Distribute
>>>>>>> Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d
>>>>>>> Status: Started
>>>>>>> Number of Bricks: 3
>>>>>>> Transport-type: tcp
>>>>>>> Bricks:
>>>>>>> Brick1: gluster-0-0:/mseas-data-0-0
>>>>>>> Brick2: gluster-0-1:/mseas-data-0-1
>>>>>>> Brick3: gluster-data:/data
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [root at mseas-data save]# gluster volume status
>>>>>>> Status of volume: gdata
>>>>>>> Gluster process Port
>>>>>> Online Pid
>>>>>> ------------------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>>> Brick gluster-0-0:/mseas-data-0-0 24009 Y 27006
>>>>>>> Brick gluster-0-1:/mseas-data-0-1 24009 Y 7063
>>>>>>> Brick gluster-data:/data 24010 N 8007
>>>>>>> NFS Server on localhost 38467 Y 8013
>>>>>>> NFS Server on gluster-0-1 38467 Y 10228
>>>>>>> NFS Server on 10.1.1.10 38467 Y 3867
>>>>>>>
>>>>>>>
>>>>>>> Noticing that the brick gluster-data:/data was appearing as N
>>>>>>> in the "online" column, I tried (1) detaching gluster-data (using
>>>>>>> gluster peer detach gluster-data issued from gluster-0-0), (2)
>>>>>>> removing
>>>>>>> /var/lib/glusterd, (3) restarting glusterd on gluster-data,
>>>>>>> (4) reattaching /gluster-data (using gluster peer probe
>>>>>>> gluster-data
>>>>>>> issued from gluster-0-0) then (5) restart glusterd one more time
>>>>>>> on all
>>>>>>> 3 bricks. The brick gluster-data:/data still appears as N in the
>>>>>>> Online
>>>>>>> column.
>>>>>>>
>>>>>>> [root at mseas-data save]# gluster peer status
>>>>>>> Number of Peers: 2
>>>>>>>
>>>>>>> Hostname: gluster-0-1
>>>>>>> Uuid: 393fc4a6-1573-4564-971e-1b1aec434167
>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>
>>>>>>> Hostname: 10.1.1.10
>>>>>>> Uuid: 3619440a-4ca3-4151-b62e-d4d6bf2e0c03
>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>
>>>>>>> (similarly from the other bricks)
>>>>>>>
>>>>>>> Ping works between all bricks too.
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> We are using gluster to present 3 bricks as a single name space.
>>>>>>>> We appear to have a situation in which gluster thinks there
>>>>>>>> is no disk space when there is actually plenty. I have restarted
>>>>>>>> the glusterd deamons on all three bricks and I still get the
>>>>>>>> following message
>>>>>>>>
>>>>>>>> /bin/cp: cannot create regular file
>>>>>>>> `./Bottom_Gravity_Current_25/344.mat': No space left on device
>>>>>>>>
>>>>>>>> This is a 6Mbyte file. The total space available on
>>>>>>>> gluster is 3.6T
>>>>>>>>
>>>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>>>> mseas-data:/gdata 55T 51T 3.6T 94% /gdata
>>>>>>>>
>>>>>>>> Also, no single brick is full:
>>>>>>>>
>>>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>>>> /dev/mapper/the_raid-lv_data
>>>>>>>> 15T 14T 804G 95% /data
>>>>>>>>
>>>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>>>> /dev/sdb1 21T 18T 2.1T 90% /mseas-data-0-0
>>>>>>>>
>>>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>>>> /dev/sdb1 21T 20T 784G 97% /mseas-data-0-1
>>>>>>>>
>>>>>>>> What should we do to fix this problem or look at to diagnose
>>>>>>>> this problem?
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>> Pat Haley Email: phaley at mit.edu
>>>>>>>> Center for Ocean Engineering Phone: (617) 253-6824
>>>>>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>>>>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>>>>>>> 77 Massachusetts Avenue
>>>>>>>> Cambridge, MA 02139-4301
>>>>>>>> _______________________________________________
>>>>>>>> Gluster-users mailing list
>>>>>>>> Gluster-users at gluster.org
>>>>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>> Pat Haley Email: phaley at mit.edu
>>>>>> Center for Ocean Engineering Phone: (617) 253-6824
>>>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>>>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>>>>> 77 Massachusetts Avenue
>>>>>> Cambridge, MA 02139-4301
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>>
>>>>>> This email and any files transmitted with it are confidential and are
>>>>>> intended solely for the use of the individual or entity to whom they
>>>>>> are addressed. If you are not the original recipient or the person
>>>>>> responsible for delivering the email to the intended recipient, be
>>>>>> advised that you have received this email in error, and that any use,
>>>>>> dissemination, forwarding, printing, or copying of this email is
>>>>>> strictly prohibited. If you received this email in error, please
>>>>>> immediately notify the sender and delete the original.
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: phaley at mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
More information about the Gluster-users
mailing list