[Gluster-users] Gluster not recognizing available space

Pat Haley phaley at MIT.EDU
Tue Jan 21 15:52:01 UTC 2014


Hi Lala,

The glusterfsd process is running (see below).  I also tried
"service iptables stop" (followed by restarting
gluserd) but still have the N in the OnLine column.
What should I look at next?

Thanks

# ps aux | grep gluster
root      2916  1.4  1.2 4442964 105740 ?      Ssl   2013 870:40 
/usr/sbin/glusterfsd -s localhost --volfile-id gdata.gluster-data.data 
-p /var/lib/glusterd/vols/gdata/run/gluster-data-data.pid -S 
/tmp/e3eac7ce95e786a3d909b8fc65ed2059.socket --brick-name /data -l 
/var/log/glusterfs/bricks/data.log --xlator-option 
*-posix.glusterd-uuid=e1bd42a5-9061-4930-8fe3-17e8e11e3e52 --brick-port 
24009 --xlator-option gdata-server.listen-port=24009
root      3882  0.0  0.4 201724 38220 ?        Ssl   2013   0:00 
/usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p 
/var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S 
/tmp/1ed43f585eab7d47ff1383c7dbf3e9bb.socket
root      7623  0.0  0.4 136192 38180 ?        Ssl  Jan17   0:00 
/usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p 
/var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S 
/tmp/ef02475524de51574a2984c0b399bfd6.socket
root     15778  0.0  0.4 136180 38180 ?        Ssl  09:54   0:00 
/usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p 
/var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S 
/tmp/a981bce11177b462b2335098e6498145.socket
root     17119  0.6  0.1 110640 10820 ?        Ssl  10:48   0:00 
/usr/sbin/glusterd -p /var/run/glusterd.pid
root     17145  0.0  0.4 136192 38168 ?        Ssl  10:48   0:00 
/usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p 
/var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S 
/tmp/43428f984ba9bca411868252631f17bc.socket
root     17206  0.0  0.0  61192   752 pts/1    S+   10:49   0:00 grep 
gluster


> On 01/21/2014 08:34 PM, Pat Haley wrote:
>>
>> Also, going back to an earlier Email,
>> should I be concerned that in the output
>> from "gluster volume status" the
>> brick "gluster-data:/data" has an "N"
>> in the "Online" column?  Does this suggest
>> an additional debugging route?
>>
> 
> Yes, it means  brick i.e. gluster-data:/data is not reachable. Check the 
> glusterfsd process in "gluster-data", it should be running .  An 
> unreachable brick can be a potential issue, specially when you are using 
> a distributed volume. I hope you have disabled selinux and have iptables 
> rules in favor of glusterfs processes.
> 
> -Lala
>> gluster volume status
>> Status of volume: gdata
>> Gluster process                                         Port Online  Pid
>> ------------------------------------------------------------------------------ 
>>
>> Brick gluster-0-0:/mseas-data-0-0                       24009   Y 27006
>> Brick gluster-0-1:/mseas-data-0-1                       24009 Y       
>> 7063
>> Brick gluster-data:/data                                24010   N 15772
>> NFS Server on localhost                                 38467   Y 14936
>> NFS Server on gluster-data                              38467   Y 15778
>> NFS Server on gluster-0-1                               38467   Y 21083
>>
>>>
>>> First, another update on my test of writing
>>> a directory with 480 6Mb files.  Not only do
>>> over 3/4 of the files appear, but the are
>>> written on all 3 bricks.  Again, it is random
>>> which files are not written but what I seem
>>> to see is that files are written to each brick
>>> even after the failures.  Does this suggest
>>> anything else I should be looking at?
>>>
>>> As to Brian's suggestion, how exactly do I perform
>>> a "quick inode allocation test"?
>>>
>>> Thanks
>>>
>>> Pat
>>>
>>>> On 01/17/2014 07:48 PM, Pat Haley wrote:
>>>>> Hi Franco,
>>>>>
>>>>> I checked using df -i on all 3 bricks.  No brick is over
>>>>> 1% inode usage.
>>>>>
>>>>
>>>> It might be worth a quick inode allocation test on the fs for each
>>>> brick, regardless. There are other non-obvious scenarios that can cause
>>>> inode allocation to fail, at least on xfs (i.e., contiguous block
>>>> allocation). Ideally, you'll have the ability to do this in a
>>>> subdirectory outside the actual glusterfs brick.
>>>>
>>>> Brian
>>>>
>>>>> Thanks.
>>>>>
>>>>> Pat
>>>>>
>>>>>> Have you run out of inodes on the underlying  filesystems?
>>>>>>
>>>>>> On 18 Jan 2014 05:41, Pat Haley <phaley at MIT.EDU> wrote:
>>>>>>
>>>>>> Latest updates:
>>>>>>
>>>>>> no error messages were found on the log files of the bricks.
>>>>>>
>>>>>> The error messages appear on the client log files. Writing
>>>>>> from a second client also has the same errors.
>>>>>>
>>>>>> Note that if I try to write a directory with 480 6Mb files
>>>>>> to /projects, over 3/4 of the files are written.  It is
>>>>>> random which files are not written (i.e. it is not the
>>>>>> last 1/4 of the files which fail)
>>>>>>
>>>>>>>  Hi,
>>>>>>>
>>>>>>>  Some additional data
>>>>>>>
>>>>>>>
>>>>>>>  [root at mseas-data save]# gluster volume info
>>>>>>>
>>>>>>>  Volume Name: gdata
>>>>>>>  Type: Distribute
>>>>>>>  Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d
>>>>>>>  Status: Started
>>>>>>>  Number of Bricks: 3
>>>>>>>  Transport-type: tcp
>>>>>>>  Bricks:
>>>>>>>  Brick1: gluster-0-0:/mseas-data-0-0
>>>>>>>  Brick2: gluster-0-1:/mseas-data-0-1
>>>>>>>  Brick3: gluster-data:/data
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>  [root at mseas-data save]# gluster volume status
>>>>>>>  Status of volume: gdata
>>>>>>>  Gluster process Port 
>>>>>> Online  Pid
>>>>>> ------------------------------------------------------------------------------ 
>>>>>>
>>>>>>
>>>>>>>  Brick gluster-0-0:/mseas-data-0-0 24009   Y 27006
>>>>>>>  Brick gluster-0-1:/mseas-data-0-1 24009  Y       7063
>>>>>>>  Brick gluster-data:/data 24010  N       8007
>>>>>>>  NFS Server on localhost 38467  Y       8013
>>>>>>>  NFS Server on gluster-0-1 38467   Y 10228
>>>>>>>  NFS Server on 10.1.1.10 38467  Y       3867
>>>>>>>
>>>>>>>
>>>>>>>  Noticing that the brick gluster-data:/data was appearing as N
>>>>>>>  in the "online" column, I tried (1) detaching gluster-data (using
>>>>>>>  gluster peer detach gluster-data issued from gluster-0-0), (2) 
>>>>>>> removing
>>>>>>>  /var/lib/glusterd, (3) restarting glusterd on gluster-data,
>>>>>>>  (4) reattaching /gluster-data (using gluster peer probe 
>>>>>>> gluster-data
>>>>>>>  issued from gluster-0-0) then (5) restart glusterd one more time 
>>>>>>> on all
>>>>>>>  3 bricks.  The brick gluster-data:/data still appears as N in the
>>>>>>> Online
>>>>>>>  column.
>>>>>>>
>>>>>>>  [root at mseas-data save]# gluster peer status
>>>>>>>  Number of Peers: 2
>>>>>>>
>>>>>>>  Hostname: gluster-0-1
>>>>>>>  Uuid: 393fc4a6-1573-4564-971e-1b1aec434167
>>>>>>>  State: Peer in Cluster (Connected)
>>>>>>>
>>>>>>>  Hostname: 10.1.1.10
>>>>>>>  Uuid: 3619440a-4ca3-4151-b62e-d4d6bf2e0c03
>>>>>>>  State: Peer in Cluster (Connected)
>>>>>>>
>>>>>>>  (similarly from the other bricks)
>>>>>>>
>>>>>>>  Ping works between all bricks too.
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> We are using gluster to present 3 bricks as a single name space.
>>>>>>>> We appear to have a situation in which gluster thinks there
>>>>>>>> is no disk space when there is actually plenty. I have restarted
>>>>>>>> the glusterd deamons on all three bricks and I still get the
>>>>>>>> following message
>>>>>>>>
>>>>>>>> /bin/cp: cannot create regular file
>>>>>>>> `./Bottom_Gravity_Current_25/344.mat': No space left on device
>>>>>>>>
>>>>>>>> This is a 6Mbyte file.  The total space available on
>>>>>>>> gluster is 3.6T
>>>>>>>>
>>>>>>>> Filesystem            Size  Used Avail Use% Mounted on
>>>>>>>> mseas-data:/gdata      55T   51T  3.6T  94% /gdata
>>>>>>>>
>>>>>>>> Also, no single brick is full:
>>>>>>>>
>>>>>>>> Filesystem            Size  Used Avail Use% Mounted on
>>>>>>>> /dev/mapper/the_raid-lv_data
>>>>>>>>                        15T   14T  804G  95% /data
>>>>>>>>
>>>>>>>> Filesystem            Size  Used Avail Use% Mounted on
>>>>>>>> /dev/sdb1              21T   18T  2.1T  90% /mseas-data-0-0
>>>>>>>>
>>>>>>>> Filesystem            Size  Used Avail Use% Mounted on
>>>>>>>> /dev/sdb1              21T   20T  784G  97% /mseas-data-0-1
>>>>>>>>
>>>>>>>> What should we do to fix this problem or look at to diagnose
>>>>>>>> this problem?
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>>>> Pat Haley                          Email: phaley at mit.edu
>>>>>>>> Center for Ocean Engineering       Phone:  (617) 253-6824
>>>>>>>> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>>>>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>>>>>>> 77 Massachusetts Avenue
>>>>>>>> Cambridge, MA  02139-4301
>>>>>>>> _______________________________________________
>>>>>>>> Gluster-users mailing list
>>>>>>>> Gluster-users at gluster.org
>>>>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>>>
>>>>>>
>>>>>> -- 
>>>>>>
>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>>>>> Pat Haley                          Email:  phaley at mit.edu
>>>>>> Center for Ocean Engineering       Phone:  (617) 253-6824
>>>>>> Dept. of Mechanical Engineering    Fax:    (617) 253-8125
>>>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>>>>>> 77 Massachusetts Avenue
>>>>>> Cambridge, MA  02139-4301
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>>
>>>>>> ------------------------------------------------------------------------ 
>>>>>>
>>>>>>
>>>>>>
>>>>>> This email and any files transmitted with it are confidential and are
>>>>>> intended solely for the use of the individual or entity to whom they
>>>>>> are addressed. If you are not the original recipient or the person
>>>>>> responsible for delivering the email to the intended recipient, be
>>>>>> advised that you have received this email in error, and that any use,
>>>>>> dissemination, forwarding, printing, or copying of this email is
>>>>>> strictly prohibited. If you received this email in error, please
>>>>>> immediately notify the sender and delete the original.
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
> 


-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301



More information about the Gluster-users mailing list