[Gluster-users] high memory usage of mount

Pranith Kumar Karampuri pkarampu at redhat.com
Wed Aug 6 12:20:03 UTC 2014


On 08/05/2014 09:16 PM, Ryan Clough wrote:
> I spoke too soon. Runing "ls -lR" on one of our largest directory 
> structures overnight has caused glusterfs to use lots of memory. It 
> appears as though glusterfs process is still gradually consuming more 
> and more memory. I have tried to release the memory forcefully by 
> issuing this command:
> sync; echo 3 > /proc/sys/vm/drop_caches
> But glusterfs holds on to its memory. The high memory is also 
> expressed on the client side as well as on the server side.
> Right now both of my brick servers are using about 7GB of RAM for the 
> glusterfs process and the client that is running the "ls -lR" is using 
> about 8GB of RAM. Below are some basic specifications of my hardware. 
> Both server and client are running version 3.5.2. I have attached a 
> statedump of the client glusterfs.
Could you please send out statedumps please. That should help us figure 
out what the problem is.

Pranith
>
> Brick server hardware:
> Dual 6-core Intel Xeon CPU E5-2620 0 @ 2.00GHz (HT is on)
> 32GB SDRAM
> 2 - 500GB SATA drives in RAID1 for OS
> 12 - 3TB SATA drives in RAID6 with LVM and XFS for data
>
>
> Client hardware:
> Dual 8-core AMD OpteronProcessor 6128
> 32GB SDRAM
>
> Ryan Clough
> Information Systems
> Decision Sciences International Corporation 
> <http://www.decisionsciencescorp.com/><http://www.decisionsciencescorp.com/>
>
>
> On Mon, Aug 4, 2014 at 12:07 PM, Ryan Clough <ryan.clough at dsic.com 
> <mailto:ryan.clough at dsic.com>> wrote:
>
>     Hi,
>     I too was experiencing this issue on my bricks. I am using version
>     3.5.2 and after setting io-cache and quick-read to "off", as
>     Poornima suggested, I am no longer seeing glusterfs gobbling
>     memory. I noticed it first when I enabled quotas and during the
>     quota-crawl glusterfs process would be OOM killed by the kernel.
>     Before, my bricks would consume all available memory until swap
>     was exhausted and the kernel OOMs the glusterfs process. There is
>     a rebalance running right now and glusterfs is behaving. Here is
>     some output of my current config. Let me know if I can provide
>     anything else to help.
>
>     [root at tgluster01 ~]# gluster volume status all detail
>     Status of volume: tgluster_volume
>     ------------------------------------------------------------------------------
>     Brick                : Brick tgluster01:/gluster_data
>     Port                 : 49153
>     Online               : Y
>     Pid                  : 2407
>     File System          : xfs
>     Device               : /dev/mapper/vg_data-lv_data
>     Mount Options        :
>     rw,noatime,nodiratime,logbufs=8,logbsize=256k,inode64,nobarrier
>     Inode Size           : 512
>     Disk Space Free      : 3.5TB
>     Total Disk Space     : 27.3TB
>     Inode Count          : 2929685696
>     Free Inodes          : 2863589912
>     ------------------------------------------------------------------------------
>     Brick                : Brick tgluster02:/gluster_data
>     Port                 : 49152
>     Online               : Y
>     Pid                  : 2402
>     File System          : xfs
>     Device               : /dev/mapper/vg_data-lv_data
>     Mount Options        :
>     rw,noatime,nodiratime,logbufs=8,logbsize=256k,inode64,nobarrier
>     Inode Size           : 512
>     Disk Space Free      : 5.4TB
>     Total Disk Space     : 27.3TB
>     Inode Count          : 2929685696
>     Free Inodes          : 2864874648
>
>     [root at tgluster01 ~]# gluster volume status
>     Status of volume: tgluster_volume
>     Gluster process                        Port    Online Pid
>     ------------------------------------------------------------------------------
>     Brick tgluster01:/gluster_data                49153 Y    2407
>     Brick tgluster02:/gluster_data                49152 Y    2402
>     Quota Daemon on localhost                N/A    Y    2415
>     Quota Daemon on tgluster02                N/A    Y    2565
>
>     Task Status of Volume tgluster_volume
>     ------------------------------------------------------------------------------
>     Task                 : Rebalance
>     ID                   : 31fd1edb-dd6d-4c25-b4b5-1ce7bc0670f3
>     Status               : in progress
>
>     [root at tgluster01 ~]# gluster volume info
>     Volume Name: tgluster_volume
>     Type: Distribute
>     Volume ID: 796774f8-f9ec-476c-9d08-0f5f937d5ad9
>     Status: Started
>     Number of Bricks: 2
>     Transport-type: tcp
>     Bricks:
>     Brick1: tgluster01:/gluster_data
>     Brick2: tgluster02:/gluster_data
>     Options Reconfigured:
>     features.quota-deem-statfs: on
>     performance.client-io-threads: on
>     performance.md-cache-timeout: 1
>     performance.cache-max-file-size: 10MB
>     network.ping-timeout: 60
>     performance.write-behind-window-size: 4MB
>     performance.read-ahead: on
>     performance.cache-refresh-timeout: 1
>     performance.cache-size: 10GB
>     performance.quick-read: off
>     nfs.disable: on
>     features.quota: on
>     performance.io-thread-count: 24
>     cluster.eager-lock: on
>     server.statedump-path: /var/log/glusterfs/
>     performance.flush-behind: on
>     performance.write-behind: on
>     performance.stat-prefetch: on
>     performance.io-cache: off
>
>     [root at tgluster01 ~]# gluster volume status all mem
>     Memory status for volume : tgluster_volume
>     ----------------------------------------------
>     Brick : tgluster01:/gluster_data
>     Mallinfo
>     --------
>     Arena    : 25788416
>     Ordblks  : 7222
>     Smblks   : 1
>     Hblks    : 12
>     Hblkhd   : 16060416
>     Usmblks  : 0
>     Fsmblks  : 80
>     Uordblks : 25037744
>     Fordblks : 750672
>     Keepcost : 132816
>
>     Mempool Stats
>     -------------
>     Name                            HotCount ColdCount PaddedSizeof
>     AllocCount MaxAlloc   Misses Max-StdAlloc
>     ----                            -------- --------- ------------
>     ---------- -------- -------- ------------
>     tgluster_volume-server:fd_t           11 1013          108    
>     194246       22        0 0
>     tgluster_volume-server:dentry_t     16384 0           84   
>     1280505    16384   481095        32968
>     tgluster_volume-server:inode_t     16383 1          156  
>     13974240    16384  7625153        39688
>     tgluster_volume-changelog:changelog_local_t 0        64         
>     108          0        0 0            0
>     tgluster_volume-locks:pl_local_t         0 32          148   
>     3922857        4        0            0
>     tgluster_volume-marker:marker_local_t         0 128         
>     332    6163938        8        0            0
>     tgluster_volume-quota:struct saved_frame         0 16         
>     124      65000        6        0            0
>     tgluster_volume-quota:struct rpc_req         0 16         
>     588      65000        6        0            0
>     tgluster_volume-quota:quota_local_t         0 64          404   
>     4476051        8        0            0
>     tgluster_volume-server:rpcsvc_request_t         0 512        
>     2828    6694494        8        0            0
>     glusterfs:struct saved_frame           0 8          124         
>     2        2        0            0
>     glusterfs:struct rpc_req               0 8          588         
>     2        2        0            0
>     glusterfs:rpcsvc_request_t             1         7 2828         
>     2        1        0            0
>     glusterfs:data_t                     164 16219           52  
>     60680465     2012        0 0
>     glusterfs:data_pair_t                159 16224           68  
>     34718980     1348        0 0
>     glusterfs:dict_t                      15 4081          140  
>     24689263      714        0 0
>     glusterfs:call_stub_t                  0      1024 3756   
>     8263013        9        0            0
>     glusterfs:call_stack_t                 1      1023 1836   
>     6675669        8        0            0
>     glusterfs:call_frame_t                 0 4096          172  
>     55532603      251        0 0
>     ----------------------------------------------
>     Brick : tgluster02:/gluster_data
>     Mallinfo
>     --------
>     Arena    : 18714624
>     Ordblks  : 4211
>     Smblks   : 1
>     Hblks    : 12
>     Hblkhd   : 16060416
>     Usmblks  : 0
>     Fsmblks  : 80
>     Uordblks : 18250608
>     Fordblks : 464016
>     Keepcost : 131360
>
>     Mempool Stats
>     -------------
>     Name                            HotCount ColdCount PaddedSizeof
>     AllocCount MaxAlloc   Misses Max-StdAlloc
>     ----                            -------- --------- ------------
>     ---------- -------- -------- ------------
>     tgluster_volume-server:fd_t           11 1013          108    
>     155373       22        0 0
>     tgluster_volume-server:dentry_t     16383 1           84   
>     1297732    16384   396012        21124
>     tgluster_volume-server:inode_t     16384 0          156  
>     13896002    16384  7434842        24494
>     tgluster_volume-changelog:changelog_local_t 0        64         
>     108          0        0 0            0
>     tgluster_volume-locks:pl_local_t         2 30          148   
>     5578625       17        0            0
>     tgluster_volume-marker:marker_local_t         3 125         
>     332    6834019       68        0            0
>     tgluster_volume-quota:struct saved_frame         0 16         
>     124      64922       10        0            0
>     tgluster_volume-quota:struct rpc_req         0 16         
>     588      65000       10        0            0
>     tgluster_volume-quota:quota_local_t         3 61          404   
>     4216852       64        0            0
>     tgluster_volume-server:rpcsvc_request_t         3 509        
>     2828    6406870       64        0            0
>     glusterfs:struct saved_frame           0 8          124         
>     2        2        0            0
>     glusterfs:struct rpc_req               0 8          588         
>     2        2        0            0
>     glusterfs:rpcsvc_request_t             1         7 2828         
>     2        1        0            0
>     glusterfs:data_t                     185 16198           52  
>     80402618     1427        0 0
>     glusterfs:data_pair_t                177 16206           68  
>     40014499      737        0 0
>     glusterfs:dict_t                      18 4078          140  
>     35345779      729        0 0
>     glusterfs:call_stub_t                  3      1021 3756  
>     21374090       68        0            0
>     glusterfs:call_stack_t                 4      1020 1836   
>     6824400       68        0            0
>     glusterfs:call_frame_t                20 4076          172  
>     97255627      388        0 0
>     ----------------------------------------------
>
>     Ryan Clough
>     Information Systems
>     Decision Sciences International Corporation
>     <http://www.decisionsciencescorp.com/><http://www.decisionsciencescorp.com/>
>
>
>     On Sun, Aug 3, 2014 at 11:36 PM, Poornima Gurusiddaiah
>     <pgurusid at redhat.com <mailto:pgurusid at redhat.com>> wrote:
>
>         Hi,
>
>         From the statedump it is evident that the iobufs are leaking.
>         Also the hot count of the
>         pool-name=w-vol-io-cache:rbthash_entry_t is 10053, implies
>         io-cache xlator could be the cause of the leak.
>         From the logs, it looks like, quick-read performance xlator is
>         calling iobuf_free with NULL pointers, implies quick-read
>         could be leaking iobufs as well.
>
>         As a temperory solution, could you disable io-cache and/or
>         quick-read and see if the leak still persists?
>
>         $gluster volume set io-cache off
>         $gluster volume set quick-read off
>
>         This may reduce the performance to certain extent.
>
>         For further debugging, could you provide the core dump or
>         steps to reproduce if avaiable?
>
>         Regards,
>         Poornima
>
>         ----- Original Message -----
>         From: "Tamas Papp" <tompos at martos.bme.hu
>         <mailto:tompos at martos.bme.hu>>
>         To: "Poornima Gurusiddaiah" <pgurusid at redhat.com
>         <mailto:pgurusid at redhat.com>>
>         Cc: Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>         Sent: Sunday, August 3, 2014 10:33:17 PM
>         Subject: Re: [Gluster-users] high memory usage of mount
>
>
>         On 07/31/2014 09:17 AM, Tamas Papp wrote:
>         >
>         > On 07/31/2014 09:02 AM, Poornima Gurusiddaiah wrote:
>         >> Hi,
>         >
>         > hi,
>         >
>         >> Can you provide the statedump of the process, it can be
>         obtained as
>         >> follows:
>         >> $ gluster --print-statedumpdir  #create this directory if
>         it doesn't
>         >> exist.
>         >> $ kill -USR1 <pid-of-glusterfs-process>  #generates state dump.
>         >
>         > http://rtfm.co.hu/glusterdump.2464.dump.1406790562.zip
>         >
>         >> Also, xporting Gluster via Samba-VFS-plugin method is
>         preferred over
>         >> Fuse mount export. For more details refer to:
>         >>
>         http://lalatendumohanty.wordpress.com/2014/02/11/using-glusterfs-with-samba-and-samba-vfs-plugin-for-glusterfs-on-fedora-20/
>         >>
>         >
>         > When I tried it about half year ago it didn't work properly.
>         Clients
>         > lost mounts, access errors etc.
>         >
>         > But I will give it a try, though it's not included in
>         ubuntu's samba
>         > AFAIK.
>         >
>         >
>         > Thank you,
>         > tamas
>         >
>         > ps. I forget to mention, I can see this issue only one node.
>         The rest
>         > of nodes are fine.
>
>         hi Poornima,
>
>         Do you have  idea, what's going on here?
>
>         Thanks,
>         tamas
>         _______________________________________________
>         Gluster-users mailing list
>         Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>         http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> This communication is intended only for the person(s) to whom it is 
> addressed and may contain confidential and/or privileged information. 
> Any review, re-transmission, dissemination, copying or other use of, 
> or taking of any action in reliance upon, this information by persons 
> or entities other than the intended recipient(s) is prohibited. If you 
> received this communication in error, please report the error to the 
> sender by return email and delete this communication from your records.
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140806/870c2502/attachment-0001.html>


More information about the Gluster-users mailing list