[Gluster-users] Memory leak in GlusterFS FUSE client

Soumya Koduri skoduri at redhat.com
Fri Dec 25 14:58:13 UTC 2015



On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:
> Another addition: it seems to be GlusterFS API library memory leak
> because NFS-Ganesha also consumes huge amount of memory while doing
> ordinary "find . -type f" via NFSv4.2 on remote client. Here is memory
> usage:
>
> ===
> root      5416 34.2 78.5 2047176 1480552 ?     Ssl  12:02 117:54
> /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
> /etc/ganesha/ganesha.conf -N NIV_EVENT
> ===
>
> 1.4G is too much for simple stat() :(.
>
> Ideas?
nfs-ganesha also has cache layer which can scale to millions of entries 
depending on the number of files/directories being looked upon. However 
there are parameters to tune it. So either try stat with few entries or 
add below block in nfs-ganesha.conf file, set low limits and check the 
difference. That may help us narrow down how much memory actually 
consumed by core nfs-ganesha and gfAPI.

CACHEINODE {
	Cache_Size(uint32, range 1 to UINT32_MAX, default 32633); # cache size
	Entries_HWMark(uint32, range 1 to UINT32_MAX, default 100000); #Max no. 
of entries in the cache.
}

Thanks,
Soumya

>
> 24.12.2015 16:32, Oleksandr Natalenko написав:
>> Still actual issue for 3.7.6. Any suggestions?
>>
>> 24.09.2015 10:14, Oleksandr Natalenko написав:
>>> In our GlusterFS deployment we've encountered something like memory
>>> leak in GlusterFS FUSE client.
>>>
>>> We use replicated (×2) GlusterFS volume to store mail (exim+dovecot,
>>> maildir format). Here is inode stats for both bricks and mountpoint:
>>>
>>> ===
>>> Brick 1 (Server 1):
>>>
>>> Filesystem                                             Inodes    IUsed
>>>      IFree IUse% Mounted on
>>> /dev/mapper/vg_vd1_misc-lv08_mail                   578768144 10954918
>>>  567813226    2% /bricks/r6sdLV08_vd1_mail
>>>
>>> Brick 2 (Server 2):
>>>
>>> Filesystem                                             Inodes    IUsed
>>>      IFree IUse% Mounted on
>>> /dev/mapper/vg_vd0_misc-lv07_mail                   578767984 10954913
>>>  567813071    2% /bricks/r6sdLV07_vd0_mail
>>>
>>> Mountpoint (Server 3):
>>>
>>> Filesystem                              Inodes    IUsed      IFree
>>> IUse% Mounted on
>>> glusterfs.xxx:mail                   578767760 10954915  567812845
>>> 2% /var/spool/mail/virtual
>>> ===
>>>
>>> glusterfs.xxx domain has two A records for both Server 1 and Server 2.
>>>
>>> Here is volume info:
>>>
>>> ===
>>> Volume Name: mail
>>> Type: Replicate
>>> Volume ID: f564e85c-7aa6-4170-9417-1f501aa98cd2
>>> Status: Started
>>> Number of Bricks: 1 x 2 = 2
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
>>> Brick2: server2.xxx:/bricks/r6sdLV07_vd0_mail/mail
>>> Options Reconfigured:
>>> nfs.rpc-auth-allow: 1.2.4.0/24,4.5.6.0/24
>>> features.cache-invalidation-timeout: 10
>>> performance.stat-prefetch: off
>>> performance.quick-read: on
>>> performance.read-ahead: off
>>> performance.flush-behind: on
>>> performance.write-behind: on
>>> performance.io-thread-count: 4
>>> performance.cache-max-file-size: 1048576
>>> performance.cache-size: 67108864
>>> performance.readdir-ahead: off
>>> ===
>>>
>>> Soon enough after mounting and exim/dovecot start, glusterfs client
>>> process begins to consume huge amount of RAM:
>>>
>>> ===
>>> user at server3 ~$ ps aux | grep glusterfs | grep mail
>>> root     28895 14.4 15.0 15510324 14908868 ?   Ssl  Sep03 4310:05
>>> /usr/sbin/glusterfs --fopen-keep-cache --direct-io-mode=disable
>>> --volfile-server=glusterfs.xxx --volfile-id=mail
>>> /var/spool/mail/virtual
>>> ===
>>>
>>> That is, ~15 GiB of RAM.
>>>
>>> Also we've tried to use mountpoint withing separate KVM VM with 2 or 3
>>> GiB of RAM, and soon after starting mail daemons got OOM killer for
>>> glusterfs client process.
>>>
>>> Mounting same share via NFS works just fine. Also, we have much less
>>> iowait and loadavg on client side with NFS.
>>>
>>> Also, we've tried to change IO threads count and cache size in order
>>> to limit memory usage with no luck. As you can see, total cache size
>>> is 4×64==256 MiB (compare to 15 GiB).
>>>
>>> Enabling-disabling stat-prefetch, read-ahead and readdir-ahead didn't
>>> help as well.
>>>
>>> Here are volume memory stats:
>>>
>>> ===
>>> Memory status for volume : mail
>>> ----------------------------------------------
>>> Brick : server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
>>> Mallinfo
>>> --------
>>> Arena    : 36859904
>>> Ordblks  : 10357
>>> Smblks   : 519
>>> Hblks    : 21
>>> Hblkhd   : 30515200
>>> Usmblks  : 0
>>> Fsmblks  : 53440
>>> Uordblks : 18604144
>>> Fordblks : 18255760
>>> Keepcost : 114112
>>>
>>> Mempool Stats
>>> -------------
>>> Name                            HotCount ColdCount PaddedSizeof
>>> AllocCount MaxAlloc   Misses Max-StdAlloc
>>> ----                            -------- --------- ------------
>>> ---------- -------- -------- ------------
>>> mail-server:fd_t                       0      1024          108
>>> 30773120      137        0            0
>>> mail-server:dentry_t               16110       274           84
>>> 235676148    16384  1106499         1152
>>> mail-server:inode_t                16363        21          156
>>> 237216876    16384  1876651         1169
>>> mail-trash:fd_t                        0      1024          108
>>>   0        0        0            0
>>> mail-trash:dentry_t                    0     32768           84
>>>   0        0        0            0
>>> mail-trash:inode_t                     4     32764          156
>>>   4        4        0            0
>>> mail-trash:trash_local_t               0        64         8628
>>>   0        0        0            0
>>> mail-changetimerecorder:gf_ctr_local_t         0        64
>>> 16540          0        0        0            0
>>> mail-changelog:rpcsvc_request_t         0         8         2828
>>>    0        0        0            0
>>> mail-changelog:changelog_local_t         0        64          116
>>>     0        0        0            0
>>> mail-bitrot-stub:br_stub_local_t         0       512           84
>>> 79204        4        0            0
>>> mail-locks:pl_local_t                  0        32          148
>>> 6812757        4        0            0
>>> mail-upcall:upcall_local_t             0       512          108
>>>   0        0        0            0
>>> mail-marker:marker_local_t             0       128          332
>>> 64980        3        0            0
>>> mail-quota:quota_local_t               0        64          476
>>>   0        0        0            0
>>> mail-server:rpcsvc_request_t           0       512         2828
>>> 45462533       34        0            0
>>> glusterfs:struct saved_frame           0         8          124
>>>   2        2        0            0
>>> glusterfs:struct rpc_req               0         8          588
>>>   2        2        0            0
>>> glusterfs:rpcsvc_request_t             1         7         2828
>>>   2        1        0            0
>>> glusterfs:log_buf_t                    5       251          140
>>> 3452        6        0            0
>>> glusterfs:data_t                     242     16141           52
>>> 480115498      664        0            0
>>> glusterfs:data_pair_t                230     16153           68
>>> 179483528      275        0            0
>>> glusterfs:dict_t                      23      4073          140
>>> 303751675      627        0            0
>>> glusterfs:call_stub_t                  0      1024         3764
>>> 45290655       34        0            0
>>> glusterfs:call_stack_t                 1      1023         1708
>>> 43598469       34        0            0
>>> glusterfs:call_frame_t                 1      4095          172
>>> 336219655      184        0            0
>>> ----------------------------------------------
>>> Brick : server2.xxx:/bricks/r6sdLV07_vd0_mail/mail
>>> Mallinfo
>>> --------
>>> Arena    : 38174720
>>> Ordblks  : 9041
>>> Smblks   : 507
>>> Hblks    : 21
>>> Hblkhd   : 30515200
>>> Usmblks  : 0
>>> Fsmblks  : 51712
>>> Uordblks : 19415008
>>> Fordblks : 18759712
>>> Keepcost : 114848
>>>
>>> Mempool Stats
>>> -------------
>>> Name                            HotCount ColdCount PaddedSizeof
>>> AllocCount MaxAlloc   Misses Max-StdAlloc
>>> ----                            -------- --------- ------------
>>> ---------- -------- -------- ------------
>>> mail-server:fd_t                       0      1024          108
>>> 2373075      133        0            0
>>> mail-server:dentry_t               14114      2270           84
>>> 3513654    16384     2300          267
>>> mail-server:inode_t                16374        10          156
>>> 6766642    16384   194635         1279
>>> mail-trash:fd_t                        0      1024          108
>>>   0        0        0            0
>>> mail-trash:dentry_t                    0     32768           84
>>>   0        0        0            0
>>> mail-trash:inode_t                     4     32764          156
>>>   4        4        0            0
>>> mail-trash:trash_local_t               0        64         8628
>>>   0        0        0            0
>>> mail-changetimerecorder:gf_ctr_local_t         0        64
>>> 16540          0        0        0            0
>>> mail-changelog:rpcsvc_request_t         0         8         2828
>>>    0        0        0            0
>>> mail-changelog:changelog_local_t         0        64          116
>>>     0        0        0            0
>>> mail-bitrot-stub:br_stub_local_t         0       512           84
>>> 71354        4        0            0
>>> mail-locks:pl_local_t                  0        32          148
>>> 8135032        4        0            0
>>> mail-upcall:upcall_local_t             0       512          108
>>>   0        0        0            0
>>> mail-marker:marker_local_t             0       128          332
>>> 65005        3        0            0
>>> mail-quota:quota_local_t               0        64          476
>>>   0        0        0            0
>>> mail-server:rpcsvc_request_t           0       512         2828
>>> 12882393       30        0            0
>>> glusterfs:struct saved_frame           0         8          124
>>>   2        2        0            0
>>> glusterfs:struct rpc_req               0         8          588
>>>   2        2        0            0
>>> glusterfs:rpcsvc_request_t             1         7         2828
>>>   2        1        0            0
>>> glusterfs:log_buf_t                    5       251          140
>>> 3443        6        0            0
>>> glusterfs:data_t                     242     16141           52
>>> 138743429      290        0            0
>>> glusterfs:data_pair_t                230     16153           68
>>> 126649864      270        0            0
>>> glusterfs:dict_t                      23      4073          140
>>> 20356289       63        0            0
>>> glusterfs:call_stub_t                  0      1024         3764
>>> 13678560       31        0            0
>>> glusterfs:call_stack_t                 1      1023         1708
>>> 11011561       30        0            0
>>> glusterfs:call_frame_t                 1      4095          172
>>> 125764190      193        0            0
>>> ----------------------------------------------
>>> ===
>>>
>>> So, my questions are:
>>>
>>> 1) what one should do to limit GlusterFS FUSE client memory usage?
>>> 2) what one should do to prevent client high loadavg because of high
>>> iowait because of multiple concurrent volume users?
>>>
>>> Server/client OS is CentOS 7.1, GlusterFS server version is 3.7.3,
>>> GlusterFS client version is 3.7.4.
>>>
>>> Any additional info needed?
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users


More information about the Gluster-users mailing list