[Gluster-users] Memory leak in GlusterFS FUSE client

Soumya Koduri skoduri at redhat.com
Thu Jan 7 06:36:42 UTC 2016



On 01/06/2016 01:58 PM, Oleksandr Natalenko wrote:
> OK, here is valgrind log of patched Ganesha (I took recent version of
> your patchset, 8685abfc6d) with Entries_HWMARK set to 500.
>
> https://gist.github.com/5397c152a259b9600af0
>
> See no huge runtime leaks now.

Glad to hear this :)

However, I've repeated this test with
> another volume in replica and got the following Ganesha error:
>
> ===
> ganesha.nfsd: inode.c:716: __inode_forget: Assertion `inode->nlookup >=
> nlookup' failed.
> ===

I repeated the tests on replica volume as well. But haven't hit any 
assert. Could you confirm if you have taken the latest gluster patch set 
#3 ?
  - http://review.gluster.org/#/c/13096/3

If you are hitting the issue even then, please provide the core if possible.

Thanks,
Soumya

>
> 06.01.2016 08:40, Soumya Koduri написав:
>> On 01/06/2016 03:53 AM, Oleksandr Natalenko wrote:
>>> OK, I've repeated the same traversing test with patched GlusterFS
>>> API, and
>>> here is new Valgrind log:
>>>
>>> https://gist.github.com/17ecb16a11c9aed957f5
>>>
>> Fuse mount doesn't use gfapi helper. Does your above GlusterFS API
>> application call glfs_fini() during exit? glfs_fini() is responsible
>> for freeing the memory consumed by gfAPI applications.
>>
>> Could you repeat the test with nfs-ganesha (which for sure calls
>> glfs_fini() and purges inodes if exceeds its inode cache limit) if
>> possible.
>>
>> Thanks,
>> Soumya
>>
>>> Still leaks.
>>>
>>> On вівторок, 5 січня 2016 р. 22:52:25 EET Soumya Koduri wrote:
>>>> On 01/05/2016 05:56 PM, Oleksandr Natalenko wrote:
>>>>> Unfortunately, both patches didn't make any difference for me.
>>>>>
>>>>> I've patched 3.7.6 with both patches, recompiled and installed patched
>>>>> GlusterFS package on client side and mounted volume with ~2M of files.
>>>>> The I performed usual tree traverse with simple "find".
>>>>>
>>>>> Memory RES value went from ~130M at the moment of mounting to ~1.5G
>>>>> after traversing the volume for ~40 mins. Valgrind log still shows
>>>>> lots
>>>>> of leaks. Here it is:
>>>>>
>>>>> https://gist.github.com/56906ca6e657c4ffa4a1
>>>>
>>>> Looks like you had done fuse mount. The patches which I have pasted
>>>> below apply to gfapi/nfs-ganesha applications.
>>>>
>>>> Also, to resolve the nfs-ganesha issue which I had mentioned below (in
>>>> case if Entries_HWMARK option gets changed), I have posted below fix -
>>>>     https://review.gerrithub.io/#/c/258687
>>>>
>>>> Thanks,
>>>> Soumya
>>>>
>>>>> Ideas?
>>>>>
>>>>> 05.01.2016 12:31, Soumya Koduri написав:
>>>>>> I tried to debug the inode* related leaks and seen some improvements
>>>>>> after applying the below patches when ran the same test (but will
>>>>>> smaller load). Could you please apply those patches & confirm the
>>>>>> same?
>>>>>>
>>>>>> a) http://review.gluster.org/13125
>>>>>>
>>>>>> This will fix the inodes & their ctx related leaks during unexport
>>>>>> and
>>>>>> the program exit. Please check the valgrind output after applying the
>>>>>> patch. It should not list any inodes related memory as lost.
>>>>>>
>>>>>> b) http://review.gluster.org/13096
>>>>>>
>>>>>> The reason the change in Entries_HWMARK (in your earlier mail) dint
>>>>>> have much effect is that the inode_nlookup count doesn't become zero
>>>>>> for those handles/inodes being closed by ganesha. Hence those inodes
>>>>>> shall get added to inode lru list instead of purge list which shall
>>>>>> get forcefully purged only when the number of gfapi inode table
>>>>>> entries reaches its limit (which is 137012).
>>>>>>
>>>>>> This patch fixes those 'nlookup' counts. Please apply this patch and
>>>>>> reduce 'Entries_HWMARK' to much lower value and check if it decreases
>>>>>> the in-memory being consumed by ganesha process while being active.
>>>>>>
>>>>>> CACHEINODE {
>>>>>>
>>>>>>          Entries_HWMark = 500;
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> Note: I see an issue with nfs-ganesha during exit when the option
>>>>>> 'Entries_HWMARK' gets changed. This is not related to any of the
>>>>>> above
>>>>>> patches (or rather Gluster) and I am currently debugging it.
>>>>>>
>>>>>> Thanks,
>>>>>> Soumya
>>>>>>
>>>>>> On 12/25/2015 11:34 PM, Oleksandr Natalenko wrote:
>>>>>>> 1. test with Cache_Size = 256 and Entries_HWMark = 4096
>>>>>>>
>>>>>>> Before find . -type f:
>>>>>>>
>>>>>>> root      3120  0.6 11.0 879120 208408 ?       Ssl  17:39   0:00
>>>>>>> /usr/bin/
>>>>>>> ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
>>>>>>> NIV_EVENT
>>>>>>>
>>>>>>> After:
>>>>>>>
>>>>>>> root      3120 11.4 24.3 1170076 458168 ?      Ssl  17:39  13:39
>>>>>>> /usr/bin/
>>>>>>> ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
>>>>>>> NIV_EVENT
>>>>>>>
>>>>>>> ~250M leak.
>>>>>>>
>>>>>>> 2. test with default values (after ganesha restart)
>>>>>>>
>>>>>>> Before:
>>>>>>>
>>>>>>> root     24937  1.3 10.4 875016 197808 ?       Ssl  19:39   0:00
>>>>>>> /usr/bin/
>>>>>>> ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
>>>>>>> NIV_EVENT
>>>>>>>
>>>>>>> After:
>>>>>>>
>>>>>>> root     24937  3.5 18.9 1022544 356340 ?      Ssl  19:39   0:40
>>>>>>> /usr/bin/
>>>>>>> ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N
>>>>>>> NIV_EVENT
>>>>>>>
>>>>>>> ~159M leak.
>>>>>>>
>>>>>>> No reasonable correlation detected. Second test was finished much
>>>>>>> faster than
>>>>>>> first (I guess, server-side GlusterFS cache or server kernel page
>>>>>>> cache is the
>>>>>>> cause).
>>>>>>>
>>>>>>> There are ~1.8M files on this test volume.
>>>>>>>
>>>>>>> On пʼятниця, 25 грудня 2015 р. 20:28:13 EET Soumya Koduri wrote:
>>>>>>>> On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:
>>>>>>>>> Another addition: it seems to be GlusterFS API library memory leak
>>>>>>>>> because NFS-Ganesha also consumes huge amount of memory while
>>>>>>>>> doing
>>>>>>>>> ordinary "find . -type f" via NFSv4.2 on remote client. Here is
>>>>>>>>> memory
>>>>>>>>> usage:
>>>>>>>>>
>>>>>>>>> ===
>>>>>>>>> root      5416 34.2 78.5 2047176 1480552 ?     Ssl  12:02 117:54
>>>>>>>>> /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f
>>>>>>>>> /etc/ganesha/ganesha.conf -N NIV_EVENT
>>>>>>>>> ===
>>>>>>>>>
>>>>>>>>> 1.4G is too much for simple stat() :(.
>>>>>>>>>
>>>>>>>>> Ideas?
>>>>>>>>
>>>>>>>> nfs-ganesha also has cache layer which can scale to millions of
>>>>>>>> entries
>>>>>>>> depending on the number of files/directories being looked upon.
>>>>>>>> However
>>>>>>>> there are parameters to tune it. So either try stat with few
>>>>>>>> entries or
>>>>>>>> add below block in nfs-ganesha.conf file, set low limits and
>>>>>>>> check the
>>>>>>>> difference. That may help us narrow down how much memory actually
>>>>>>>> consumed by core nfs-ganesha and gfAPI.
>>>>>>>>
>>>>>>>> CACHEINODE {
>>>>>>>>
>>>>>>>>      Cache_Size(uint32, range 1 to UINT32_MAX, default 32633); #
>>>>>>>>
>>>>>>>> cache size
>>>>>>>>
>>>>>>>>      Entries_HWMark(uint32, range 1 to UINT32_MAX, default 100000);
>>>>>>>>
>>>>>>>> #Max no.
>>>>>>>> of entries in the cache.
>>>>>>>> }
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Soumya
>>>>>>>>
>>>>>>>>> 24.12.2015 16:32, Oleksandr Natalenko написав:
>>>>>>>>>> Still actual issue for 3.7.6. Any suggestions?
>>>>>>>>>>
>>>>>>>>>> 24.09.2015 10:14, Oleksandr Natalenko написав:
>>>>>>>>>>> In our GlusterFS deployment we've encountered something like
>>>>>>>>>>> memory
>>>>>>>>>>> leak in GlusterFS FUSE client.
>>>>>>>>>>>
>>>>>>>>>>> We use replicated (×2) GlusterFS volume to store mail
>>>>>>>>>>> (exim+dovecot,
>>>>>>>>>>> maildir format). Here is inode stats for both bricks and
>>>>>>>>>>> mountpoint:
>>>>>>>>>>>
>>>>>>>>>>> ===
>>>>>>>>>>> Brick 1 (Server 1):
>>>>>>>>>>>
>>>>>>>>>>> Filesystem                                             Inodes
>>>>>>>>>>> IUsed
>>>>>>>>>>>
>>>>>>>>>>>        IFree IUse% Mounted on
>>>>>>>>>>>
>>>>>>>>>>> /dev/mapper/vg_vd1_misc-lv08_mail                   578768144
>>>>>>>>>>> 10954918
>>>>>>>>>>>
>>>>>>>>>>>    567813226    2% /bricks/r6sdLV08_vd1_mail
>>>>>>>>>>>
>>>>>>>>>>> Brick 2 (Server 2):
>>>>>>>>>>>
>>>>>>>>>>> Filesystem                                             Inodes
>>>>>>>>>>> IUsed
>>>>>>>>>>>
>>>>>>>>>>>        IFree IUse% Mounted on
>>>>>>>>>>>
>>>>>>>>>>> /dev/mapper/vg_vd0_misc-lv07_mail                   578767984
>>>>>>>>>>> 10954913
>>>>>>>>>>>
>>>>>>>>>>>    567813071    2% /bricks/r6sdLV07_vd0_mail
>>>>>>>>>>>
>>>>>>>>>>> Mountpoint (Server 3):
>>>>>>>>>>>
>>>>>>>>>>> Filesystem                              Inodes    IUsed IFree
>>>>>>>>>>> IUse% Mounted on
>>>>>>>>>>> glusterfs.xxx:mail                   578767760 10954915
>>>>>>>>>>> 567812845
>>>>>>>>>>> 2% /var/spool/mail/virtual
>>>>>>>>>>> ===
>>>>>>>>>>>
>>>>>>>>>>> glusterfs.xxx domain has two A records for both Server 1 and
>>>>>>>>>>> Server 2.
>>>>>>>>>>>
>>>>>>>>>>> Here is volume info:
>>>>>>>>>>>
>>>>>>>>>>> ===
>>>>>>>>>>> Volume Name: mail
>>>>>>>>>>> Type: Replicate
>>>>>>>>>>> Volume ID: f564e85c-7aa6-4170-9417-1f501aa98cd2
>>>>>>>>>>> Status: Started
>>>>>>>>>>> Number of Bricks: 1 x 2 = 2
>>>>>>>>>>> Transport-type: tcp
>>>>>>>>>>> Bricks:
>>>>>>>>>>> Brick1: server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
>>>>>>>>>>> Brick2: server2.xxx:/bricks/r6sdLV07_vd0_mail/mail
>>>>>>>>>>> Options Reconfigured:
>>>>>>>>>>> nfs.rpc-auth-allow: 1.2.4.0/24,4.5.6.0/24
>>>>>>>>>>> features.cache-invalidation-timeout: 10
>>>>>>>>>>> performance.stat-prefetch: off
>>>>>>>>>>> performance.quick-read: on
>>>>>>>>>>> performance.read-ahead: off
>>>>>>>>>>> performance.flush-behind: on
>>>>>>>>>>> performance.write-behind: on
>>>>>>>>>>> performance.io-thread-count: 4
>>>>>>>>>>> performance.cache-max-file-size: 1048576
>>>>>>>>>>> performance.cache-size: 67108864
>>>>>>>>>>> performance.readdir-ahead: off
>>>>>>>>>>> ===
>>>>>>>>>>>
>>>>>>>>>>> Soon enough after mounting and exim/dovecot start, glusterfs
>>>>>>>>>>> client
>>>>>>>>>>> process begins to consume huge amount of RAM:
>>>>>>>>>>>
>>>>>>>>>>> ===
>>>>>>>>>>> user at server3 ~$ ps aux | grep glusterfs | grep mail
>>>>>>>>>>> root     28895 14.4 15.0 15510324 14908868 ?   Ssl  Sep03
>>>>>>>>>>> 4310:05
>>>>>>>>>>> /usr/sbin/glusterfs --fopen-keep-cache --direct-io-mode=disable
>>>>>>>>>>> --volfile-server=glusterfs.xxx --volfile-id=mail
>>>>>>>>>>> /var/spool/mail/virtual
>>>>>>>>>>> ===
>>>>>>>>>>>
>>>>>>>>>>> That is, ~15 GiB of RAM.
>>>>>>>>>>>
>>>>>>>>>>> Also we've tried to use mountpoint withing separate KVM VM
>>>>>>>>>>> with 2
>>>>>>>>>>> or 3
>>>>>>>>>>> GiB of RAM, and soon after starting mail daemons got OOM
>>>>>>>>>>> killer for
>>>>>>>>>>> glusterfs client process.
>>>>>>>>>>>
>>>>>>>>>>> Mounting same share via NFS works just fine. Also, we have
>>>>>>>>>>> much less
>>>>>>>>>>> iowait and loadavg on client side with NFS.
>>>>>>>>>>>
>>>>>>>>>>> Also, we've tried to change IO threads count and cache size
>>>>>>>>>>> in order
>>>>>>>>>>> to limit memory usage with no luck. As you can see, total
>>>>>>>>>>> cache size
>>>>>>>>>>> is 4×64==256 MiB (compare to 15 GiB).
>>>>>>>>>>>
>>>>>>>>>>> Enabling-disabling stat-prefetch, read-ahead and readdir-ahead
>>>>>>>>>>> didn't
>>>>>>>>>>> help as well.
>>>>>>>>>>>
>>>>>>>>>>> Here are volume memory stats:
>>>>>>>>>>>
>>>>>>>>>>> ===
>>>>>>>>>>> Memory status for volume : mail
>>>>>>>>>>> ----------------------------------------------
>>>>>>>>>>> Brick : server1.xxx:/bricks/r6sdLV08_vd1_mail/mail
>>>>>>>>>>> Mallinfo
>>>>>>>>>>> --------
>>>>>>>>>>> Arena    : 36859904
>>>>>>>>>>> Ordblks  : 10357
>>>>>>>>>>> Smblks   : 519
>>>>>>>>>>> Hblks    : 21
>>>>>>>>>>> Hblkhd   : 30515200
>>>>>>>>>>> Usmblks  : 0
>>>>>>>>>>> Fsmblks  : 53440
>>>>>>>>>>> Uordblks : 18604144
>>>>>>>>>>> Fordblks : 18255760
>>>>>>>>>>> Keepcost : 114112
>>>>>>>>>>>
>>>>>>>>>>> Mempool Stats
>>>>>>>>>>> -------------
>>>>>>>>>>> Name                            HotCount ColdCount PaddedSizeof
>>>>>>>>>>> AllocCount MaxAlloc   Misses Max-StdAlloc
>>>>>>>>>>> ----                            -------- --------- ------------
>>>>>>>>>>> ---------- -------- -------- ------------
>>>>>>>>>>> mail-server:fd_t                       0      1024 108
>>>>>>>>>>> 30773120      137        0            0
>>>>>>>>>>> mail-server:dentry_t               16110       274 84
>>>>>>>>>>> 235676148    16384  1106499         1152
>>>>>>>>>>> mail-server:inode_t                16363        21 156
>>>>>>>>>>> 237216876    16384  1876651         1169
>>>>>>>>>>> mail-trash:fd_t                        0      1024 108
>>>>>>>>>>>
>>>>>>>>>>>     0        0        0            0
>>>>>>>>>>>
>>>>>>>>>>> mail-trash:dentry_t                    0     32768 84
>>>>>>>>>>>
>>>>>>>>>>>     0        0        0            0
>>>>>>>>>>>
>>>>>>>>>>> mail-trash:inode_t                     4     32764 156
>>>>>>>>>>>
>>>>>>>>>>>     4        4        0            0
>>>>>>>>>>>
>>>>>>>>>>> mail-trash:trash_local_t               0        64 8628
>>>>>>>>>>>
>>>>>>>>>>>     0        0        0            0
>>>>>>>>>>>
>>>>>>>>>>> mail-changetimerecorder:gf_ctr_local_t         0        64
>>>>>>>>>>> 16540          0        0        0            0
>>>>>>>>>>> mail-changelog:rpcsvc_request_t         0         8 2828
>>>>>>>>>>>
>>>>>>>>>>>      0        0        0            0
>>>>>>>>>>>
>>>>>>>>>>> mail-changelog:changelog_local_t         0        64 116
>>>>>>>>>>>
>>>>>>>>>>>       0        0        0            0
>>>>>>>>>>>
>>>>>>>>>>> mail-bitrot-stub:br_stub_local_t         0       512 84
>>>>>>>>>>> 79204        4        0            0
>>>>>>>>>>> mail-locks:pl_local_t                  0        32 148
>>>>>>>>>>> 6812757        4        0            0
>>>>>>>>>>> mail-upcall:upcall_local_t             0       512 108
>>>>>>>>>>>
>>>>>>>>>>>     0        0        0            0
>>>>>>>>>>>
>>>>>>>>>>> mail-marker:marker_local_t             0       128 332
>>>>>>>>>>> 64980        3        0            0
>>>>>>>>>>> mail-quota:quota_local_t               0        64 476
>>>>>>>>>>>
>>>>>>>>>>>     0        0        0            0
>>>>>>>>>>>
>>>>>>>>>>> mail-server:rpcsvc_request_t           0       512 2828
>>>>>>>>>>> 45462533       34        0            0
>>>>>>>>>>> glusterfs:struct saved_frame           0         8 124
>>>>>>>>>>>
>>>>>>>>>>>     2        2        0            0
>>>>>>>>>>>
>>>>>>>>>>> glusterfs:struct rpc_req               0         8 588
>>>>>>>>>>>
>>>>>>>>>>>     2        2        0            0
>>>>>>>>>>>
>>>>>>>>>>> glusterfs:rpcsvc_request_t             1         7 2828
>>>>>>>>>>>
>>>>>>>>>>>     2        1        0            0
>>>>>>>>>>>
>>>>>>>>>>> glusterfs:log_buf_t                    5       251 140
>>>>>>>>>>> 3452        6        0            0
>>>>>>>>>>> glusterfs:data_t                     242     16141 52
>>>>>>>>>>> 480115498      664        0            0
>>>>>>>>>>> glusterfs:data_pair_t                230     16153 68
>>>>>>>>>>> 179483528      275        0            0
>>>>>>>>>>> glusterfs:dict_t                      23      4073 140
>>>>>>>>>>> 303751675      627        0            0
>>>>>>>>>>> glusterfs:call_stub_t                  0      1024 3764
>>>>>>>>>>> 45290655       34        0            0
>>>>>>>>>>> glusterfs:call_stack_t                 1      1023 1708
>>>>>>>>>>> 43598469       34        0            0
>>>>>>>>>>> glusterfs:call_frame_t                 1      4095 172
>>>>>>>>>>> 336219655      184        0            0
>>>>>>>>>>> ----------------------------------------------
>>>>>>>>>>> Brick : server2.xxx:/bricks/r6sdLV07_vd0_mail/mail
>>>>>>>>>>> Mallinfo
>>>>>>>>>>> --------
>>>>>>>>>>> Arena    : 38174720
>>>>>>>>>>> Ordblks  : 9041
>>>>>>>>>>> Smblks   : 507
>>>>>>>>>>> Hblks    : 21
>>>>>>>>>>> Hblkhd   : 30515200
>>>>>>>>>>> Usmblks  : 0
>>>>>>>>>>> Fsmblks  : 51712
>>>>>>>>>>> Uordblks : 19415008
>>>>>>>>>>> Fordblks : 18759712
>>>>>>>>>>> Keepcost : 114848
>>>>>>>>>>>
>>>>>>>>>>> Mempool Stats
>>>>>>>>>>> -------------
>>>>>>>>>>> Name                            HotCount ColdCount PaddedSizeof
>>>>>>>>>>> AllocCount MaxAlloc   Misses Max-StdAlloc
>>>>>>>>>>> ----                            -------- --------- ------------
>>>>>>>>>>> ---------- -------- -------- ------------
>>>>>>>>>>> mail-server:fd_t                       0      1024 108
>>>>>>>>>>> 2373075      133        0            0
>>>>>>>>>>> mail-server:dentry_t               14114      2270 84
>>>>>>>>>>> 3513654    16384     2300          267
>>>>>>>>>>> mail-server:inode_t                16374        10 156
>>>>>>>>>>> 6766642    16384   194635         1279
>>>>>>>>>>> mail-trash:fd_t                        0      1024 108
>>>>>>>>>>>
>>>>>>>>>>>     0        0        0            0
>>>>>>>>>>>
>>>>>>>>>>> mail-trash:dentry_t                    0     32768 84
>>>>>>>>>>>
>>>>>>>>>>>     0        0        0            0
>>>>>>>>>>>
>>>>>>>>>>> mail-trash:inode_t                     4     32764 156
>>>>>>>>>>>
>>>>>>>>>>>     4        4        0            0
>>>>>>>>>>>
>>>>>>>>>>> mail-trash:trash_local_t               0        64 8628
>>>>>>>>>>>
>>>>>>>>>>>     0        0        0            0
>>>>>>>>>>>
>>>>>>>>>>> mail-changetimerecorder:gf_ctr_local_t         0        64
>>>>>>>>>>> 16540          0        0        0            0
>>>>>>>>>>> mail-changelog:rpcsvc_request_t         0         8 2828
>>>>>>>>>>>
>>>>>>>>>>>      0        0        0            0
>>>>>>>>>>>
>>>>>>>>>>> mail-changelog:changelog_local_t         0        64 116
>>>>>>>>>>>
>>>>>>>>>>>       0        0        0            0
>>>>>>>>>>>
>>>>>>>>>>> mail-bitrot-stub:br_stub_local_t         0       512 84
>>>>>>>>>>> 71354        4        0            0
>>>>>>>>>>> mail-locks:pl_local_t                  0        32 148
>>>>>>>>>>> 8135032        4        0            0
>>>>>>>>>>> mail-upcall:upcall_local_t             0       512 108
>>>>>>>>>>>
>>>>>>>>>>>     0        0        0            0
>>>>>>>>>>>
>>>>>>>>>>> mail-marker:marker_local_t             0       128 332
>>>>>>>>>>> 65005        3        0            0
>>>>>>>>>>> mail-quota:quota_local_t               0        64 476
>>>>>>>>>>>
>>>>>>>>>>>     0        0        0            0
>>>>>>>>>>>
>>>>>>>>>>> mail-server:rpcsvc_request_t           0       512 2828
>>>>>>>>>>> 12882393       30        0            0
>>>>>>>>>>> glusterfs:struct saved_frame           0         8 124
>>>>>>>>>>>
>>>>>>>>>>>     2        2        0            0
>>>>>>>>>>>
>>>>>>>>>>> glusterfs:struct rpc_req               0         8 588
>>>>>>>>>>>
>>>>>>>>>>>     2        2        0            0
>>>>>>>>>>>
>>>>>>>>>>> glusterfs:rpcsvc_request_t             1         7 2828
>>>>>>>>>>>
>>>>>>>>>>>     2        1        0            0
>>>>>>>>>>>
>>>>>>>>>>> glusterfs:log_buf_t                    5       251 140
>>>>>>>>>>> 3443        6        0            0
>>>>>>>>>>> glusterfs:data_t                     242     16141 52
>>>>>>>>>>> 138743429      290        0            0
>>>>>>>>>>> glusterfs:data_pair_t                230     16153 68
>>>>>>>>>>> 126649864      270        0            0
>>>>>>>>>>> glusterfs:dict_t                      23      4073 140
>>>>>>>>>>> 20356289       63        0            0
>>>>>>>>>>> glusterfs:call_stub_t                  0      1024 3764
>>>>>>>>>>> 13678560       31        0            0
>>>>>>>>>>> glusterfs:call_stack_t                 1      1023 1708
>>>>>>>>>>> 11011561       30        0            0
>>>>>>>>>>> glusterfs:call_frame_t                 1      4095 172
>>>>>>>>>>> 125764190      193        0            0
>>>>>>>>>>> ----------------------------------------------
>>>>>>>>>>> ===
>>>>>>>>>>>
>>>>>>>>>>> So, my questions are:
>>>>>>>>>>>
>>>>>>>>>>> 1) what one should do to limit GlusterFS FUSE client memory
>>>>>>>>>>> usage?
>>>>>>>>>>> 2) what one should do to prevent client high loadavg because
>>>>>>>>>>> of high
>>>>>>>>>>> iowait because of multiple concurrent volume users?
>>>>>>>>>>>
>>>>>>>>>>> Server/client OS is CentOS 7.1, GlusterFS server version is
>>>>>>>>>>> 3.7.3,
>>>>>>>>>>> GlusterFS client version is 3.7.4.
>>>>>>>>>>>
>>>>>>>>>>> Any additional info needed?
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Gluster-users mailing list
>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>>


More information about the Gluster-users mailing list