<div dir="ltr">Another parallel effort could be trying to configure the number of inodes/dentries cached by kernel VFS using /proc/sys/vm interface.<br><br><pre>==============================================================
vfs_cache_pressure
------------------
This percentage value controls the tendency of the kernel to reclaim
the memory which is used for caching of directory and inode objects.
At the default value of vfs_cache_pressure=100 the kernel will attempt to
reclaim dentries and inodes at a "fair" rate with respect to pagecache and
swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer
to retain dentry and inode caches. When vfs_cache_pressure=0, the kernel will
never reclaim dentries and inodes due to memory pressure and this can easily
lead to out-of-memory conditions. Increasing vfs_cache_pressure beyond 100
causes the kernel to prefer to reclaim dentries and inodes.
Increasing vfs_cache_pressure significantly beyond 100 may have negative
performance impact. Reclaim code needs to take various locks to find freeable
directory and inode objects. With vfs_cache_pressure=1000, it will look for
ten times more freeable objects than there are.<br><br></pre><pre>Also we've an article for sysadmins which has a section:<br><br><quote><br></pre><pre>With GlusterFS, many users with a lot of storage and many small files
easily end up using a lot of RAM on the server side due to
'inode/dentry' caching, leading to decreased performance when the kernel
keeps crawling through data-structures on a 40GB RAM system. Changing
this value higher than 100 has helped many users to achieve fair caching
and more responsiveness from the kernel.<br><br></quote><br><br></pre><pre>Complete article can be found at:<br><a href="https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Linux%20Kernel%20Tuning/">https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Linux%20Kernel%20Tuning/</a><br><br></pre><pre>regards,<br></pre><pre></pre></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 5, 2017 at 5:20 PM, Raghavendra Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">+gluster-devel<br>
<br>
Ashish just spoke to me about need of GC of inodes due to some state in inode that is being proposed in EC. Hence adding more people to conversation.<br>
<br>
> > On 4 September 2017 at 12:34, Csaba Henk <<a href="mailto:chenk@redhat.com">chenk@redhat.com</a>> wrote:<br>
> ><br>
> > > I don't know, depends on how sophisticated GC we need/want/can get by. I<br>
> > > guess the complexity will be inherent, ie. that of the algorithm chosen<br>
> > > and<br>
> > > how we address concurrency & performance impacts, but once that's got<br>
> > > right<br>
> > > the other aspects of implementation won't be hard.<br>
> > ><br>
> > > Eg. would it be good just to maintain a simple LRU list?<br>
> > ><br>
><br>
> Yes. I was also thinking of leveraging lru list. We can invalidate first "n"<br>
> inodes from lru list of fuse inode table.<br>
><br>
> ><br>
> > That might work for starters.<br>
> ><br>
> > ><br>
> > > Csaba<br>
> > ><br>
> > > On Mon, Sep 4, 2017 at 8:48 AM, Nithya Balachandran <<a href="mailto:nbalacha@redhat.com">nbalacha@redhat.com</a>><br>
> > > wrote:<br>
> > ><br>
> > >><br>
> > >><br>
> > >> On 4 September 2017 at 12:14, Csaba Henk <<a href="mailto:chenk@redhat.com">chenk@redhat.com</a>> wrote:<br>
> > >><br>
> > >>> Basically how I see the fuse invalidate calls as rescuers of sanity.<br>
> > >>><br>
> > >>> Normally, when you have lot of certain kind of stuff that tends to<br>
> > >>> accumulate, the immediate thought is: let's set up some garbage<br>
> > >>> collection<br>
> > >>> mechanism, that will take care of keeping the accumulation at bay. But<br>
> > >>> that's what doesn't work with inodes in a naive way, as they are<br>
> > >>> referenced<br>
> > >>> from kernel, so we have to keep them around until kernel tells us it's<br>
> > >>> giving up its reference. However, with the fuse invalidate calls we can<br>
> > >>> take the initiative and instruct the kernel: "hey, kernel, give up your<br>
> > >>> references to this thing!"<br>
> > >>><br>
> > >>> So we are actually free to implement any kind of inode GC in glusterfs,<br>
> > >>> just have to take care to add the proper callback to fuse_invalidate_*<br>
> > >>> and<br>
> > >>> we are good to go.<br>
> > >>><br>
> > >>><br>
> > >> That sounds good and something we need to do in the near future. Is this<br>
> > >> something that is easy to implement?<br>
> > >><br>
> > >><br>
> > >>> Csaba<br>
> > >>><br>
> > >>> On Mon, Sep 4, 2017 at 7:00 AM, Nithya Balachandran<br>
> > >>> <<a href="mailto:nbalacha@redhat.com">nbalacha@redhat.com</a><br>
> > >>> > wrote:<br>
> > >>><br>
> > >>>><br>
> > >>>><br>
> > >>>> On 4 September 2017 at 10:25, Raghavendra Gowdappa<br>
> > >>>> <<a href="mailto:rgowdapp@redhat.com">rgowdapp@redhat.com</a><br>
> > >>>> > wrote:<br>
> > >>>><br>
> > >>>>><br>
> > >>>>><br>
> > >>>>> ----- Original Message -----<br>
> > >>>>> > From: "Nithya Balachandran" <<a href="mailto:nbalacha@redhat.com">nbalacha@redhat.com</a>><br>
> > >>>>> > Sent: Monday, September 4, 2017 10:19:37 AM<br>
> > >>>>> > Subject: Fuse mounts and inodes<br>
> > >>>>> ><br>
> > >>>>> > Hi,<br>
> > >>>>> ><br>
> > >>>>> > One of the reasons for the memory consumption in gluster fuse<br>
> > >>>>> > mounts<br>
> > >>>>> is the<br>
> > >>>>> > number of inodes in the table which are never kicked out.<br>
> > >>>>> ><br>
> > >>>>> > Is there any way to default to an entry-timeout and<br>
> > >>>>> attribute-timeout value<br>
> > >>>>> > while mounting Gluster using Fuse? Say 60s each so those entries<br>
> > >>>>> will be<br>
> > >>>>> > purged periodically?<br>
> > >>>>><br>
> > >>>>> Once the entry timeouts, inodes won't be purged. Kernel sends a<br>
> > >>>>> lookup<br>
> > >>>>> to revalidate the mapping of path to inode. AFAIK, reverse<br>
> > >>>>> invalidation<br>
> > >>>>> (see inode_invalidate) is the only way to make kernel forget<br>
> > >>>>> inodes/attributes.<br>
> > >>>>><br>
> > >>>>> Is that something that can be done from the Fuse mount ? Or is this<br>
> > >>>> something that needs to be added to Fuse?<br>
> > >>>><br>
> > >>>>> ><br>
> > >>>>> > Regards,<br>
> > >>>>> > Nithya<br>
> > >>>>> ><br>
> > >>>>><br>
> > >>>><br>
> > >>>><br>
> > >>><br>
> > >><br>
> > ><br>
> ><br>
><br>
______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Raghavendra G<br></div>
</div>