<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 6, 2017 at 4:49 PM, Raghavendra G <span dir="ltr"><<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Wed, Sep 6, 2017 at 11:16 AM, Csaba Henk <span dir="ltr"><<a href="mailto:chenk@redhat.com" target="_blank">chenk@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thanks Du, nice bit of info! It made me wander about the following:<br>
<br>
- Could it be then the default answer we give to "glusterfs client<br>
high memory usage"<br>
type of complaints to set vfs_cache_pressure to 100 + x? <br></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
- And then x = ? Was there proper performance testing done to see how<br>
performance /<br>
mem consumtion changes in terms of vfs_cache_performace?<br></blockquote><div><br></div></span><div>I had a discussion with Manoj on this. One drawback with using vfs_cache_performance tunable is that its a dynamic algorithm which decides whether to purge from page cache or inode cache looking at the current memory pressure. An obvious drawback for glusterfs is that, various of caches of glusterfs are not visible to kernel (Memory consumed by Glusterfs gets reflected neither in page cache nor in inode cache). This _might_ result in algorithm working poorly.<br></div></div></div></div></blockquote><div><br></div><div>It does not seem to be a big concern. vfs_cache_pressure was brought up in context of the inode proliferation issue, and (unlike "various caches") the size of the gluster client's inode herd is duely reflected in kernel because of the one to one mapping of the inodes of the two (which is the very cause of the phenomenon).</div><div><br></div><div>Csaba</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><span class=""><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
- vfs_cache_pressure is an allover system tunable. If 100 + x is ideal<br>
for GlusterFS, can<br>
we take the courage to propose this? Is there no risk to trash other<br>
(disk-based)<br>
filesystems' performace?<br></blockquote><div><br></div></span><div>That's a valid point. Behavior of other filesystems would be a concern.</div><div><br></div><div>I've not really thought through this suggestion of tuning /proc/sys/vm tunables and I am not even an expert who knows what tunables are at our disposal. Just wanted to bring this idea to notice of wider audience.<br></div><div><div class="h5"><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span class="m_-8295816442716753624HOEnZb"><font color="#888888"><br>
Csaba<br>
</font></span><div class="m_-8295816442716753624HOEnZb"><div class="m_-8295816442716753624h5"><br>
On Wed, Sep 6, 2017 at 6:57 AM, Raghavendra G <<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>> wrote:<br>
> Another parallel effort could be trying to configure the number of<br>
> inodes/dentries cached by kernel VFS using /proc/sys/vm interface.<br>
><br>
> ==============================<wbr>==============================<wbr>==<br>
><br>
> vfs_cache_pressure<br>
> ------------------<br>
><br>
> This percentage value controls the tendency of the kernel to reclaim<br>
> the memory which is used for caching of directory and inode objects.<br>
><br>
> At the default value of vfs_cache_pressure=100 the kernel will attempt to<br>
> reclaim dentries and inodes at a "fair" rate with respect to pagecache and<br>
> swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to<br>
> prefer<br>
> to retain dentry and inode caches. When vfs_cache_pressure=0, the kernel<br>
> will<br>
> never reclaim dentries and inodes due to memory pressure and this can easily<br>
> lead to out-of-memory conditions. Increasing vfs_cache_pressure beyond 100<br>
> causes the kernel to prefer to reclaim dentries and inodes.<br>
><br>
> Increasing vfs_cache_pressure significantly beyond 100 may have negative<br>
> performance impact. Reclaim code needs to take various locks to find<br>
> freeable<br>
> directory and inode objects. With vfs_cache_pressure=1000, it will look for<br>
> ten times more freeable objects than there are.<br>
><br>
> Also we've an article for sysadmins which has a section:<br>
><br>
> <quote><br>
><br>
> With GlusterFS, many users with a lot of storage and many small files<br>
> easily end up using a lot of RAM on the server side due to<br>
> 'inode/dentry' caching, leading to decreased performance when the kernel<br>
> keeps crawling through data-structures on a 40GB RAM system. Changing<br>
> this value higher than 100 has helped many users to achieve fair caching<br>
> and more responsiveness from the kernel.<br>
><br>
> </quote><br>
><br>
> Complete article can be found at:<br>
> <a href="https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Linux%20Kernel%20Tuning/" rel="noreferrer" target="_blank">https://gluster.readthedocs.io<wbr>/en/latest/Administrator%20Gui<wbr>de/Linux%20Kernel%20Tuning/</a><br>
><br>
> regards,<br>
><br>
><br>
> On Tue, Sep 5, 2017 at 5:20 PM, Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>><br>
> wrote:<br>
>><br>
>> +gluster-devel<br>
>><br>
>> Ashish just spoke to me about need of GC of inodes due to some state in<br>
>> inode that is being proposed in EC. Hence adding more people to<br>
>> conversation.<br>
>><br>
>> > > On 4 September 2017 at 12:34, Csaba Henk <<a href="mailto:chenk@redhat.com" target="_blank">chenk@redhat.com</a>> wrote:<br>
>> > ><br>
>> > > > I don't know, depends on how sophisticated GC we need/want/can get<br>
>> > > > by. I<br>
>> > > > guess the complexity will be inherent, ie. that of the algorithm<br>
>> > > > chosen<br>
>> > > > and<br>
>> > > > how we address concurrency & performance impacts, but once that's<br>
>> > > > got<br>
>> > > > right<br>
>> > > > the other aspects of implementation won't be hard.<br>
>> > > ><br>
>> > > > Eg. would it be good just to maintain a simple LRU list?<br>
>> > > ><br>
>> ><br>
>> > Yes. I was also thinking of leveraging lru list. We can invalidate first<br>
>> > "n"<br>
>> > inodes from lru list of fuse inode table.<br>
>> ><br>
>> > ><br>
>> > > That might work for starters.<br>
>> > ><br>
>> > > ><br>
>> > > > Csaba<br>
>> > > ><br>
>> > > > On Mon, Sep 4, 2017 at 8:48 AM, Nithya Balachandran<br>
>> > > > <<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>><br>
>> > > > wrote:<br>
>> > > ><br>
>> > > >><br>
>> > > >><br>
>> > > >> On 4 September 2017 at 12:14, Csaba Henk <<a href="mailto:chenk@redhat.com" target="_blank">chenk@redhat.com</a>> wrote:<br>
>> > > >><br>
>> > > >>> Basically how I see the fuse invalidate calls as rescuers of<br>
>> > > >>> sanity.<br>
>> > > >>><br>
>> > > >>> Normally, when you have lot of certain kind of stuff that tends to<br>
>> > > >>> accumulate, the immediate thought is: let's set up some garbage<br>
>> > > >>> collection<br>
>> > > >>> mechanism, that will take care of keeping the accumulation at bay.<br>
>> > > >>> But<br>
>> > > >>> that's what doesn't work with inodes in a naive way, as they are<br>
>> > > >>> referenced<br>
>> > > >>> from kernel, so we have to keep them around until kernel tells us<br>
>> > > >>> it's<br>
>> > > >>> giving up its reference. However, with the fuse invalidate calls<br>
>> > > >>> we can<br>
>> > > >>> take the initiative and instruct the kernel: "hey, kernel, give up<br>
>> > > >>> your<br>
>> > > >>> references to this thing!"<br>
>> > > >>><br>
>> > > >>> So we are actually free to implement any kind of inode GC in<br>
>> > > >>> glusterfs,<br>
>> > > >>> just have to take care to add the proper callback to<br>
>> > > >>> fuse_invalidate_*<br>
>> > > >>> and<br>
>> > > >>> we are good to go.<br>
>> > > >>><br>
>> > > >>><br>
>> > > >> That sounds good and something we need to do in the near future. Is<br>
>> > > >> this<br>
>> > > >> something that is easy to implement?<br>
>> > > >><br>
>> > > >><br>
>> > > >>> Csaba<br>
>> > > >>><br>
>> > > >>> On Mon, Sep 4, 2017 at 7:00 AM, Nithya Balachandran<br>
>> > > >>> <<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a><br>
>> > > >>> > wrote:<br>
>> > > >>><br>
>> > > >>>><br>
>> > > >>>><br>
>> > > >>>> On 4 September 2017 at 10:25, Raghavendra Gowdappa<br>
>> > > >>>> <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a><br>
>> > > >>>> > wrote:<br>
>> > > >>>><br>
>> > > >>>>><br>
>> > > >>>>><br>
>> > > >>>>> ----- Original Message -----<br>
>> > > >>>>> > From: "Nithya Balachandran" <<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>><br>
>> > > >>>>> > Sent: Monday, September 4, 2017 10:19:37 AM<br>
>> > > >>>>> > Subject: Fuse mounts and inodes<br>
>> > > >>>>> ><br>
>> > > >>>>> > Hi,<br>
>> > > >>>>> ><br>
>> > > >>>>> > One of the reasons for the memory consumption in gluster fuse<br>
>> > > >>>>> > mounts<br>
>> > > >>>>> is the<br>
>> > > >>>>> > number of inodes in the table which are never kicked out.<br>
>> > > >>>>> ><br>
>> > > >>>>> > Is there any way to default to an entry-timeout and<br>
>> > > >>>>> attribute-timeout value<br>
>> > > >>>>> > while mounting Gluster using Fuse? Say 60s each so those<br>
>> > > >>>>> > entries<br>
>> > > >>>>> will be<br>
>> > > >>>>> > purged periodically?<br>
>> > > >>>>><br>
>> > > >>>>> Once the entry timeouts, inodes won't be purged. Kernel sends a<br>
>> > > >>>>> lookup<br>
>> > > >>>>> to revalidate the mapping of path to inode. AFAIK, reverse<br>
>> > > >>>>> invalidation<br>
>> > > >>>>> (see inode_invalidate) is the only way to make kernel forget<br>
>> > > >>>>> inodes/attributes.<br>
>> > > >>>>><br>
>> > > >>>>> Is that something that can be done from the Fuse mount ? Or is<br>
>> > > >>>>> this<br>
>> > > >>>> something that needs to be added to Fuse?<br>
>> > > >>>><br>
>> > > >>>>> ><br>
>> > > >>>>> > Regards,<br>
>> > > >>>>> > Nithya<br>
>> > > >>>>> ><br>
>> > > >>>>><br>
>> > > >>>><br>
>> > > >>>><br>
>> > > >>><br>
>> > > >><br>
>> > > ><br>
>> > ><br>
>> ><br>
>> ______________________________<wbr>_________________<br>
>> Gluster-devel mailing list<br>
>> <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
>> <a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-devel</a><br>
><br>
><br>
><br>
><br>
> --<br>
> Raghavendra G<br>
><br>
> ______________________________<wbr>_________________<br>
> Gluster-devel mailing list<br>
> <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> <a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-devel</a><br>
______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-devel</a><br>
</div></div></blockquote></div></div></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="m_-8295816442716753624gmail_signature" data-smartmail="gmail_signature">Raghavendra G<br></div>
</font></span></div></div>
</blockquote></div><br></div></div>