[Gluster-devel] inode list memory usage

[Shehjar Tikoo]

Mike Terzo wrote:
> I have a very simple glusterfs config setup. I have 2 tcp hosts configured as:
> 
> volume LABVOL
> 	type cluster/replicate
> 	option block-size 4MB
> 	subvolumes gfs1 gfs2
> end-volume
> 
> I have io-cache set to 16MBs (i know it's low, but i'm debugging
> memory usage).
> I'm using io-threads as well, and have that set to 2, again debugging
> memory usage.
> 
> I'm copying 100's of thousands of files and directories to my
> glusterfs mount point on a client box.  After about 2 days, the
> glusterfs process on the client had allocated 10G of memory.
> 
> I recompiled gluster (3.0.2) with -O0 -g and was able to reproduce the
> memory growth.  I ran the gluster mount inside valgrind to get an idea
> of where all the memory was going.  I found that in fuse-bridge.c that
> the inode table was getting created with a 0 value for the lru_limit,
> which is used to determine how many inodes to keep in the linked list
> (inode_table_prune).  This is what's blowing out the memory on my
> installation.  Is there a reason lru_limit is hard set to 0 when
> creating the inode_table for the client?

Thats because the kernel FUSE module is supposed to tell us when to free 
the inodes inside GlusterFS and in this case it does not want them 
freed. We know about this problem and there is work underway to change 
this behaviour in the FUSE module.

To force kernel to free up inodes, try this as root:

$ echo 2 > /proc/sys/vm/drop_caches

If that doesnt work, try:
$ sync
$ echo 2 > /proc/sys/vm/drop_caches

Beware that sync will force pending kernel buffers to disk so it might 
interfere with ongoing disk IO.

BTW, how much memory do you have on this system where GlusterFS consumed 
10G?


-Shehjar


> 
> thanks
> --mike terzo
> 
> 
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel