[Gluster-devel] io-cache exceeding cache-size value

[Dan Parsons]

I will do this today. I noticed that I already have vm.drop_caches set to 3
via sysctl.conf, based on a suggestion from you from long ago. Should I
delete this under normal usage? Is it possible that this setting, enabled by
default, is causing my problems?

Dan


On Mon, Feb 23, 2009 at 3:23 PM, Amar Tumballi (bulde) <amar at gluster.com>wrote:

> Hi Gordan and Dan,
>
> It would help me a lot if its possible for you to get the info as described
> below,
>
> compile glusterfs like
>
> bash# make clean > /dev/null
> bash# make CFLAGS="-g -O0 -DDEBUG" > /dev/null
> bash# make install
>
> run the process which consumes memory (mostly client process) like below:
>
> bash# glusterfs <any argument you give generally> -N
> <this process will run in foreground now>
>
> Open another terminal
>
> bash# ps aux | grep glusterfs
> bash# kill -s SIGUSR1 <pid of glusterfs -N process>
> <Check in other terminal for memory usage stats>
>
> bash# <run your application over glusterfs as you do till you get high
> memory usage of glusterfs.. >
> bash# kill -s SIGUSR1 <pid of glusterfs -N process>
> <Check the stat in another terminal>
>
> bash# echo 3 > /proc/sys/vm/drop_caches
> bash# kill -s SIGUSR1 <pid of glusterfs -N process>
> <Check the stat in another terminal>
>
> Even after dropping caches, if 'in use bytes =' in malloc stats shows a
> high value, then it is a leak. If its showing less, but just 'system bytes =
> ' is a high value, this means glusterfs is not really consuming high memory,
> but the problem is really in the memory allocation segments.
>
> Regards,
> Amar
>
> NOTE: 'malloc_stats' will be printed to 'stdout' if we enable -DDEBUG while
> compiling glusterfs, as it hits performance badly otherwise.
>
>
> 2009/2/23 Gordan Bobic <gordan at bobich.net>
>
> Dan Parsons wrote:
>>
>>> I'm having an issue with glusterfs exceeding its cache-size value. Right
>>> now I have it set to 4000MB and I've seen it climb as high as 4800MB. If I
>>> set it to 5000, I've seen it go as high as 6000MB. This is a problem because
>>> it causes me to set the value very low so that my apps don't get pushed into
>>> swap. Is there any way to fix this? To get it to stick to the limit I set
>>> and not exceed?
>>>
>>
>> It's possible you are running into the same memory leak that I'm seeing,
>> and I'm not using io-cache or any other performance translators at all. With
>> rootfs on Gluster, doing a kernel compile (kernel source tree being on NFS,
>> so this won't be contributing to the bloat, hopefully) makes the glusterfsd
>> bloat by about 80MB per pass, and never frees it.
>>
>> Gordan
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at nongnu.org
>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
>
>
>
> --
> Amar Tumballi
> Gluster/GlusterFS Hacker
> [bulde on #gluster/irc.gnu.org]
> http://www.zresearch.com - Commoditizing Super Storage!
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20090224/ebf8c6dd/attachment-0003.html>