[Gluster-devel] how do you debug ref leaks?

[Kaleb KEITHLEY]

As a wishlist item, I think it'd be nice if debug builds (or some other 
build-time option) would disable the pools. Then valgrind might be more 
useful for finding leaks.

Maybe for GlusterFS-4.0?


On 09/18/2014 11:40 AM, Dan Lambright wrote:
> If we could disable/enable ref tracking dynamically, it may only be "heavy weight" tempoarily while the customer is being observed.
> You could get a state dump , or another idea is to take a core of the live process.   gcore $(pidof processname)
>
> ----- Original Message -----
>> From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
>> To: "Shyam" <srangana at redhat.com>, gluster-devel at gluster.org
>> Sent: Thursday, September 18, 2014 11:34:28 AM
>> Subject: Re: [Gluster-devel] how do you debug ref leaks?
>>
>>
>> On 09/18/2014 07:48 PM, Shyam wrote:
>>> On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote:
>>>> hi,
>>>>       Till now the only method I used to find ref leaks effectively is to
>>>> find what operation is causing ref leaks and read the code to find if
>>>> there is a ref-leak somewhere. Valgrind doesn't solve this problem
>>>> because it is reachable memory from inode-table etc. I am just wondering
>>>> if there is an effective way anyone else knows of. Do you guys think we
>>>> need a better mechanism of finding refleaks? At least which decreases
>>>> the search space significantly i.e. xlator y, fop f etc? It would be
>>>> better if we can come up with ways to integrate statedump and this infra
>>>> just like we did for mem-accounting.
>>>>
>>>> One way I thought was to introduce new apis called
>>>> xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops
>>>> per inode/dict/fd and increments/decrements accordingly. Dump this info
>>>> on statedump.
>>>>
>>>> I myself am not completely sure about this idea. It requires all xlators
>>>> to change.
>>>>
>>>> Any ideas?
>>>
>>> On a debug build we can use backtrace information stashed per ref and
>>> unref, this will give us history of refs taken and released. Which
>>> will also give the code path where ref was taken and released.
>>>
>>> It is heavy weight, so not for non-debug setups, but if a problem is
>>> reproducible this could be a quick way to check who is not releasing
>>> the ref's or have a history of the refs and unrefs to dig better into
>>> code.
>>>
>> Do you have any ideas for final builds also? Basically when users report
>> leaks it should not take us too long to figure out the problem area. We
>> should just ask them for statedump and should be able to figure out the
>> problem.
>>
>> Pranith
>>> Shyam
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at gluster.org
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-devel
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-devel
>>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-devel
>