[Gluster-devel] how do you debug ref leaks?

[Pranith Kumar Karampuri]

On 09/18/2014 09:31 PM, Kaleb KEITHLEY wrote:
> As a wishlist item, I think it'd be nice if debug builds (or some 
> other build-time option) would disable the pools. Then valgrind might 
> be more useful for finding leaks.
>
> Maybe for GlusterFS-4.0?
This is already available http://review.gluster.org/7835

Pranith
>
>
> On 09/18/2014 11:40 AM, Dan Lambright wrote:
>> If we could disable/enable ref tracking dynamically, it may only be 
>> "heavy weight" tempoarily while the customer is being observed.
>> You could get a state dump , or another idea is to take a core of the 
>> live process.   gcore $(pidof processname)
>>
>> ----- Original Message -----
>>> From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
>>> To: "Shyam" <srangana at redhat.com>, gluster-devel at gluster.org
>>> Sent: Thursday, September 18, 2014 11:34:28 AM
>>> Subject: Re: [Gluster-devel] how do you debug ref leaks?
>>>
>>>
>>> On 09/18/2014 07:48 PM, Shyam wrote:
>>>> On 09/17/2014 10:13 PM, Pranith Kumar Karampuri wrote:
>>>>> hi,
>>>>>       Till now the only method I used to find ref leaks 
>>>>> effectively is to
>>>>> find what operation is causing ref leaks and read the code to find if
>>>>> there is a ref-leak somewhere. Valgrind doesn't solve this problem
>>>>> because it is reachable memory from inode-table etc. I am just 
>>>>> wondering
>>>>> if there is an effective way anyone else knows of. Do you guys 
>>>>> think we
>>>>> need a better mechanism of finding refleaks? At least which decreases
>>>>> the search space significantly i.e. xlator y, fop f etc? It would be
>>>>> better if we can come up with ways to integrate statedump and this 
>>>>> infra
>>>>> just like we did for mem-accounting.
>>>>>
>>>>> One way I thought was to introduce new apis called
>>>>> xl_fop_dict/inode/fd_ref/unref (). Each xl keeps an array of num_fops
>>>>> per inode/dict/fd and increments/decrements accordingly. Dump this 
>>>>> info
>>>>> on statedump.
>>>>>
>>>>> I myself am not completely sure about this idea. It requires all 
>>>>> xlators
>>>>> to change.
>>>>>
>>>>> Any ideas?
>>>>
>>>> On a debug build we can use backtrace information stashed per ref and
>>>> unref, this will give us history of refs taken and released. Which
>>>> will also give the code path where ref was taken and released.
>>>>
>>>> It is heavy weight, so not for non-debug setups, but if a problem is
>>>> reproducible this could be a quick way to check who is not releasing
>>>> the ref's or have a history of the refs and unrefs to dig better into
>>>> code.
>>>>
>>> Do you have any ideas for final builds also? Basically when users 
>>> report
>>> leaks it should not take us too long to figure out the problem area. We
>>> should just ask them for statedump and should be able to figure out the
>>> problem.
>>>
>>> Pranith
>>>> Shyam
>>>> _______________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org
>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-devel
>>>
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at gluster.org
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-devel
>>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-devel
>>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-devel