[Gluster-devel] glusterd crashing

[Atin Mukherjee]

On 03/04/2016 07:10 AM, Joseph Fernandes wrote:
> Might be this bug can give some context on the mem-leak (fix recently merged on master but not on 3.7.x)
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1287517
Yes, this is what we'd be fixing in 3.7.x too, but if you refer to [1]
the hike is seen when a command is run in a loop which is typically not
a use case in any production setup.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1287517#c15
> 
> ~Joe
> 
> 
> ----- Original Message -----
>> From: "Atin Mukherjee" <atin.mukherjee83 at gmail.com>
>> To: "Joseph Fernandes" <josferna at redhat.com>
>> Cc: "Gluster Devel" <gluster-devel at gluster.org>, "Ajil Abraham" <ajil95.abraham at gmail.com>
>> Sent: Friday, March 4, 2016 7:01:54 AM
>> Subject: Re: [Gluster-devel] glusterd crashing
>>
>> -Atin
>> Sent from one plus one
>> On 04-Mar-2016 6:12 am, "Joseph Fernandes" <josferna at redhat.com> wrote:
>>>
>>> Hi Ajil,
>>>
>>> Well few things,
>>>
>>> 1. Whenever you see a crash its better to send across the Backtrace(BT)
>> using gdb and attach the log files (or share it via some cloud drive)
>>>
>>> 2. About the memory leak, What kind of tools are you using for profiling
>> memory, valgrind ? if so please attach the valgrind reports.
>>>    $> glusterd --xlator-option *.run-with-valgrind=yes
>>>
>>> 3. Well I am not sure if glusterd uses any of the mempools as we do in
>> client and brick processes, Atin can shed some light on this.
>>>    Well In that case you can used the statedump mechanism check for
>> mem-leaks check the glusterfs/doc/debugging/statedump.md
>> GlusterD does use mempool and it has infra for capturing statedump as well.
>> I am aware of few bytes of memory leaks in few paths which is really not a
>> huge concern but it shouldn't crash.
>>>
>>> Hope this helps
>>>
>>> ~Joe
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Ajil Abraham" <ajil95.abraham at gmail.com>
>>>> To: "Atin Mukherjee" <atin.mukherjee83 at gmail.com>
>>>> Cc: "Gluster Devel" <gluster-devel at gluster.org>
>>>> Sent: Thursday, March 3, 2016 10:48:56 PM
>>>> Subject: Re: [Gluster-devel] glusterd crashing
>>>>
>>>> Hi Atin,
>>>>
>>>> The inputs I use are as per the requirements of a project I am working
>> on for
>>>> one of the large finance institutions in Dubai. I will try to handle the
>>>> input validation within my code. I uncovered some of the issues while
>> doing
>>>> a thorough testing of my code.
>>>>
>>>> I tried with 3.7.6 and also my own build from master branch. I will
>> check
>>>> with my superiors before sending you backtrace and other details. So
>> far, I
>>>> have seen memory leak in 100s of KBs.
>>>>
>>>> -Ajil
>>>>
>>>>
>>>> On Thu, Mar 3, 2016 at 10:17 PM, Atin Mukherjee <
>> atin.mukherjee83 at gmail.com
>>>>> wrote:
>>>>
>>>>
>>>>
>>>>
>>>> Hi Ajil,
>>>>
>>>> Its good to see that you are doing a thorough testing gluster. From
>> your mail
>>>> it looks like your automation focuses on mostly negative tests. I need
>> few
>>>> additional details to get to know whether they are known:
>>>>
>>>> 1. Version of gluster
>>>> 2. Backtrace of the crash along with reproducer
>>>> 3. Amount of memory leak in terms of bytes/KB/MB?? Have you already
>>>> identified them?
>>>>
>>>> -Atin
>>>> Sent from one plus one
>>>> On 03-Mar-2016 10:01 pm, "Ajil Abraham" < ajil95.abraham at gmail.com >
>> wrote:
>>>>
>>>>
>>>>
>>>> For my project, I am trying to do some automation using glusterd. It is
>> very
>>>> frustrating to see it crashing frequently. Looks like input validation
>> is
>>>> the culprit. I also see lot of buffer overflow and memory leak issues.
>>>> Making a note of these and will try to fix them. Surprised to see such
>> basic
>>>> issues still existing in Gluster.
>>>>
>>>> -Ajil
>>>>
>>>> _______________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>