[Gluster-devel] glusterd crashing

[Joseph Fernandes]

Well that may not be completely correct !

Its  "gluster volume status all", unlike volume maintenance operation which are rare.

Status can be issued multiple times in a day or might be put in a script/cron-job to check the health of the
cluster.
But anyways the fix is ready as the bug says.

Crash is what we need to worry about.

~Joe 

----- Original Message -----
> From: "Atin Mukherjee" <amukherj at redhat.com>
> To: "Joseph Fernandes" <josferna at redhat.com>, "Atin Mukherjee" <atin.mukherjee83 at gmail.com>
> Cc: "Gluster Devel" <gluster-devel at gluster.org>, "Ajil Abraham" <ajil95.abraham at gmail.com>
> Sent: Friday, March 4, 2016 9:37:43 AM
> Subject: Re: [Gluster-devel] glusterd crashing
> 
> 
> 
> On 03/04/2016 07:10 AM, Joseph Fernandes wrote:
> > Might be this bug can give some context on the mem-leak (fix recently
> > merged on master but not on 3.7.x)
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1287517
> Yes, this is what we'd be fixing in 3.7.x too, but if you refer to [1]
> the hike is seen when a command is run in a loop which is typically not
> a use case in any production setup.
> 
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1287517#c15
> > 
> > ~Joe
> > 
> > 
> > ----- Original Message -----
> >> From: "Atin Mukherjee" <atin.mukherjee83 at gmail.com>
> >> To: "Joseph Fernandes" <josferna at redhat.com>
> >> Cc: "Gluster Devel" <gluster-devel at gluster.org>, "Ajil Abraham"
> >> <ajil95.abraham at gmail.com>
> >> Sent: Friday, March 4, 2016 7:01:54 AM
> >> Subject: Re: [Gluster-devel] glusterd crashing
> >>
> >> -Atin
> >> Sent from one plus one
> >> On 04-Mar-2016 6:12 am, "Joseph Fernandes" <josferna at redhat.com> wrote:
> >>>
> >>> Hi Ajil,
> >>>
> >>> Well few things,
> >>>
> >>> 1. Whenever you see a crash its better to send across the Backtrace(BT)
> >> using gdb and attach the log files (or share it via some cloud drive)
> >>>
> >>> 2. About the memory leak, What kind of tools are you using for profiling
> >> memory, valgrind ? if so please attach the valgrind reports.
> >>>    $> glusterd --xlator-option *.run-with-valgrind=yes
> >>>
> >>> 3. Well I am not sure if glusterd uses any of the mempools as we do in
> >> client and brick processes, Atin can shed some light on this.
> >>>    Well In that case you can used the statedump mechanism check for
> >> mem-leaks check the glusterfs/doc/debugging/statedump.md
> >> GlusterD does use mempool and it has infra for capturing statedump as
> >> well.
> >> I am aware of few bytes of memory leaks in few paths which is really not a
> >> huge concern but it shouldn't crash.
> >>>
> >>> Hope this helps
> >>>
> >>> ~Joe
> >>>
> >>>
> >>> ----- Original Message -----
> >>>> From: "Ajil Abraham" <ajil95.abraham at gmail.com>
> >>>> To: "Atin Mukherjee" <atin.mukherjee83 at gmail.com>
> >>>> Cc: "Gluster Devel" <gluster-devel at gluster.org>
> >>>> Sent: Thursday, March 3, 2016 10:48:56 PM
> >>>> Subject: Re: [Gluster-devel] glusterd crashing
> >>>>
> >>>> Hi Atin,
> >>>>
> >>>> The inputs I use are as per the requirements of a project I am working
> >> on for
> >>>> one of the large finance institutions in Dubai. I will try to handle the
> >>>> input validation within my code. I uncovered some of the issues while
> >> doing
> >>>> a thorough testing of my code.
> >>>>
> >>>> I tried with 3.7.6 and also my own build from master branch. I will
> >> check
> >>>> with my superiors before sending you backtrace and other details. So
> >> far, I
> >>>> have seen memory leak in 100s of KBs.
> >>>>
> >>>> -Ajil
> >>>>
> >>>>
> >>>> On Thu, Mar 3, 2016 at 10:17 PM, Atin Mukherjee <
> >> atin.mukherjee83 at gmail.com
> >>>>> wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Hi Ajil,
> >>>>
> >>>> Its good to see that you are doing a thorough testing gluster. From
> >> your mail
> >>>> it looks like your automation focuses on mostly negative tests. I need
> >> few
> >>>> additional details to get to know whether they are known:
> >>>>
> >>>> 1. Version of gluster
> >>>> 2. Backtrace of the crash along with reproducer
> >>>> 3. Amount of memory leak in terms of bytes/KB/MB?? Have you already
> >>>> identified them?
> >>>>
> >>>> -Atin
> >>>> Sent from one plus one
> >>>> On 03-Mar-2016 10:01 pm, "Ajil Abraham" < ajil95.abraham at gmail.com >
> >> wrote:
> >>>>
> >>>>
> >>>>
> >>>> For my project, I am trying to do some automation using glusterd. It is
> >> very
> >>>> frustrating to see it crashing frequently. Looks like input validation
> >> is
> >>>> the culprit. I also see lot of buffer overflow and memory leak issues.
> >>>> Making a note of these and will try to fix them. Surprised to see such
> >> basic
> >>>> issues still existing in Gluster.
> >>>>
> >>>> -Ajil
> >>>>
> >>>> _______________________________________________
> >>>> Gluster-devel mailing list
> >>>> Gluster-devel at gluster.org
> >>>> http://www.gluster.org/mailman/listinfo/gluster-devel
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Gluster-devel mailing list
> >>>> Gluster-devel at gluster.org
> >>>> http://www.gluster.org/mailman/listinfo/gluster-devel
> >>
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-devel
> > 
>