[Gluster-devel] glusterd crashing

Sat Mar 5 06:16:33 UTC 2016

Thanks for all the support.  After handling the input validation in my
code, Glusterd no longer crashes.  I am still waiting for clearance from my
superior to pass on all the details. Expecting him to revert by this Sunday.

- Ajil

On Fri, Mar 4, 2016 at 10:20 AM, Joseph Fernandes <josferna at redhat.com>
wrote:

> Well that may not be completely correct !
>
> Its  "gluster volume status all", unlike volume maintenance operation
> which are rare.
>
> Status can be issued multiple times in a day or might be put in a
> script/cron-job to check the health of the
> cluster.
> But anyways the fix is ready as the bug says.
>
> Crash is what we need to worry about.
>
> ~Joe
>
> ----- Original Message -----
> > From: "Atin Mukherjee" <amukherj at redhat.com>
> > To: "Joseph Fernandes" <josferna at redhat.com>, "Atin Mukherjee" <
> atin.mukherjee83 at gmail.com>
> > Cc: "Gluster Devel" <gluster-devel at gluster.org>, "Ajil Abraham" <
> ajil95.abraham at gmail.com>
> > Sent: Friday, March 4, 2016 9:37:43 AM
> > Subject: Re: [Gluster-devel] glusterd crashing
> >
> >
> >
> > On 03/04/2016 07:10 AM, Joseph Fernandes wrote:
> > > Might be this bug can give some context on the mem-leak (fix recently
> > > merged on master but not on 3.7.x)
> > >
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1287517
> > Yes, this is what we'd be fixing in 3.7.x too, but if you refer to [1]
> > the hike is seen when a command is run in a loop which is typically not
> > a use case in any production setup.
> >
> > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1287517#c15
> > >
> > > ~Joe
> > >
> > >
> > > ----- Original Message -----
> > >> From: "Atin Mukherjee" <atin.mukherjee83 at gmail.com>
> > >> To: "Joseph Fernandes" <josferna at redhat.com>
> > >> Cc: "Gluster Devel" <gluster-devel at gluster.org>, "Ajil Abraham"
> > >> <ajil95.abraham at gmail.com>
> > >> Sent: Friday, March 4, 2016 7:01:54 AM
> > >> Subject: Re: [Gluster-devel] glusterd crashing
> > >>
> > >> -Atin
> > >> Sent from one plus one
> > >> On 04-Mar-2016 6:12 am, "Joseph Fernandes" <josferna at redhat.com>
> wrote:
> > >>>
> > >>> Hi Ajil,
> > >>>
> > >>> Well few things,
> > >>>
> > >>> 1. Whenever you see a crash its better to send across the
> Backtrace(BT)
> > >> using gdb and attach the log files (or share it via some cloud drive)
> > >>>
> > >>> 2. About the memory leak, What kind of tools are you using for
> profiling
> > >> memory, valgrind ? if so please attach the valgrind reports.
> > >>>    $> glusterd --xlator-option *.run-with-valgrind=yes
> > >>>
> > >>> 3. Well I am not sure if glusterd uses any of the mempools as we do
> in
> > >> client and brick processes, Atin can shed some light on this.
> > >>>    Well In that case you can used the statedump mechanism check for
> > >> mem-leaks check the glusterfs/doc/debugging/statedump.md
> > >> GlusterD does use mempool and it has infra for capturing statedump as
> > >> well.
> > >> I am aware of few bytes of memory leaks in few paths which is really
> not a
> > >> huge concern but it shouldn't crash.
> > >>>
> > >>> Hope this helps
> > >>>
> > >>> ~Joe
> > >>>
> > >>>
> > >>> ----- Original Message -----
> > >>>> From: "Ajil Abraham" <ajil95.abraham at gmail.com>
> > >>>> To: "Atin Mukherjee" <atin.mukherjee83 at gmail.com>
> > >>>> Cc: "Gluster Devel" <gluster-devel at gluster.org>
> > >>>> Sent: Thursday, March 3, 2016 10:48:56 PM
> > >>>> Subject: Re: [Gluster-devel] glusterd crashing
> > >>>>
> > >>>> Hi Atin,
> > >>>>
> > >>>> The inputs I use are as per the requirements of a project I am
> working
> > >> on for
> > >>>> one of the large finance institutions in Dubai. I will try to
> handle the
> > >>>> input validation within my code. I uncovered some of the issues
> while
> > >> doing
> > >>>> a thorough testing of my code.
> > >>>>
> > >>>> I tried with 3.7.6 and also my own build from master branch. I will
> > >> check
> > >>>> with my superiors before sending you backtrace and other details. So
> > >> far, I
> > >>>> have seen memory leak in 100s of KBs.
> > >>>>
> > >>>> -Ajil
> > >>>>
> > >>>>
> > >>>> On Thu, Mar 3, 2016 at 10:17 PM, Atin Mukherjee <
> > >> atin.mukherjee83 at gmail.com
> > >>>>> wrote:
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> Hi Ajil,
> > >>>>
> > >>>> Its good to see that you are doing a thorough testing gluster. From
> > >> your mail
> > >>>> it looks like your automation focuses on mostly negative tests. I
> need
> > >> few
> > >>>> additional details to get to know whether they are known:
> > >>>>
> > >>>> 1. Version of gluster
> > >>>> 2. Backtrace of the crash along with reproducer
> > >>>> 3. Amount of memory leak in terms of bytes/KB/MB?? Have you already
> > >>>> identified them?
> > >>>>
> > >>>> -Atin
> > >>>> Sent from one plus one
> > >>>> On 03-Mar-2016 10:01 pm, "Ajil Abraham" < ajil95.abraham at gmail.com
> >
> > >> wrote:
> > >>>>
> > >>>>
> > >>>>
> > >>>> For my project, I am trying to do some automation using glusterd.
> It is
> > >> very
> > >>>> frustrating to see it crashing frequently. Looks like input
> validation
> > >> is
> > >>>> the culprit. I also see lot of buffer overflow and memory leak
> issues.
> > >>>> Making a note of these and will try to fix them. Surprised to see
> such
> > >> basic
> > >>>> issues still existing in Gluster.
> > >>>>
> > >>>> -Ajil
> > >>>>
> > >>>> _______________________________________________
> > >>>> Gluster-devel mailing list
> > >>>> Gluster-devel at gluster.org
> > >>>> http://www.gluster.org/mailman/listinfo/gluster-devel
> > >>>>
> > >>>>
> > >>>> _______________________________________________
> > >>>> Gluster-devel mailing list
> > >>>> Gluster-devel at gluster.org
> > >>>> http://www.gluster.org/mailman/listinfo/gluster-devel
> > >>
> > > _______________________________________________
> > > Gluster-devel mailing list
> > > Gluster-devel at gluster.org
> > > http://www.gluster.org/mailman/listinfo/gluster-devel
> > >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160305/1c56b60f/attachment-0001.html>