[Gluster-devel] glusterd crashing

Sat Mar 5 16:49:36 UTC 2016

http://www.gluster.org/community/documentation/index.php/Archives/Development_Work_Flow

http://www.gluster.org/community/documentation/index.php/Simplified_dev_workflow

Leaving the fun of exploration to you :)

~Joe

----- Original Message -----
> From: "Ajil Abraham" <ajil95.abraham at gmail.com>
> To: "Atin Mukherjee" <atin.mukherjee83 at gmail.com>
> Cc: "Joseph Fernandes" <josferna at redhat.com>, "Gluster Devel" <gluster-devel at gluster.org>
> Sent: Saturday, March 5, 2016 10:06:37 PM
> Subject: Re: [Gluster-devel] glusterd crashing
> 
> Sure Atin.  I am itching to contribute code. But worried due to lack of
> experience in sending patches. Can somebody please send me across how to do
> this? Consider me a total newbie and please be as descriptive as possible
> :).
> 
> -Ajil
> 
> On Sat, Mar 5, 2016 at 12:46 PM, Atin Mukherjee <atin.mukherjee83 at gmail.com>
> wrote:
> 
> > -Atin
> > Sent from one plus one
> > On 05-Mar-2016 11:46 am, "Ajil Abraham" <ajil95.abraham at gmail.com> wrote:
> > >
> > > Thanks for all the support.  After handling the input validation in my
> > code, Glusterd no longer crashes.  I am still waiting for clearance from my
> > superior to pass on all the details. Expecting him to revert by this
> > Sunday.
> > Great to know that and we appreciate your contribution, if you happen to
> > find any issues feel free to send patches :)
> > >
> > > - Ajil
> > >
> > > On Fri, Mar 4, 2016 at 10:20 AM, Joseph Fernandes <josferna at redhat.com>
> > wrote:
> > >>
> > >> Well that may not be completely correct !
> > >>
> > >> Its  "gluster volume status all", unlike volume maintenance operation
> > which are rare.
> > >>
> > >> Status can be issued multiple times in a day or might be put in a
> > script/cron-job to check the health of the
> > >> cluster.
> > >> But anyways the fix is ready as the bug says.
> > >>
> > >> Crash is what we need to worry about.
> > >>
> > >> ~Joe
> > >>
> > >> ----- Original Message -----
> > >> > From: "Atin Mukherjee" <amukherj at redhat.com>
> > >> > To: "Joseph Fernandes" <josferna at redhat.com>, "Atin Mukherjee" <
> > atin.mukherjee83 at gmail.com>
> > >> > Cc: "Gluster Devel" <gluster-devel at gluster.org>, "Ajil Abraham" <
> > ajil95.abraham at gmail.com>
> > >> > Sent: Friday, March 4, 2016 9:37:43 AM
> > >> > Subject: Re: [Gluster-devel] glusterd crashing
> > >> >
> > >> >
> > >> >
> > >> > On 03/04/2016 07:10 AM, Joseph Fernandes wrote:
> > >> > > Might be this bug can give some context on the mem-leak (fix
> > recently
> > >> > > merged on master but not on 3.7.x)
> > >> > >
> > >> > > https://bugzilla.redhat.com/show_bug.cgi?id=1287517
> > >> > Yes, this is what we'd be fixing in 3.7.x too, but if you refer to [1]
> > >> > the hike is seen when a command is run in a loop which is typically
> > not
> > >> > a use case in any production setup.
> > >> >
> > >> > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1287517#c15
> > >> > >
> > >> > > ~Joe
> > >> > >
> > >> > >
> > >> > > ----- Original Message -----
> > >> > >> From: "Atin Mukherjee" <atin.mukherjee83 at gmail.com>
> > >> > >> To: "Joseph Fernandes" <josferna at redhat.com>
> > >> > >> Cc: "Gluster Devel" <gluster-devel at gluster.org>, "Ajil Abraham"
> > >> > >> <ajil95.abraham at gmail.com>
> > >> > >> Sent: Friday, March 4, 2016 7:01:54 AM
> > >> > >> Subject: Re: [Gluster-devel] glusterd crashing
> > >> > >>
> > >> > >> -Atin
> > >> > >> Sent from one plus one
> > >> > >> On 04-Mar-2016 6:12 am, "Joseph Fernandes" <josferna at redhat.com>
> > wrote:
> > >> > >>>
> > >> > >>> Hi Ajil,
> > >> > >>>
> > >> > >>> Well few things,
> > >> > >>>
> > >> > >>> 1. Whenever you see a crash its better to send across the
> > Backtrace(BT)
> > >> > >> using gdb and attach the log files (or share it via some cloud
> > drive)
> > >> > >>>
> > >> > >>> 2. About the memory leak, What kind of tools are you using for
> > profiling
> > >> > >> memory, valgrind ? if so please attach the valgrind reports.
> > >> > >>>    $> glusterd --xlator-option *.run-with-valgrind=yes
> > >> > >>>
> > >> > >>> 3. Well I am not sure if glusterd uses any of the mempools as we
> > do in
> > >> > >> client and brick processes, Atin can shed some light on this.
> > >> > >>>    Well In that case you can used the statedump mechanism check
> > for
> > >> > >> mem-leaks check the glusterfs/doc/debugging/statedump.md
> > >> > >> GlusterD does use mempool and it has infra for capturing statedump
> > as
> > >> > >> well.
> > >> > >> I am aware of few bytes of memory leaks in few paths which is
> > really not a
> > >> > >> huge concern but it shouldn't crash.
> > >> > >>>
> > >> > >>> Hope this helps
> > >> > >>>
> > >> > >>> ~Joe
> > >> > >>>
> > >> > >>>
> > >> > >>> ----- Original Message -----
> > >> > >>>> From: "Ajil Abraham" <ajil95.abraham at gmail.com>
> > >> > >>>> To: "Atin Mukherjee" <atin.mukherjee83 at gmail.com>
> > >> > >>>> Cc: "Gluster Devel" <gluster-devel at gluster.org>
> > >> > >>>> Sent: Thursday, March 3, 2016 10:48:56 PM
> > >> > >>>> Subject: Re: [Gluster-devel] glusterd crashing
> > >> > >>>>
> > >> > >>>> Hi Atin,
> > >> > >>>>
> > >> > >>>> The inputs I use are as per the requirements of a project I am
> > working
> > >> > >> on for
> > >> > >>>> one of the large finance institutions in Dubai. I will try to
> > handle the
> > >> > >>>> input validation within my code. I uncovered some of the issues
> > while
> > >> > >> doing
> > >> > >>>> a thorough testing of my code.
> > >> > >>>>
> > >> > >>>> I tried with 3.7.6 and also my own build from master branch. I
> > will
> > >> > >> check
> > >> > >>>> with my superiors before sending you backtrace and other
> > details. So
> > >> > >> far, I
> > >> > >>>> have seen memory leak in 100s of KBs.
> > >> > >>>>
> > >> > >>>> -Ajil
> > >> > >>>>
> > >> > >>>>
> > >> > >>>> On Thu, Mar 3, 2016 at 10:17 PM, Atin Mukherjee <
> > >> > >> atin.mukherjee83 at gmail.com
> > >> > >>>>> wrote:
> > >> > >>>>
> > >> > >>>>
> > >> > >>>>
> > >> > >>>>
> > >> > >>>> Hi Ajil,
> > >> > >>>>
> > >> > >>>> Its good to see that you are doing a thorough testing gluster.
> > From
> > >> > >> your mail
> > >> > >>>> it looks like your automation focuses on mostly negative tests.
> > I need
> > >> > >> few
> > >> > >>>> additional details to get to know whether they are known:
> > >> > >>>>
> > >> > >>>> 1. Version of gluster
> > >> > >>>> 2. Backtrace of the crash along with reproducer
> > >> > >>>> 3. Amount of memory leak in terms of bytes/KB/MB?? Have you
> > already
> > >> > >>>> identified them?
> > >> > >>>>
> > >> > >>>> -Atin
> > >> > >>>> Sent from one plus one
> > >> > >>>> On 03-Mar-2016 10:01 pm, "Ajil Abraham" <
> > ajil95.abraham at gmail.com >
> > >> > >> wrote:
> > >> > >>>>
> > >> > >>>>
> > >> > >>>>
> > >> > >>>> For my project, I am trying to do some automation using
> > glusterd. It is
> > >> > >> very
> > >> > >>>> frustrating to see it crashing frequently. Looks like input
> > validation
> > >> > >> is
> > >> > >>>> the culprit. I also see lot of buffer overflow and memory leak
> > issues.
> > >> > >>>> Making a note of these and will try to fix them. Surprised to
> > see such
> > >> > >> basic
> > >> > >>>> issues still existing in Gluster.
> > >> > >>>>
> > >> > >>>> -Ajil
> > >> > >>>>
> > >> > >>>> _______________________________________________
> > >> > >>>> Gluster-devel mailing list
> > >> > >>>> Gluster-devel at gluster.org
> > >> > >>>> http://www.gluster.org/mailman/listinfo/gluster-devel
> > >> > >>>>
> > >> > >>>>
> > >> > >>>> _______________________________________________
> > >> > >>>> Gluster-devel mailing list
> > >> > >>>> Gluster-devel at gluster.org
> > >> > >>>> http://www.gluster.org/mailman/listinfo/gluster-devel
> > >> > >>
> > >> > > _______________________________________________
> > >> > > Gluster-devel mailing list
> > >> > > Gluster-devel at gluster.org
> > >> > > http://www.gluster.org/mailman/listinfo/gluster-devel
> > >> > >
> > >> >
> > >
> > >
> > >
> > > _______________________________________________
> > > Gluster-devel mailing list
> > > Gluster-devel at gluster.org
> > > http://www.gluster.org/mailman/listinfo/gluster-devel
> >
> >
>