[Gluster-devel] glusterd crashing

Wed Mar 9 17:46:01 UTC 2016

Thank you Joseph.  I fell sick and stayed out of action.  Hope to continue
end of this week.  Planning to subscribe to gluster-devel. Any criteria to
be met for approving my request?

-Ajil

On Sat, Mar 5, 2016 at 10:19 PM, Joseph Fernandes <josferna at redhat.com>
wrote:

>
> http://www.gluster.org/community/documentation/index.php/Archives/Development_Work_Flow
>
>
> http://www.gluster.org/community/documentation/index.php/Simplified_dev_workflow
>
> Leaving the fun of exploration to you :)
>
> ~Joe
>
> ----- Original Message -----
> > From: "Ajil Abraham" <ajil95.abraham at gmail.com>
> > To: "Atin Mukherjee" <atin.mukherjee83 at gmail.com>
> > Cc: "Joseph Fernandes" <josferna at redhat.com>, "Gluster Devel" <
> gluster-devel at gluster.org>
> > Sent: Saturday, March 5, 2016 10:06:37 PM
> > Subject: Re: [Gluster-devel] glusterd crashing
> >
> > Sure Atin.  I am itching to contribute code. But worried due to lack of
> > experience in sending patches. Can somebody please send me across how to
> do
> > this? Consider me a total newbie and please be as descriptive as possible
> > :).
> >
> > -Ajil
> >
> > On Sat, Mar 5, 2016 at 12:46 PM, Atin Mukherjee <
> atin.mukherjee83 at gmail.com>
> > wrote:
> >
> > > -Atin
> > > Sent from one plus one
> > > On 05-Mar-2016 11:46 am, "Ajil Abraham" <ajil95.abraham at gmail.com>
> wrote:
> > > >
> > > > Thanks for all the support.  After handling the input validation in
> my
> > > code, Glusterd no longer crashes.  I am still waiting for clearance
> from my
> > > superior to pass on all the details. Expecting him to revert by this
> > > Sunday.
> > > Great to know that and we appreciate your contribution, if you happen
> to
> > > find any issues feel free to send patches :)
> > > >
> > > > - Ajil
> > > >
> > > > On Fri, Mar 4, 2016 at 10:20 AM, Joseph Fernandes <
> josferna at redhat.com>
> > > wrote:
> > > >>
> > > >> Well that may not be completely correct !
> > > >>
> > > >> Its  "gluster volume status all", unlike volume maintenance
> operation
> > > which are rare.
> > > >>
> > > >> Status can be issued multiple times in a day or might be put in a
> > > script/cron-job to check the health of the
> > > >> cluster.
> > > >> But anyways the fix is ready as the bug says.
> > > >>
> > > >> Crash is what we need to worry about.
> > > >>
> > > >> ~Joe
> > > >>
> > > >> ----- Original Message -----
> > > >> > From: "Atin Mukherjee" <amukherj at redhat.com>
> > > >> > To: "Joseph Fernandes" <josferna at redhat.com>, "Atin Mukherjee" <
> > > atin.mukherjee83 at gmail.com>
> > > >> > Cc: "Gluster Devel" <gluster-devel at gluster.org>, "Ajil Abraham" <
> > > ajil95.abraham at gmail.com>
> > > >> > Sent: Friday, March 4, 2016 9:37:43 AM
> > > >> > Subject: Re: [Gluster-devel] glusterd crashing
> > > >> >
> > > >> >
> > > >> >
> > > >> > On 03/04/2016 07:10 AM, Joseph Fernandes wrote:
> > > >> > > Might be this bug can give some context on the mem-leak (fix
> > > recently
> > > >> > > merged on master but not on 3.7.x)
> > > >> > >
> > > >> > > https://bugzilla.redhat.com/show_bug.cgi?id=1287517
> > > >> > Yes, this is what we'd be fixing in 3.7.x too, but if you refer
> to [1]
> > > >> > the hike is seen when a command is run in a loop which is
> typically
> > > not
> > > >> > a use case in any production setup.
> > > >> >
> > > >> > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1287517#c15
> > > >> > >
> > > >> > > ~Joe
> > > >> > >
> > > >> > >
> > > >> > > ----- Original Message -----
> > > >> > >> From: "Atin Mukherjee" <atin.mukherjee83 at gmail.com>
> > > >> > >> To: "Joseph Fernandes" <josferna at redhat.com>
> > > >> > >> Cc: "Gluster Devel" <gluster-devel at gluster.org>, "Ajil
> Abraham"
> > > >> > >> <ajil95.abraham at gmail.com>
> > > >> > >> Sent: Friday, March 4, 2016 7:01:54 AM
> > > >> > >> Subject: Re: [Gluster-devel] glusterd crashing
> > > >> > >>
> > > >> > >> -Atin
> > > >> > >> Sent from one plus one
> > > >> > >> On 04-Mar-2016 6:12 am, "Joseph Fernandes" <
> josferna at redhat.com>
> > > wrote:
> > > >> > >>>
> > > >> > >>> Hi Ajil,
> > > >> > >>>
> > > >> > >>> Well few things,
> > > >> > >>>
> > > >> > >>> 1. Whenever you see a crash its better to send across the
> > > Backtrace(BT)
> > > >> > >> using gdb and attach the log files (or share it via some cloud
> > > drive)
> > > >> > >>>
> > > >> > >>> 2. About the memory leak, What kind of tools are you using for
> > > profiling
> > > >> > >> memory, valgrind ? if so please attach the valgrind reports.
> > > >> > >>>    $> glusterd --xlator-option *.run-with-valgrind=yes
> > > >> > >>>
> > > >> > >>> 3. Well I am not sure if glusterd uses any of the mempools as
> we
> > > do in
> > > >> > >> client and brick processes, Atin can shed some light on this.
> > > >> > >>>    Well In that case you can used the statedump mechanism
> check
> > > for
> > > >> > >> mem-leaks check the glusterfs/doc/debugging/statedump.md
> > > >> > >> GlusterD does use mempool and it has infra for capturing
> statedump
> > > as
> > > >> > >> well.
> > > >> > >> I am aware of few bytes of memory leaks in few paths which is
> > > really not a
> > > >> > >> huge concern but it shouldn't crash.
> > > >> > >>>
> > > >> > >>> Hope this helps
> > > >> > >>>
> > > >> > >>> ~Joe
> > > >> > >>>
> > > >> > >>>
> > > >> > >>> ----- Original Message -----
> > > >> > >>>> From: "Ajil Abraham" <ajil95.abraham at gmail.com>
> > > >> > >>>> To: "Atin Mukherjee" <atin.mukherjee83 at gmail.com>
> > > >> > >>>> Cc: "Gluster Devel" <gluster-devel at gluster.org>
> > > >> > >>>> Sent: Thursday, March 3, 2016 10:48:56 PM
> > > >> > >>>> Subject: Re: [Gluster-devel] glusterd crashing
> > > >> > >>>>
> > > >> > >>>> Hi Atin,
> > > >> > >>>>
> > > >> > >>>> The inputs I use are as per the requirements of a project I
> am
> > > working
> > > >> > >> on for
> > > >> > >>>> one of the large finance institutions in Dubai. I will try to
> > > handle the
> > > >> > >>>> input validation within my code. I uncovered some of the
> issues
> > > while
> > > >> > >> doing
> > > >> > >>>> a thorough testing of my code.
> > > >> > >>>>
> > > >> > >>>> I tried with 3.7.6 and also my own build from master branch.
> I
> > > will
> > > >> > >> check
> > > >> > >>>> with my superiors before sending you backtrace and other
> > > details. So
> > > >> > >> far, I
> > > >> > >>>> have seen memory leak in 100s of KBs.
> > > >> > >>>>
> > > >> > >>>> -Ajil
> > > >> > >>>>
> > > >> > >>>>
> > > >> > >>>> On Thu, Mar 3, 2016 at 10:17 PM, Atin Mukherjee <
> > > >> > >> atin.mukherjee83 at gmail.com
> > > >> > >>>>> wrote:
> > > >> > >>>>
> > > >> > >>>>
> > > >> > >>>>
> > > >> > >>>>
> > > >> > >>>> Hi Ajil,
> > > >> > >>>>
> > > >> > >>>> Its good to see that you are doing a thorough testing
> gluster.
> > > From
> > > >> > >> your mail
> > > >> > >>>> it looks like your automation focuses on mostly negative
> tests.
> > > I need
> > > >> > >> few
> > > >> > >>>> additional details to get to know whether they are known:
> > > >> > >>>>
> > > >> > >>>> 1. Version of gluster
> > > >> > >>>> 2. Backtrace of the crash along with reproducer
> > > >> > >>>> 3. Amount of memory leak in terms of bytes/KB/MB?? Have you
> > > already
> > > >> > >>>> identified them?
> > > >> > >>>>
> > > >> > >>>> -Atin
> > > >> > >>>> Sent from one plus one
> > > >> > >>>> On 03-Mar-2016 10:01 pm, "Ajil Abraham" <
> > > ajil95.abraham at gmail.com >
> > > >> > >> wrote:
> > > >> > >>>>
> > > >> > >>>>
> > > >> > >>>>
> > > >> > >>>> For my project, I am trying to do some automation using
> > > glusterd. It is
> > > >> > >> very
> > > >> > >>>> frustrating to see it crashing frequently. Looks like input
> > > validation
> > > >> > >> is
> > > >> > >>>> the culprit. I also see lot of buffer overflow and memory
> leak
> > > issues.
> > > >> > >>>> Making a note of these and will try to fix them. Surprised to
> > > see such
> > > >> > >> basic
> > > >> > >>>> issues still existing in Gluster.
> > > >> > >>>>
> > > >> > >>>> -Ajil
> > > >> > >>>>
> > > >> > >>>> _______________________________________________
> > > >> > >>>> Gluster-devel mailing list
> > > >> > >>>> Gluster-devel at gluster.org
> > > >> > >>>> http://www.gluster.org/mailman/listinfo/gluster-devel
> > > >> > >>>>
> > > >> > >>>>
> > > >> > >>>> _______________________________________________
> > > >> > >>>> Gluster-devel mailing list
> > > >> > >>>> Gluster-devel at gluster.org
> > > >> > >>>> http://www.gluster.org/mailman/listinfo/gluster-devel
> > > >> > >>
> > > >> > > _______________________________________________
> > > >> > > Gluster-devel mailing list
> > > >> > > Gluster-devel at gluster.org
> > > >> > > http://www.gluster.org/mailman/listinfo/gluster-devel
> > > >> > >
> > > >> >
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Gluster-devel mailing list
> > > > Gluster-devel at gluster.org
> > > > http://www.gluster.org/mailman/listinfo/gluster-devel
> > >
> > >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160309/53d879f6/attachment.html>