[Gluster-devel] glusterd crashing
Atin Mukherjee
amukherj at redhat.com
Thu Mar 10 03:42:48 UTC 2016
On 03/09/2016 11:16 PM, Ajil Abraham wrote:
> Thank you Joseph. I fell sick and stayed out of action. Hope to
> continue end of this week. Planning to subscribe to gluster-devel. Any
> criteria to be met for approving my request?
No, you can subscribe to the list.
>
> -Ajil
>
> On Sat, Mar 5, 2016 at 10:19 PM, Joseph Fernandes <josferna at redhat.com
> <mailto:josferna at redhat.com>> wrote:
>
> http://www.gluster.org/community/documentation/index.php/Archives/Development_Work_Flow
>
> http://www.gluster.org/community/documentation/index.php/Simplified_dev_workflow
>
> Leaving the fun of exploration to you :)
>
> ~Joe
>
> ----- Original Message -----
> > From: "Ajil Abraham" <ajil95.abraham at gmail.com <mailto:ajil95.abraham at gmail.com>>
> > To: "Atin Mukherjee" <atin.mukherjee83 at gmail.com <mailto:atin.mukherjee83 at gmail.com>>
> > Cc: "Joseph Fernandes" <josferna at redhat.com
> <mailto:josferna at redhat.com>>, "Gluster Devel"
> <gluster-devel at gluster.org <mailto:gluster-devel at gluster.org>>
> > Sent: Saturday, March 5, 2016 10:06:37 PM
> > Subject: Re: [Gluster-devel] glusterd crashing
> >
> > Sure Atin. I am itching to contribute code. But worried due to
> lack of
> > experience in sending patches. Can somebody please send me across
> how to do
> > this? Consider me a total newbie and please be as descriptive as
> possible
> > :).
> >
> > -Ajil
> >
> > On Sat, Mar 5, 2016 at 12:46 PM, Atin Mukherjee
> <atin.mukherjee83 at gmail.com <mailto:atin.mukherjee83 at gmail.com>>
> > wrote:
> >
> > > -Atin
> > > Sent from one plus one
> > > On 05-Mar-2016 11:46 am, "Ajil Abraham"
> <ajil95.abraham at gmail.com <mailto:ajil95.abraham at gmail.com>> wrote:
> > > >
> > > > Thanks for all the support. After handling the input
> validation in my
> > > code, Glusterd no longer crashes. I am still waiting for
> clearance from my
> > > superior to pass on all the details. Expecting him to revert by this
> > > Sunday.
> > > Great to know that and we appreciate your contribution, if you
> happen to
> > > find any issues feel free to send patches :)
> > > >
> > > > - Ajil
> > > >
> > > > On Fri, Mar 4, 2016 at 10:20 AM, Joseph Fernandes
> <josferna at redhat.com <mailto:josferna at redhat.com>>
> > > wrote:
> > > >>
> > > >> Well that may not be completely correct !
> > > >>
> > > >> Its "gluster volume status all", unlike volume maintenance
> operation
> > > which are rare.
> > > >>
> > > >> Status can be issued multiple times in a day or might be put in a
> > > script/cron-job to check the health of the
> > > >> cluster.
> > > >> But anyways the fix is ready as the bug says.
> > > >>
> > > >> Crash is what we need to worry about.
> > > >>
> > > >> ~Joe
> > > >>
> > > >> ----- Original Message -----
> > > >> > From: "Atin Mukherjee" <amukherj at redhat.com
> <mailto:amukherj at redhat.com>>
> > > >> > To: "Joseph Fernandes" <josferna at redhat.com
> <mailto:josferna at redhat.com>>, "Atin Mukherjee" <
> > > atin.mukherjee83 at gmail.com <mailto:atin.mukherjee83 at gmail.com>>
> > > >> > Cc: "Gluster Devel" <gluster-devel at gluster.org
> <mailto:gluster-devel at gluster.org>>, "Ajil Abraham" <
> > > ajil95.abraham at gmail.com <mailto:ajil95.abraham at gmail.com>>
> > > >> > Sent: Friday, March 4, 2016 9:37:43 AM
> > > >> > Subject: Re: [Gluster-devel] glusterd crashing
> > > >> >
> > > >> >
> > > >> >
> > > >> > On 03/04/2016 07:10 AM, Joseph Fernandes wrote:
> > > >> > > Might be this bug can give some context on the mem-leak (fix
> > > recently
> > > >> > > merged on master but not on 3.7.x)
> > > >> > >
> > > >> > > https://bugzilla.redhat.com/show_bug.cgi?id=1287517
> > > >> > Yes, this is what we'd be fixing in 3.7.x too, but if you
> refer to [1]
> > > >> > the hike is seen when a command is run in a loop which is
> typically
> > > not
> > > >> > a use case in any production setup.
> > > >> >
> > > >> > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1287517#c15
> > > >> > >
> > > >> > > ~Joe
> > > >> > >
> > > >> > >
> > > >> > > ----- Original Message -----
> > > >> > >> From: "Atin Mukherjee" <atin.mukherjee83 at gmail.com
> <mailto:atin.mukherjee83 at gmail.com>>
> > > >> > >> To: "Joseph Fernandes" <josferna at redhat.com
> <mailto:josferna at redhat.com>>
> > > >> > >> Cc: "Gluster Devel" <gluster-devel at gluster.org
> <mailto:gluster-devel at gluster.org>>, "Ajil Abraham"
> > > >> > >> <ajil95.abraham at gmail.com <mailto:ajil95.abraham at gmail.com>>
> > > >> > >> Sent: Friday, March 4, 2016 7:01:54 AM
> > > >> > >> Subject: Re: [Gluster-devel] glusterd crashing
> > > >> > >>
> > > >> > >> -Atin
> > > >> > >> Sent from one plus one
> > > >> > >> On 04-Mar-2016 6:12 am, "Joseph Fernandes"
> <josferna at redhat.com <mailto:josferna at redhat.com>>
> > > wrote:
> > > >> > >>>
> > > >> > >>> Hi Ajil,
> > > >> > >>>
> > > >> > >>> Well few things,
> > > >> > >>>
> > > >> > >>> 1. Whenever you see a crash its better to send across the
> > > Backtrace(BT)
> > > >> > >> using gdb and attach the log files (or share it via some
> cloud
> > > drive)
> > > >> > >>>
> > > >> > >>> 2. About the memory leak, What kind of tools are you
> using for
> > > profiling
> > > >> > >> memory, valgrind ? if so please attach the valgrind reports.
> > > >> > >>> $> glusterd --xlator-option *.run-with-valgrind=yes
> > > >> > >>>
> > > >> > >>> 3. Well I am not sure if glusterd uses any of the
> mempools as we
> > > do in
> > > >> > >> client and brick processes, Atin can shed some light on
> this.
> > > >> > >>> Well In that case you can used the statedump
> mechanism check
> > > for
> > > >> > >> mem-leaks check the glusterfs/doc/debugging/statedump.md
> <http://statedump.md>
> > > >> > >> GlusterD does use mempool and it has infra for capturing
> statedump
> > > as
> > > >> > >> well.
> > > >> > >> I am aware of few bytes of memory leaks in few paths
> which is
> > > really not a
> > > >> > >> huge concern but it shouldn't crash.
> > > >> > >>>
> > > >> > >>> Hope this helps
> > > >> > >>>
> > > >> > >>> ~Joe
> > > >> > >>>
> > > >> > >>>
> > > >> > >>> ----- Original Message -----
> > > >> > >>>> From: "Ajil Abraham" <ajil95.abraham at gmail.com
> <mailto:ajil95.abraham at gmail.com>>
> > > >> > >>>> To: "Atin Mukherjee" <atin.mukherjee83 at gmail.com
> <mailto:atin.mukherjee83 at gmail.com>>
> > > >> > >>>> Cc: "Gluster Devel" <gluster-devel at gluster.org
> <mailto:gluster-devel at gluster.org>>
> > > >> > >>>> Sent: Thursday, March 3, 2016 10:48:56 PM
> > > >> > >>>> Subject: Re: [Gluster-devel] glusterd crashing
> > > >> > >>>>
> > > >> > >>>> Hi Atin,
> > > >> > >>>>
> > > >> > >>>> The inputs I use are as per the requirements of a
> project I am
> > > working
> > > >> > >> on for
> > > >> > >>>> one of the large finance institutions in Dubai. I will
> try to
> > > handle the
> > > >> > >>>> input validation within my code. I uncovered some of
> the issues
> > > while
> > > >> > >> doing
> > > >> > >>>> a thorough testing of my code.
> > > >> > >>>>
> > > >> > >>>> I tried with 3.7.6 and also my own build from master
> branch. I
> > > will
> > > >> > >> check
> > > >> > >>>> with my superiors before sending you backtrace and other
> > > details. So
> > > >> > >> far, I
> > > >> > >>>> have seen memory leak in 100s of KBs.
> > > >> > >>>>
> > > >> > >>>> -Ajil
> > > >> > >>>>
> > > >> > >>>>
> > > >> > >>>> On Thu, Mar 3, 2016 at 10:17 PM, Atin Mukherjee <
> > > >> > >> atin.mukherjee83 at gmail.com
> <mailto:atin.mukherjee83 at gmail.com>
> > > >> > >>>>> wrote:
> > > >> > >>>>
> > > >> > >>>>
> > > >> > >>>>
> > > >> > >>>>
> > > >> > >>>> Hi Ajil,
> > > >> > >>>>
> > > >> > >>>> Its good to see that you are doing a thorough testing
> gluster.
> > > From
> > > >> > >> your mail
> > > >> > >>>> it looks like your automation focuses on mostly
> negative tests.
> > > I need
> > > >> > >> few
> > > >> > >>>> additional details to get to know whether they are known:
> > > >> > >>>>
> > > >> > >>>> 1. Version of gluster
> > > >> > >>>> 2. Backtrace of the crash along with reproducer
> > > >> > >>>> 3. Amount of memory leak in terms of bytes/KB/MB??
> Have you
> > > already
> > > >> > >>>> identified them?
> > > >> > >>>>
> > > >> > >>>> -Atin
> > > >> > >>>> Sent from one plus one
> > > >> > >>>> On 03-Mar-2016 10:01 pm, "Ajil Abraham" <
> > > ajil95.abraham at gmail.com <mailto:ajil95.abraham at gmail.com> >
> > > >> > >> wrote:
> > > >> > >>>>
> > > >> > >>>>
> > > >> > >>>>
> > > >> > >>>> For my project, I am trying to do some automation using
> > > glusterd. It is
> > > >> > >> very
> > > >> > >>>> frustrating to see it crashing frequently. Looks like
> input
> > > validation
> > > >> > >> is
> > > >> > >>>> the culprit. I also see lot of buffer overflow and
> memory leak
> > > issues.
> > > >> > >>>> Making a note of these and will try to fix them.
> Surprised to
> > > see such
> > > >> > >> basic
> > > >> > >>>> issues still existing in Gluster.
> > > >> > >>>>
> > > >> > >>>> -Ajil
> > > >> > >>>>
> > > >> > >>>> _______________________________________________
> > > >> > >>>> Gluster-devel mailing list
> > > >> > >>>> Gluster-devel at gluster.org
> <mailto:Gluster-devel at gluster.org>
> > > >> > >>>> http://www.gluster.org/mailman/listinfo/gluster-devel
> > > >> > >>>>
> > > >> > >>>>
> > > >> > >>>> _______________________________________________
> > > >> > >>>> Gluster-devel mailing list
> > > >> > >>>> Gluster-devel at gluster.org
> <mailto:Gluster-devel at gluster.org>
> > > >> > >>>> http://www.gluster.org/mailman/listinfo/gluster-devel
> > > >> > >>
> > > >> > > _______________________________________________
> > > >> > > Gluster-devel mailing list
> > > >> > > Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
> > > >> > > http://www.gluster.org/mailman/listinfo/gluster-devel
> > > >> > >
> > > >> >
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Gluster-devel mailing list
> > > > Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
> > > > http://www.gluster.org/mailman/listinfo/gluster-devel
> > >
> > >
> >
>
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
More information about the Gluster-devel
mailing list