[Gluster-devel] glusterd crashing

Sat Mar 5 07:16:07 UTC 2016

-Atin
Sent from one plus one
On 05-Mar-2016 11:46 am, "Ajil Abraham" <ajil95.abraham at gmail.com> wrote:
>
> Thanks for all the support.  After handling the input validation in my
code, Glusterd no longer crashes.  I am still waiting for clearance from my
superior to pass on all the details. Expecting him to revert by this Sunday.
Great to know that and we appreciate your contribution, if you happen to
find any issues feel free to send patches :)
>
> - Ajil
>
> On Fri, Mar 4, 2016 at 10:20 AM, Joseph Fernandes <josferna at redhat.com>
wrote:
>>
>> Well that may not be completely correct !
>>
>> Its  "gluster volume status all", unlike volume maintenance operation
which are rare.
>>
>> Status can be issued multiple times in a day or might be put in a
script/cron-job to check the health of the
>> cluster.
>> But anyways the fix is ready as the bug says.
>>
>> Crash is what we need to worry about.
>>
>> ~Joe
>>
>> ----- Original Message -----
>> > From: "Atin Mukherjee" <amukherj at redhat.com>
>> > To: "Joseph Fernandes" <josferna at redhat.com>, "Atin Mukherjee" <
atin.mukherjee83 at gmail.com>
>> > Cc: "Gluster Devel" <gluster-devel at gluster.org>, "Ajil Abraham" <
ajil95.abraham at gmail.com>
>> > Sent: Friday, March 4, 2016 9:37:43 AM
>> > Subject: Re: [Gluster-devel] glusterd crashing
>> >
>> >
>> >
>> > On 03/04/2016 07:10 AM, Joseph Fernandes wrote:
>> > > Might be this bug can give some context on the mem-leak (fix recently
>> > > merged on master but not on 3.7.x)
>> > >
>> > > https://bugzilla.redhat.com/show_bug.cgi?id=1287517
>> > Yes, this is what we'd be fixing in 3.7.x too, but if you refer to [1]
>> > the hike is seen when a command is run in a loop which is typically not
>> > a use case in any production setup.
>> >
>> > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1287517#c15
>> > >
>> > > ~Joe
>> > >
>> > >
>> > > ----- Original Message -----
>> > >> From: "Atin Mukherjee" <atin.mukherjee83 at gmail.com>
>> > >> To: "Joseph Fernandes" <josferna at redhat.com>
>> > >> Cc: "Gluster Devel" <gluster-devel at gluster.org>, "Ajil Abraham"
>> > >> <ajil95.abraham at gmail.com>
>> > >> Sent: Friday, March 4, 2016 7:01:54 AM
>> > >> Subject: Re: [Gluster-devel] glusterd crashing
>> > >>
>> > >> -Atin
>> > >> Sent from one plus one
>> > >> On 04-Mar-2016 6:12 am, "Joseph Fernandes" <josferna at redhat.com>
wrote:
>> > >>>
>> > >>> Hi Ajil,
>> > >>>
>> > >>> Well few things,
>> > >>>
>> > >>> 1. Whenever you see a crash its better to send across the
Backtrace(BT)
>> > >> using gdb and attach the log files (or share it via some cloud
drive)
>> > >>>
>> > >>> 2. About the memory leak, What kind of tools are you using for
profiling
>> > >> memory, valgrind ? if so please attach the valgrind reports.
>> > >>>    $> glusterd --xlator-option *.run-with-valgrind=yes
>> > >>>
>> > >>> 3. Well I am not sure if glusterd uses any of the mempools as we
do in
>> > >> client and brick processes, Atin can shed some light on this.
>> > >>>    Well In that case you can used the statedump mechanism check for
>> > >> mem-leaks check the glusterfs/doc/debugging/statedump.md
>> > >> GlusterD does use mempool and it has infra for capturing statedump
as
>> > >> well.
>> > >> I am aware of few bytes of memory leaks in few paths which is
really not a
>> > >> huge concern but it shouldn't crash.
>> > >>>
>> > >>> Hope this helps
>> > >>>
>> > >>> ~Joe
>> > >>>
>> > >>>
>> > >>> ----- Original Message -----
>> > >>>> From: "Ajil Abraham" <ajil95.abraham at gmail.com>
>> > >>>> To: "Atin Mukherjee" <atin.mukherjee83 at gmail.com>
>> > >>>> Cc: "Gluster Devel" <gluster-devel at gluster.org>
>> > >>>> Sent: Thursday, March 3, 2016 10:48:56 PM
>> > >>>> Subject: Re: [Gluster-devel] glusterd crashing
>> > >>>>
>> > >>>> Hi Atin,
>> > >>>>
>> > >>>> The inputs I use are as per the requirements of a project I am
working
>> > >> on for
>> > >>>> one of the large finance institutions in Dubai. I will try to
handle the
>> > >>>> input validation within my code. I uncovered some of the issues
while
>> > >> doing
>> > >>>> a thorough testing of my code.
>> > >>>>
>> > >>>> I tried with 3.7.6 and also my own build from master branch. I
will
>> > >> check
>> > >>>> with my superiors before sending you backtrace and other details.
So
>> > >> far, I
>> > >>>> have seen memory leak in 100s of KBs.
>> > >>>>
>> > >>>> -Ajil
>> > >>>>
>> > >>>>
>> > >>>> On Thu, Mar 3, 2016 at 10:17 PM, Atin Mukherjee <
>> > >> atin.mukherjee83 at gmail.com
>> > >>>>> wrote:
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>> Hi Ajil,
>> > >>>>
>> > >>>> Its good to see that you are doing a thorough testing gluster.
From
>> > >> your mail
>> > >>>> it looks like your automation focuses on mostly negative tests. I
need
>> > >> few
>> > >>>> additional details to get to know whether they are known:
>> > >>>>
>> > >>>> 1. Version of gluster
>> > >>>> 2. Backtrace of the crash along with reproducer
>> > >>>> 3. Amount of memory leak in terms of bytes/KB/MB?? Have you
already
>> > >>>> identified them?
>> > >>>>
>> > >>>> -Atin
>> > >>>> Sent from one plus one
>> > >>>> On 03-Mar-2016 10:01 pm, "Ajil Abraham" < ajil95.abraham at gmail.com
>
>> > >> wrote:
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>> For my project, I am trying to do some automation using glusterd.
It is
>> > >> very
>> > >>>> frustrating to see it crashing frequently. Looks like input
validation
>> > >> is
>> > >>>> the culprit. I also see lot of buffer overflow and memory leak
issues.
>> > >>>> Making a note of these and will try to fix them. Surprised to see
such
>> > >> basic
>> > >>>> issues still existing in Gluster.
>> > >>>>
>> > >>>> -Ajil
>> > >>>>
>> > >>>> _______________________________________________
>> > >>>> Gluster-devel mailing list
>> > >>>> Gluster-devel at gluster.org
>> > >>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>> > >>>>
>> > >>>>
>> > >>>> _______________________________________________
>> > >>>> Gluster-devel mailing list
>> > >>>> Gluster-devel at gluster.org
>> > >>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>> > >>
>> > > _______________________________________________
>> > > Gluster-devel mailing list
>> > > Gluster-devel at gluster.org
>> > > http://www.gluster.org/mailman/listinfo/gluster-devel
>> > >
>> >
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160305/06bf352c/attachment.html>