[Gluster-devel] glusterd crashes on /tests/bugs/replicate/bug-884328.t

[Atin Mukherjee]

Thanks Xavi. I have +2ed it. Any one can merge it once it passes the
regression.

On Fri, Dec 15, 2017 at 6:45 PM, Xavi Hernandez <jahernan at redhat.com> wrote:

> I've uploaded a patch to fix this problem: https://review.
> gluster.org/19040
>
> On Fri, Dec 15, 2017 at 11:33 AM, Xavi Hernandez <jahernan at redhat.com>
> wrote:
>
>> I've checked the size of 'gluster volume set help' on current master and
>> it's 51176 bytes. Only 24 bytes below the size of the buffer.
>>
>> I think the reason why regression tests fail is that it enables bd
>> xlator, which adds some more options that make the help output to grow
>> beyond the buffer size.
>>
>> I'll send a patch to fix the problem.
>>
>> Xavi
>>
>> On Fri, Dec 15, 2017 at 10:05 AM, Xavi Hernandez <jahernan at redhat.com>
>> wrote:
>>
>>> On Fri, Dec 15, 2017 at 9:57 AM, Atin Mukherjee <amukherj at redhat.com>
>>> wrote:
>>>
>>>> But why doesn't it crash every time if this is the RCA? None of us
>>>> could actually reproduce it locally.
>>>>
>>>
>>> That's a good question. One of my patches has failed and it doesn't add
>>> any new option (in fact it's a very trivial change), so I'm not sure why it
>>> may or may not crash.
>>>
>>> I'll analyze it. Anyway, that function needs a patch because there's no
>>> space limit check before writing to the buffer.
>>>
>>> Xavi
>>>
>>>
>>>> On Fri, Dec 15, 2017 at 2:23 PM, Xavi Hernandez <jahernan at redhat.com>
>>>> wrote:
>>>>
>>>>> I've seen this failure in one of my local tests and I've done a quick
>>>>> analysis:
>>>>>
>>>>> (gdb) bt
>>>>> #0  0x00007ff29e1fce07 in ?? () from /lib64/libgcc_s.so.1
>>>>> #1  0x00007ff29e1fe9b8 in _Unwind_Backtrace () from
>>>>> /lib64/libgcc_s.so.1
>>>>> #2  0x00007ff2aa9fb458 in backtrace () from /lib64/libc.so.6
>>>>> #3  0x00007ff2ac14af30 in _gf_msg_backtrace_nomem (level=GF_LOG_ALERT,
>>>>> stacksize=200) at logging.c:1128
>>>>> #4  0x00007ff2ac151170 in gf_print_trace (signum=11, ctx=0xdec260) at
>>>>> common-utils.c:762
>>>>> #5  0x000000000040a2c6 in glusterfsd_print_trace (signum=11) at
>>>>> glusterfsd.c:2274
>>>>> #6  <signal handler called>
>>>>> #7  0x00007ff2ac466751 in _dl_close () from
>>>>> /lib64/ld-linux-x86-64.so.2
>>>>> #8  0x00007ff2aaa304df in _dl_catch_error () from /lib64/libc.so.6
>>>>> #9  0x00007ff2ab35f715 in _dlerror_run () from /lib64/libdl.so.2
>>>>> #10 0x00007ff2ab35f08f in dlclose () from /lib64/libdl.so.2
>>>>> #11 0x00007ff2a06af786 in glusterd_get_volopt_content
>>>>> (ctx=0x7ff298000d88, xml_out=false) at glusterd-utils.c:13150
>>>>> #12 0x00007ff2a06a2896 in glusterd_volset_help
>>>>> (dict=0x70616e732d776f68, op_errstr=0x732e736572757461) at
>>>>> glusterd-utils.c:9199
>>>>> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>>>>> (gdb) f 11
>>>>> #11 0x00007ff2a06af786 in glusterd_get_volopt_content
>>>>> (ctx=0x7ff298000d88, xml_out=false) at glusterd-utils.c:13150
>>>>> 13150                           dlclose (dl_handle);
>>>>> (gdb) print dl_handle
>>>>> $1 = (void *) 0x6978656c7069746c
>>>>> (gdb) x/s &dl_handle
>>>>> 0x7ff294206500: "ltiplexing feature is disabled.\n\n"
>>>>> (gdb)
>>>>>
>>>>> So I think the problem is a buffer overflow.
>>>>>
>>>>> Looking at the code in glusterd-utils.c, function
>>>>> glusterd_get_volopt_content(), I guess that we are writing too much data
>>>>> into output_string, which is a stack defined array of 50 KB, and we have an
>>>>> overflow there. Probably the number of options and its description has
>>>>> grown beyond this limit.
>>>>>
>>>>> I'll send a patch for this shortly.
>>>>>
>>>>> Xavi
>>>>>
>>>>> On Fri, Dec 15, 2017 at 8:31 AM, Sunny Kumar <sunkumar at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> Console log
>>>>>> https://build.gluster.org/job/centos6-regression/8021/console
>>>>>>
>>>>>> Regard
>>>>>> Sunny
>>>>>>
>>>>>> On Fri, Dec 15, 2017 at 12:32 PM, Ravishankar N <
>>>>>> ravishankar at redhat.com> wrote:
>>>>>> > ...for a lot of patches on master .The crash is in volume set; the
>>>>>> .t just
>>>>>> > does a volume set help. Can the glusterd devs take a look as it is
>>>>>> blocking
>>>>>> > merging patches? I have raised BZ 1526268 with the details.
>>>>>> >
>>>>>> > Thanks!
>>>>>> >
>>>>>> > Ravi
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > Gluster-devel mailing list
>>>>>> > Gluster-devel at gluster.org
>>>>>> > http://lists.gluster.org/mailman/listinfo/gluster-devel
>>>>>> _______________________________________________
>>>>>> Gluster-devel mailing list
>>>>>> Gluster-devel at gluster.org
>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-devel mailing list
>>>>> Gluster-devel at gluster.org
>>>>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20171215/e338bf77/attachment-0001.html>