[Gluster-devel] glusterd crashes on /tests/bugs/replicate/bug-884328.t

[Xavi Hernandez]

I've checked the size of 'gluster volume set help' on current master and
it's 51176 bytes. Only 24 bytes below the size of the buffer.

I think the reason why regression tests fail is that it enables bd xlator,
which adds some more options that make the help output to grow beyond the
buffer size.

I'll send a patch to fix the problem.

Xavi

On Fri, Dec 15, 2017 at 10:05 AM, Xavi Hernandez <jahernan at redhat.com>
wrote:

> On Fri, Dec 15, 2017 at 9:57 AM, Atin Mukherjee <amukherj at redhat.com>
> wrote:
>
>> But why doesn't it crash every time if this is the RCA? None of us could
>> actually reproduce it locally.
>>
>
> That's a good question. One of my patches has failed and it doesn't add
> any new option (in fact it's a very trivial change), so I'm not sure why it
> may or may not crash.
>
> I'll analyze it. Anyway, that function needs a patch because there's no
> space limit check before writing to the buffer.
>
> Xavi
>
>
>> On Fri, Dec 15, 2017 at 2:23 PM, Xavi Hernandez <jahernan at redhat.com>
>> wrote:
>>
>>> I've seen this failure in one of my local tests and I've done a quick
>>> analysis:
>>>
>>> (gdb) bt
>>> #0  0x00007ff29e1fce07 in ?? () from /lib64/libgcc_s.so.1
>>> #1  0x00007ff29e1fe9b8 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
>>> #2  0x00007ff2aa9fb458 in backtrace () from /lib64/libc.so.6
>>> #3  0x00007ff2ac14af30 in _gf_msg_backtrace_nomem (level=GF_LOG_ALERT,
>>> stacksize=200) at logging.c:1128
>>> #4  0x00007ff2ac151170 in gf_print_trace (signum=11, ctx=0xdec260) at
>>> common-utils.c:762
>>> #5  0x000000000040a2c6 in glusterfsd_print_trace (signum=11) at
>>> glusterfsd.c:2274
>>> #6  <signal handler called>
>>> #7  0x00007ff2ac466751 in _dl_close () from /lib64/ld-linux-x86-64.so.2
>>> #8  0x00007ff2aaa304df in _dl_catch_error () from /lib64/libc.so.6
>>> #9  0x00007ff2ab35f715 in _dlerror_run () from /lib64/libdl.so.2
>>> #10 0x00007ff2ab35f08f in dlclose () from /lib64/libdl.so.2
>>> #11 0x00007ff2a06af786 in glusterd_get_volopt_content
>>> (ctx=0x7ff298000d88, xml_out=false) at glusterd-utils.c:13150
>>> #12 0x00007ff2a06a2896 in glusterd_volset_help (dict=0x70616e732d776f68,
>>> op_errstr=0x732e736572757461) at glusterd-utils.c:9199
>>> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
>>> (gdb) f 11
>>> #11 0x00007ff2a06af786 in glusterd_get_volopt_content
>>> (ctx=0x7ff298000d88, xml_out=false) at glusterd-utils.c:13150
>>> 13150                           dlclose (dl_handle);
>>> (gdb) print dl_handle
>>> $1 = (void *) 0x6978656c7069746c
>>> (gdb) x/s &dl_handle
>>> 0x7ff294206500: "ltiplexing feature is disabled.\n\n"
>>> (gdb)
>>>
>>> So I think the problem is a buffer overflow.
>>>
>>> Looking at the code in glusterd-utils.c, function
>>> glusterd_get_volopt_content(), I guess that we are writing too much data
>>> into output_string, which is a stack defined array of 50 KB, and we have an
>>> overflow there. Probably the number of options and its description has
>>> grown beyond this limit.
>>>
>>> I'll send a patch for this shortly.
>>>
>>> Xavi
>>>
>>> On Fri, Dec 15, 2017 at 8:31 AM, Sunny Kumar <sunkumar at redhat.com>
>>> wrote:
>>>
>>>> +1
>>>>
>>>> Console log
>>>> https://build.gluster.org/job/centos6-regression/8021/console
>>>>
>>>> Regard
>>>> Sunny
>>>>
>>>> On Fri, Dec 15, 2017 at 12:32 PM, Ravishankar N <ravishankar at redhat.com>
>>>> wrote:
>>>> > ...for a lot of patches on master .The crash is in volume set; the .t
>>>> just
>>>> > does a volume set help. Can the glusterd devs take a look as it is
>>>> blocking
>>>> > merging patches? I have raised BZ 1526268 with the details.
>>>> >
>>>> > Thanks!
>>>> >
>>>> > Ravi
>>>> >
>>>> > _______________________________________________
>>>> > Gluster-devel mailing list
>>>> > Gluster-devel at gluster.org
>>>> > http://lists.gluster.org/mailman/listinfo/gluster-devel
>>>> _______________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org
>>>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>>>
>>>
>>>
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20171215/d2df2dc3/attachment.html>