<div dir="ltr">I've uploaded a patch to fix this problem: <a href="https://review.gluster.org/19040">https://review.gluster.org/19040</a></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Dec 15, 2017 at 11:33 AM, Xavi Hernandez <span dir="ltr"><<a href="mailto:jahernan@redhat.com" target="_blank">jahernan@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I've checked the size of 'gluster volume set help' on current master and it's 51176 bytes. Only 24 bytes below the size of the buffer.<div><br></div><div>I think the reason why regression tests fail is that it enables bd xlator, which adds some more options that make the help output to grow beyond the buffer size.</div><div><br></div><div>I'll send a patch to fix the problem.</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>Xavi</div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Dec 15, 2017 at 10:05 AM, Xavi Hernandez <span dir="ltr"><<a href="mailto:jahernan@redhat.com" target="_blank">jahernan@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Fri, Dec 15, 2017 at 9:57 AM, Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">But why doesn't it crash every time if this is the RCA? None of us could actually reproduce it locally.<br></div></blockquote><div><br></div></span><div>That's a good question. One of my patches has failed and it doesn't add any new option (in fact it's a very trivial change), so I'm not sure why it may or may not crash.</div><div><br></div><div>I'll analyze it. Anyway, that function needs a patch because there's no space limit check before writing to the buffer.</div><div><div class="m_-6189325551265631258h5"><div><br></div><div>Xavi</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"></div><div class="m_-6189325551265631258m_-4696475053653887083HOEnZb"><div class="m_-6189325551265631258m_-4696475053653887083h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Dec 15, 2017 at 2:23 PM, Xavi Hernandez <span dir="ltr"><<a href="mailto:jahernan@redhat.com" target="_blank">jahernan@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I've seen this failure in one of my local tests and I've done a quick analysis:<div><br></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">(gdb) bt
</span><br>#0 0x00007ff29e1fce07 in ?? () from /lib64/libgcc_s.so.1
<br>#1 0x00007ff29e1fe9b8 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
<br>#2 0x00007ff2aa9fb458 in backtrace () from /lib64/libc.so.6
<br>#3 0x00007ff2ac14af30 in _gf_msg_backtrace_nomem (level=GF_LOG_ALERT, stacksize=200) at logging.c:1128
<br>#4 0x00007ff2ac151170 in gf_print_trace (signum=11, ctx=0xdec260) at common-utils.c:762
<br>#5 0x000000000040a2c6 in glusterfsd_print_trace (signum=11) at glusterfsd.c:2274
<br>#6 <signal handler called>
<br>#7 0x00007ff2ac466751 in _dl_close () from /lib64/ld-linux-x86-64.so.2
<br>#8 0x00007ff2aaa304df in _dl_catch_error () from /lib64/libc.so.6
<br>#9 0x00007ff2ab35f715 in _dlerror_run () from /lib64/libdl.so.2
<br>#10 0x00007ff2ab35f08f in dlclose () from /lib64/libdl.so.2
<br>#11 0x00007ff2a06af786 in glusterd_get_volopt_content (ctx=0x7ff298000d88, xml_out=false) at glusterd-utils.c:13150
<br>#12 0x00007ff2a06a2896 in glusterd_volset_help (dict=0x70616e732d776f68, op_errstr=0x732e736572757461) at glusterd-utils.c:9199
<br>Backtrace stopped: previous frame inner to this frame (corrupt stack?)
<br>(gdb) f 11
<br>#11 0x00007ff2a06af786 in glusterd_get_volopt_content (ctx=0x7ff298000d88, xml_out=false) at glusterd-utils.c:13150
<br>13150 dlcl<wbr>ose (dl_handle);
<br>(gdb) print dl_handle
<br>$1 = (void *) 0x6978656c7069746c
<br>(gdb) x/s &dl_handle
<br>0x7ff294206500: "ltiplexing feature is disabled.\n\n"
<br>(gdb)<br></span></div><div><span style="font-family:monospace"><br></span></div><div><font face="arial, helvetica, sans-serif">So I think the problem is a buffer overflow.</font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">Looking at the code in glusterd-utils.c, function glusterd_get_volopt_content(), I guess that we are writing too much data into output_string, which is a stack defined array of 50 KB, and we have an overflow there. Probably the number of options and its description has grown beyond this limit.</font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">I'll send a patch for this shortly.</font></div><span class="m_-6189325551265631258m_-4696475053653887083m_3395623168181788247HOEnZb"><font color="#888888"><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">Xavi</font></div></font></span></div><div class="m_-6189325551265631258m_-4696475053653887083m_3395623168181788247HOEnZb"><div class="m_-6189325551265631258m_-4696475053653887083m_3395623168181788247h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Dec 15, 2017 at 8:31 AM, Sunny Kumar <span dir="ltr"><<a href="mailto:sunkumar@redhat.com" target="_blank">sunkumar@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">+1<br>
<br>
Console log<br>
<a href="https://build.gluster.org/job/centos6-regression/8021/console" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>centos6-regression/8021/consol<wbr>e</a><br>
<br>
Regard<br>
<span class="m_-6189325551265631258m_-4696475053653887083m_3395623168181788247m_6133415260981124733HOEnZb"><font color="#888888">Sunny<br>
</font></span><div class="m_-6189325551265631258m_-4696475053653887083m_3395623168181788247m_6133415260981124733HOEnZb"><div class="m_-6189325551265631258m_-4696475053653887083m_3395623168181788247m_6133415260981124733h5"><br>
On Fri, Dec 15, 2017 at 12:32 PM, Ravishankar N <<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>> wrote:<br>
> ...for a lot of patches on master .The crash is in volume set; the .t just<br>
> does a volume set help. Can the glusterd devs take a look as it is blocking<br>
> merging patches? I have raised BZ 1526268 with the details.<br>
><br>
> Thanks!<br>
><br>
> Ravi<br>
><br>
> ______________________________<wbr>_________________<br>
> Gluster-devel mailing list<br>
> <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> <a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-devel</a><br>
______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-devel</a><br>
</div></div></blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-devel</a><br></blockquote></div><br></div>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>