<div dir="ltr">On Mon, Dec 3, 2018 at 8:32 AM Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com">rgowdapp@redhat.com</a>> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr">On Mon, Dec 3, 2018 at 8:25 AM Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">On Sat, Dec 1, 2018 at 11:02 AM Milind Changire <<a href="mailto:mchangir@redhat.com" target="_blank">mchangir@redhat.com</a>> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>failed brick-mux-regression job:<br></div><div dir="ltr"><a href="https://build.gluster.org/job/regression-on-demand-multiplex/411/console" target="_blank">https://build.gluster.org/job/regression-on-demand-multiplex/411/console</a></div><div dir="ltr"><br></div><div>patch:</div><div><a href="https://review.gluster.org/c/glusterfs/+/21719" target="_blank">https://review.gluster.org/c/glusterfs/+/21719</a></div></div></div></div></blockquote><div><br></div><div>Does this happen only with the above patch? Does brick-mux regression succeed on current master without this patch? Wondering whether the parallelism introduced by bumping up event-threads to 2, is opening up some races in multiplexed environment (though there were always more than one event-thread when more than one brick is multiplexed). <br></div></div></div></blockquote><div><br></div><div>Also, is this bug locally reproducible on your setup if you run test following test with brick-mux enabled (with and without your patch)?<br><pre class="m_3310358445309888247gmail-console-output">./tests/bugs/glusterd/add-brick-and-validate-replicated-volume-options.t</pre></div></div></div></div></blockquote><div>running the above test with brick-mux enabled works well with and without the "event-threads bump" patch on my centos7 setup</div><div>so, the issue is not reproducible on my setup</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div dir="ltr"><br></div><div>stack trace:</div><div>$ gdb -ex 'set sysroot ./' -ex 'core-file ./build/install/cores/glfs_epoll000-964.core' ./build/install/sbin/glusterfsd<br>GNU gdb (GDB) Fedora 8.2-4.fc29<br>Copyright (C) 2018 Free Software Foundation, Inc.<br>License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html" target="_blank">http://gnu.org/licenses/gpl.html</a>><br>This is free software: you are free to change and redistribute it.<br>There is NO WARRANTY, to the extent permitted by law.<br>Type "show copying" and "show warranty" for details.<br>This GDB was configured as "x86_64-redhat-linux-gnu".<br>Type "show configuration" for configuration details.<br>For bug reporting instructions, please see:<br><<a href="http://www.gnu.org/software/gdb/bugs/" target="_blank">http://www.gnu.org/software/gdb/bugs/</a>>.<br>Find the GDB manual and other documentation resources online at:<br> <<a href="http://www.gnu.org/software/gdb/documentation/" target="_blank">http://www.gnu.org/software/gdb/documentation/</a>>.<br><br>For help, type "help".<br>Type "apropos word" to search for commands related to "word"...<br>Reading symbols from ./build/install/sbin/glusterfsd...done.<br>[New LWP 970]<br>[New LWP 992]<br>[New LWP 993]<br>[New LWP 1005]<br>[New LWP 1241]<br>[New LWP 964]<br>[New LWP 968]<br>[New LWP 996]<br>[New LWP 995]<br>[New LWP 994]<br>[New LWP 967]<br>[New LWP 969]<br>[New LWP 1003]<br>[New LWP 1181]<br>[New LWP 1242]<br>[New LWP 966]<br>[New LWP 965]<br>[New LWP 999]<br>[New LWP 1000]<br>[New LWP 1002]<br>[New LWP 989]<br>[New LWP 990]<br>[New LWP 991]<br>[New LWP 971]<br>warning: Ignoring non-absolute filename: <./lib64/libz.so.1><br>Missing separate debuginfo for ./lib64/libz.so.1<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/ea/8e45dc8e395cc5e26890470112d97a1f1e0b65.debug<br>warning: Ignoring non-absolute filename: <./lib64/libuuid.so.1><br>Missing separate debuginfo for ./lib64/libuuid.so.1<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/71/de190dc0c93504abacc17b9747cd772a1e4b0d.debug<br>warning: Ignoring non-absolute filename: <./lib64/libm.so.6><br>Missing separate debuginfo for ./lib64/libm.so.6<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/f4/cae74047f9aa2d5a71fdec67c4285d75753eba.debug<br>warning: Ignoring non-absolute filename: <./lib64/librt.so.1><br>Missing separate debuginfo for ./lib64/librt.so.1<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/d3/3989ec31efe745eb0d3b68a92d19e77d7ddfda.debug<br>warning: Ignoring non-absolute filename: <./lib64/libdl.so.2><br>Missing separate debuginfo for ./lib64/libdl.so.2<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/5c/db5a56336e7e2bd14ffa189411e44a834afcd8.debug<br>warning: Ignoring non-absolute filename: <./lib64/libpthread.so.0><br>Missing separate debuginfo for ./lib64/libpthread.so.0<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/f4/c04bce85d2d269d0a2af4972fc69805b50345b.debug<br>warning: Expected absolute pathname for libpthread in the inferior, but got ./lib64/libpthread.so.0.<br>warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.<br>warning: Ignoring non-absolute filename: <./lib64/libcrypto.so.10><br>Missing separate debuginfo for ./lib64/libcrypto.so.10<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/67/ceb4edd36bfe0eb31cd92da2694aca5377a599.debug<br>warning: Ignoring non-absolute filename: <./lib64/libc.so.6><br>Missing separate debuginfo for ./lib64/libc.so.6<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/cb/4b7554d1adbef2f001142dd6f0a5139fc9aa69.debug<br>warning: Ignoring non-absolute filename: <./lib64/ld-linux-x86-64.so.2><br>Missing separate debuginfo for ./lib64/ld-linux-x86-64.so.2<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/d2/66b1f6650927e18108323bcca8f7b68e68eb92.debug<br>warning: Ignoring non-absolute filename: <./lib64/libssl.so.10><br>Missing separate debuginfo for ./lib64/libssl.so.10<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/64/68a4e28a19cdd885a3cbc30e009589ca4c2e92.debug<br>warning: Ignoring non-absolute filename: <./lib64/libgssapi_krb5.so.2><br>Missing separate debuginfo for ./lib64/libgssapi_krb5.so.2<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/16/fe0dc6cefc5f444bc876516d02efe9cc2d432f.debug<br>warning: Ignoring non-absolute filename: <./lib64/libkrb5.so.3><br>Missing separate debuginfo for ./lib64/libkrb5.so.3<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/d1/cd1b94855a85fbc735c745db39bc096f7d8cc3.debug<br>warning: Ignoring non-absolute filename: <./lib64/libcom_err.so.2><br>Missing separate debuginfo for ./lib64/libcom_err.so.2<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/2c/7ef64ef0c5af8bcfa8f9e628e5605a7d8c52d3.debug<br>warning: Ignoring non-absolute filename: <./lib64/libk5crypto.so.3><br>Missing separate debuginfo for ./lib64/libk5crypto.so.3<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/a2/0f715c514b3ea873f4cc77d585a50cb670e266.debug<br>warning: Ignoring non-absolute filename: <./lib64/libkrb5support.so.0><br>Missing separate debuginfo for ./lib64/libkrb5support.so.0<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/9f/82b06ce44724a8ace827a2c95a9a76619ea314.debug<br>warning: Ignoring non-absolute filename: <./lib64/libkeyutils.so.1><br>Missing separate debuginfo for ./lib64/libkeyutils.so.1<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/2e/01d5ac08c1280d013aab96b292ac58bc30a263.debug<br>warning: Ignoring non-absolute filename: <./lib64/libresolv.so.2><br>Missing separate debuginfo for ./lib64/libresolv.so.2<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/2b/dc2b6ff0b2c204cce34d139a9eada0272eb070.debug<br>warning: Ignoring non-absolute filename: <./lib64/libselinux.so.1><br>Missing separate debuginfo for ./lib64/libselinux.so.1<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/1b/0c0dc4f9054c5922082921c1447739f27ac8f4.debug<br>warning: Ignoring non-absolute filename: <./lib64/libpcre.so.1><br>Missing separate debuginfo for ./lib64/libpcre.so.1<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/9c/a3d11f018beeb719cdb34be800bf1641350d0a.debug<br>warning: Ignoring non-absolute filename: <./lib64/libgcc_s.so.1><br>Missing separate debuginfo for ./lib64/libgcc_s.so.1<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/53/1aa1391ea4e1489d5ef11aa5dc2ffd9e2bdfee.debug<br>warning: Ignoring non-absolute filename: <./lib64/libnss_files.so.2><br>Missing separate debuginfo for ./lib64/libnss_files.so.2<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/52/ed84166576539e50b20c4edf9a104c1e04ae45.debug<br>warning: Ignoring non-absolute filename: <./lib64/libaio.so.1><br>Missing separate debuginfo for ./lib64/libaio.so.1<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/62/bdce334f05919d9568bf4c44e4bc470dcf8c56.debug<br>warning: Ignoring non-absolute filename: <./lib64/libacl.so.1><br>Missing separate debuginfo for ./lib64/libacl.so.1<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/7f/39882fc0b80be53790c2eac307d39f7de1ad6e.debug<br>warning: Ignoring non-absolute filename: <./lib64/libattr.so.1><br>Missing separate debuginfo for ./lib64/libattr.so.1<br>Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/26/17ecc6738047e207ae3add990bd6a34d11b265.debug<br>warning: Expected absolute pathname for libpthread in the inferior, but got ./lib64/libpthread.so.0.<br>warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.<br>Core was generated by `/build/install/sbin/glusterfsd -s <a href="http://builder105.cloud.gluster.org" target="_blank">builder105.cloud.gluster.org</a> --volfile-id pat'.<br>Program terminated with signal SIGSEGV, Segmentation fault.<br>#0 0x000000000040bc33 in STACK_DESTROY (stack=0x3) at /home/jenkins/root/workspace/regression-on-demand-multiplex/libglusterfs/src/stack.h:182<br>182 LOCK(&stack->pool->lock);<br>[Current thread is 1 (LWP 970)]<br>(gdb) bt<br>#0 0x000000000040bc33 in STACK_DESTROY (stack=0x3) at /home/jenkins/root/workspace/regression-on-demand-multiplex/libglusterfs/src/stack.h:182<br>#1 0x000000000041286e in mgmt_pmap_signin_cbk (req=0x7f9af0349768, iov=0x7f9af03497a0, count=1, myframe=0x7f9af0141648)<br> at /home/jenkins/root/workspace/regression-on-demand-multiplex/glusterfsd/src/glusterfsd-mgmt.c:2829<br>#2 0x00007f9b0dc5c59e in rpc_clnt_handle_reply (clnt=0x7fa5b0, pollin=0x7f9af0008750)<br> at /home/jenkins/root/workspace/regression-on-demand-multiplex/rpc/rpc-lib/src/rpc-clnt.c:755<br>#3 0x00007f9b0dc5cac7 in rpc_clnt_notify (trans=0x7fa910, mydata=0x7fa5e0, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f9af0008750)<br> at /home/jenkins/root/workspace/regression-on-demand-multiplex/rpc/rpc-lib/src/rpc-clnt.c:922<br>#4 0x00007f9b0dc58b23 in rpc_transport_notify (this=0x7fa910, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f9af0008750)<br> at /home/jenkins/root/workspace/regression-on-demand-multiplex/rpc/rpc-lib/src/rpc-transport.c:525<br>#5 0x00007f9b02a6a6a6 in socket_event_poll_in (this=0x7fa910, notify_handled=true)<br> at /home/jenkins/root/workspace/regression-on-demand-multiplex/rpc/rpc-transport/socket/src/socket.c:2530<br>#6 0x00007f9b02a6b687 in socket_event_handler (fd=16, idx=8, gen=10, data=0x7fa910, poll_in=1, poll_out=0, poll_err=0)<br> at /home/jenkins/root/workspace/regression-on-demand-multiplex/rpc/rpc-transport/socket/src/socket.c:2931<br>#7 0x00007f9b0df146c1 in event_dispatch_epoll_handler (event_pool=0x7b0c30, event=0x7f9b00fa1ea0)<br> at /home/jenkins/root/workspace/regression-on-demand-multiplex/libglusterfs/src/event-epoll.c:591<br>#8 0x00007f9b0df149ba in event_dispatch_epoll_worker (data=0x7fc110)<br> at /home/jenkins/root/workspace/regression-on-demand-multiplex/libglusterfs/src/event-epoll.c:668<br>#9 0x00007f9b0cce5e25 in start_thread () from ./lib64/libpthread.so.0<br>#10 0x00007f9b0c5aebad in clone () from ./lib64/libc.so.6<br>(gdb) up<br>#1 0x000000000041286e in mgmt_pmap_signin_cbk (req=0x7f9af0349768, iov=0x7f9af03497a0, count=1, myframe=0x7f9af0141648)<br> at /home/jenkins/root/workspace/regression-on-demand-multiplex/glusterfsd/src/glusterfsd-mgmt.c:2829<br>2829 STACK_DESTROY(frame->root);<br>(gdb) p *frame<br>$1 = {root = 0x3, parent = 0x300000001, frames = {next = 0xffffffffffffffff, prev = 0x7f9af01416a8}, local = 0x7f9af0003958, this = 0x0, <br> ret = 0x0, ref_count = 0, lock = {spinlock = 0, mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, <br> __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x7f9af0003958}}, <br> __size = '\000' <repeats 32 times>, "X9\000\360\232\177\000", __align = 0}}, cookie = 0x0, complete = 232, op = 32666, begin = {<br> tv_sec = 0, tv_nsec = 140303428253256}, end = {tv_sec = 140303428244368, tv_nsec = 245890744}, <br> wind_from = 0x1 <error: Cannot access memory at address 0x1>, wind_to = 0x0, unwind_from = 0x0, unwind_to = 0x0}<br>(gdb) <br><br></div></div></div></div>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a></blockquote></div></div>
</blockquote></div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr">Milind<br><br></div></div></div></div></div>