<div dir="ltr">+</div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 9, 2017 at 11:33 AM, Mohit Agrawal <span dir="ltr"><<a href="mailto:moagrawa@redhat.com" target="_blank">moagrawa@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><div dir="ltr"></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 9, 2017 at 11:16 AM, Mohit Agrawal <span dir="ltr"><<a href="mailto:moagrawa@redhat.com" target="_blank">moagrawa@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi All,</div><div><br></div><div><br></div><div>For specific to this patch(<a href="https://review.gluster.org/#/c/18436/" target="_blank">https://review.gluster.o<wbr>rg/#/c/18436/</a>) i am getting crash in nfs(only once) for the </div><div>test case (./tests/basic/mount-nfs-auth.<wbr>t), although i tried to execute the same test case in a loop on centos</div><div>machine but i have not found any crash.</div><div><br></div><div>After anaylys the crash it seems cache(entry) is invalidate in thread 10 and same it is trying to access</div><div>in thread 1. </div><div><br></div><div>>>>>>>>>>>>>>>>>>>>.</div><div><br></div><div>(gdb) thread 1</div><div>[Switching to thread 1 (Thread 0x7fe852cfe700 (LWP 19073))]#0 0x00007fe859665c85 in auth_cache_lookup (</div><div> cache=0x7fe854027db0, fh=0x7fe84466684c, host_addr=0x7fe844565e40 "23.253.175.80", </div><div> timestamp=0x7fe852cfb1e0, can_write=0x7fe852cfb1dc)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/xlators/nfs/server/src/au<wbr>th-cache.c:295</div><div>295<span class="m_-1414313188914909489m_-8157473174835730716gmail-Apple-tab-span" style="white-space:pre-wrap"> </span> *can_write = lookup_res->item->opts->rw;</div><div>(gdb) bt</div><div>#0 0x00007fe859665c85 in auth_cache_lookup (cache=0x7fe854027db0, fh=0x7fe84466684c, </div><div> host_addr=0x7fe844565e40 "23.253.175.80", timestamp=0x7fe852cfb1e0, can_write=0x7fe852cfb1dc)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/xlators/nfs/server/src/au<wbr>th-cache.c:295</div><div>#1 0x00007fe859665ebc in is_nfs_fh_cached (cache=0x7fe854027db0, fh=0x7fe84466684c, </div><div> host_addr=0x7fe844565e40 "23.253.175.80")</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/xlators/nfs/server/src/au<wbr>th-cache.c:390</div><div>#2 0x00007fe85962b82c in mnt3_check_cached_fh (ms=0x7fe854023d60, fh=0x7fe84466684c, </div><div> host_addr=0x7fe844565e40 "23.253.175.80", is_write_op=_gf_false)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/xlators/nfs/server/src/mo<wbr>unt3.c:1954</div><div>#3 0x00007fe85962ba92 in _mnt3_authenticate_req (ms=0x7fe854023d60, req=0x7fe844679148, </div><div> fh=0x7fe84466684c, path=0x0, authorized_export=0x0, authorized_host=0x0, is_write_op=_gf_false)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/xlators/nfs/server/src/mo<wbr>unt3.c:2011</div><div>#4 0x00007fe85962bf65 in mnt3_authenticate_request (ms=0x7fe854023d60, req=0x7fe844679148, </div><div> fh=0x7fe84466684c, volname=0x0, path=0x0, authorized_path=0x0, authorized_host=0x0, </div><div> is_write_op=_gf_false)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/xlators/nfs/server/src/mo<wbr>unt3.c:2130</div><div>#5 0x00007fe859652370 in nfs3_fh_auth_nfsop (cs=0x7fe8446663c8, is_write_op=_gf_false)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/xlators/nfs/server/src/nf<wbr>s3-helpers.c:3981</div><div>#6 0x00007fe85963631a in nfs3_lookup_resume (carg=0x7fe8446663c8)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/xlators/nfs/server/src/nf<wbr>s3.c:155---Type <return> to continue, or q <return> to quit---</div><div>9</div><div>#7 0x00007fe859651b98 in nfs3_fh_resolve_entry_hard (cs=0x7fe8446663c8)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/xlators/nfs/server/src/nf<wbr>s3-helpers.c:3791</div><div>#8 0x00007fe859651e35 in nfs3_fh_resolve_entry (cs=0x7fe8446663c8)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/xlators/nfs/server/src/nf<wbr>s3-helpers.c:3844</div><div>#9 0x00007fe859651e94 in nfs3_fh_resolve_resume (cs=0x7fe8446663c8)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/xlators/nfs/server/src/nf<wbr>s3-helpers.c:3862</div><div>#10 0x00007fe8596520ad in nfs3_fh_resolve_root (cs=0x7fe8446663c8)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/xlators/nfs/server/src/nf<wbr>s3-helpers.c:3915</div><div>#11 0x00007fe85965245f in nfs3_fh_resolve_and_resume (cs=0x7fe8446663c8, fh=0x7fe852cfc980, </div><div> entry=0x7fe852cfc9c0 "test-bg-write", resum_fn=0x7fe85963621d <nfs3_lookup_resume>)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/xlators/nfs/server/src/nf<wbr>s3-helpers.c:4011</div><div>#12 0x00007fe859636dcf in nfs3_lookup (req=0x7fe844679148, fh=0x7fe852cfc980, fhlen=52, </div><div> name=0x7fe852cfc9c0 "test-bg-write")</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/xlators/nfs/server/src/nf<wbr>s3.c:1620</div><div>#13 0x00007fe85963703f in nfs3svc_lookup (req=0x7fe844679148)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/xlators/nfs/server/src/nf<wbr>s3.c:1666</div><div>#14 0x00007fe86765f585 in rpcsvc_handle_rpc_call (svc=0x7fe854022a00, trans=0x7fe8545c1fa0, </div><div> msg=0x7fe844334610)</div><div>---Type <return> to continue, or q <return> to quit---</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/rpc/rpc-lib/src/rpcsvc.c:<wbr>711</div><div>#15 0x00007fe86765f8f8 in rpcsvc_notify (trans=0x7fe8545c1fa0, mydata=0x7fe854022a00, </div><div> event=RPC_TRANSPORT_MSG_RECEIV<wbr>ED, data=0x7fe844334610)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/rpc/rpc-lib/src/rpcsvc.c:<wbr>805</div><div>#16 0x00007fe867665458 in rpc_transport_notify (this=0x7fe8545c1fa0, event=RPC_TRANSPORT_MSG_RECEIV<wbr>ED, </div><div> data=0x7fe844334610)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/rpc/rpc-lib/src/rpc-trans<wbr>port.c:538</div><div>#17 0x00007fe85c44561e in socket_event_poll_in (this=0x7fe8545c1fa0, notify_handled=_gf_true)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/rpc/rpc-transport/socket/<wbr>src/socket.c:2319</div><div>#18 0x00007fe85c445cb1 in socket_event_handler (fd=12, idx=8, gen=103, data=0x7fe8545c1fa0, poll_in=1, </div><div> poll_out=0, poll_err=0)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/rpc/rpc-transport/socket/<wbr>src/socket.c:2475</div><div>#19 0x00007fe867917fd7 in event_dispatch_epoll_handler (event_pool=0x7030d0, event=0x7fe852cfde70)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/libglusterfs/src/event-ep<wbr>oll.c:583</div><div>#20 0x00007fe8679182d9 in event_dispatch_epoll_worker (data=0x7fe85403d060)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/libglusterfs/src/event-ep<wbr>oll.c:659</div><div>#21 0x00007fe866b7baa1 in start_thread () from /lib64/libpthread.so.0</div><div>#22 0x00007fe8664e3bcd in clone () from /lib64/libc.so.6</div><div><br></div><div>(gdb) thread 10</div><div>[Switching to thread 10 (Thread 0x7fe858ed2700 (LWP 19051))]#0 0x00007fe866b82334 in __lll_lock_wait ()</div><div> from /lib64/libpthread.so.0</div><div>(gdb) bt</div><div>#0 0x00007fe866b82334 in __lll_lock_wait () from /lib64/libpthread.so.0</div><div>#1 0x00007fe866b7d5d8 in _L_lock_854 () from /lib64/libpthread.so.0</div><div>#2 0x00007fe866b7d4a7 in pthread_mutex_lock () from /lib64/libpthread.so.0</div><div>#3 0x00007fe8678a9844 in _gf_msg (</div><div> domain=0x7fe85966a448 "ot/workspace/my_glusterfs_bui<wbr>ld/glusterfs-4.0dev/xlators/<wbr>nfs/server/src/mount3.c", file=0x7fe85966a3f8 "/lib/glusterd/nfs/exports", function=0x7fe85966b5e0 "init", line=3878, </div><div> level=GF_LOG_INFO, errnum=0, trace=0, msgid=112151, fmt=0x7fe85966b3b4 "")</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/libglusterfs/src/logging.<wbr>c:2081</div><div>#4 0x00007fe859630287 in _mnt3_auth_param_refresh_threa<wbr>d (argv=0x7fe854023d60)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/xlators/nfs/server/src/mo<wbr>unt3.c:3877</div><div>#5 0x00007fe866b7baa1 in start_thread () from /lib64/libpthread.so.0</div><div>#6 0x00007fe8664e3bcd in clone () from /lib64/libc.so.6</div><div>(gdb) p mstate</div><div>No symbol "mstate" in current context.</div><div>(gdb) f 4</div><div>#4 0x00007fe859630287 in _mnt3_auth_param_refresh_threa<wbr>d (argv=0x7fe854023d60)</div><div> at /home/jenkins/root/workspace/m<wbr>y_glusterfs_build/glusterfs-4.<wbr>0dev/xlators/nfs/server/src/mo<wbr>unt3.c:3877</div><div>3877</div><div><br></div><div>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>><wbr>>>.</div><div><br></div><div>In first level analysis i don't think it is related to my patch, please check and response on the same if you have seen earlier also.</div><div><br></div><div>For specific to core you can access core from this link <a href="https://build.gluster.org/job/centos6-regression/6759/console" target="_blank">https://build.gluster.org/job/<wbr>centos6-regression/6759/consol<wbr>e</a></div><div><br></div><div>Regards</div><span class="m_-1414313188914909489HOEnZb"><font color="#888888"><div>Mohit Agrawal</div></font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>