<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br><div>While analysing the logs of the runs where uss.t failed made following observations.</div><div><br></div><div>1) In the first iteration of uss.t, the time difference between the first test of the .t file and the last test of the .t file is just within 1 minute.</div><div><br></div><div>But, I think it is the cleanup sequence which is taking more time. One of the reasons I guess this is happening is, we dont see the brick process shutting down message</div><div>in the logs.</div><div><br></div><div><br></div><div>2) In the 2nd iteration of uss.t (because 1st iteration failed because of timeout) it fails because something has not been completed in the cleanup sequence of the previous iteration.</div><div><br></div><div>The volume start command itself fails in the 2nd iteration. Because of that the remaining tests also fail</div><div><br></div><div>This is from cmd_history.log </div><div><br></div><div><div>uster.org:/d/backends/2/patchy_snap_mnt builder202.int.aws.gluster.org:/d/backends/3/patchy_snap_mnt ++++++++++</div><div>[2019-04-10 19:54:09.145086] : volume create patchy builder202.int.aws.gluster.org:/d/backends/1/patchy_snap_mnt builder202.int.aws.gluster.org:/d/backends/2/patchy_snap_mnt builder202.int.aws.gluster.org:/d/backends/3/patchy_snap_mnt : SUCCESS</div><div>[2019-04-10 19:54:09.156221]:++++++++++ G_LOG:./tests/basic/uss.t: TEST: 39 gluster --mode=script --wignore volume set patchy nfs.disable false ++++++++++</div><div>[2019-04-10 19:54:09.265138] : volume set patchy nfs.disable false : SUCCESS</div><div>[2019-04-10 19:54:09.274386]:++++++++++ G_LOG:./tests/basic/uss.t: TEST: 42 gluster --mode=script --wignore volume start patchy ++++++++++</div><div>[2019-04-10 19:54:09.565086] : volume start patchy : FAILED : Commit failed on localhost. Please check log file for details.</div><div>[2019-04-10 19:54:09.572753]:++++++++++ G_LOG:./tests/basic/uss.t: TEST: 44 _GFS --attribute-timeout=0 --entry-timeout=0 --volfile-server=<a href="http://builder202.int.aws.gluster.org">builder202.int.aws.gluster.org</a> --volfile-id=patchy /mnt/glusterfs/0 ++++++++++</div></div><div><br></div><div><br></div><div>And this is from the brick showing some issue with the export directory not being present properly.</div><div><br></div><div><div>[2019-04-10 19:54:09.544476] I [MSGID: 100030] [glusterfsd.c:2857:main] 0-/build/install/sbin/glusterfsd: Started running /build/install/sbin/glusterfsd version 7dev (args: /build/install/sbin/glusterfsd -s buil</div><div><a href="http://der202.int.aws.gluster.org">der202.int.aws.gluster.org</a> --volfile-id patchy.builder202.int.aws.gluster.org.d-backends-1-patchy_snap_mnt -p /var/run/gluster/vols/patchy/builder202.int.aws.gluster.org-d-backends-1-patchy_snap_mnt.pid -S /var/</div><div>run/gluster/7ac65190b72da80a.socket --brick-name /d/backends/1/patchy_snap_mnt -l /var/log/glusterfs/bricks/d-backends-1-patchy_snap_mnt.log --xlator-option *-posix.glusterd-uuid=695c060d-74d3-440e-8cdb-327ec297</div><div>f2d2 --process-name brick --brick-port 49152 --xlator-option patchy-server.listen-port=49152)</div><div>[2019-04-10 19:54:09.549394] I [socket.c:962:__socket_server_bind] 0-socket.glusterfsd: closing (AF_UNIX) reuse check socket 9</div><div>[2019-04-10 19:54:09.553190] I [MSGID: 101190] [event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1</div><div>[2019-04-10 19:54:09.553209] I [MSGID: 101190] [event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0</div><div>[2019-04-10 19:54:09.556932] I [rpcsvc.c:2694:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64</div><div>[2019-04-10 19:54:09.557859] E [MSGID: 138001] [index.c:2392:init] 0-patchy-index: Failed to find parent dir (/d/backends/1/patchy_snap_mnt/.glusterfs) of index basepath /d/backends/1/patchy_snap_mnt/.glusterfs/</div><div>indices. [No such file or directory] ============================> (.glusterfs is absent)</div><div>[2019-04-10 19:54:09.557884] E [MSGID: 101019] [xlator.c:629:xlator_init] 0-patchy-index: Initialization of volume 'patchy-index' failed, review your volfile again</div><div>[2019-04-10 19:54:09.557892] E [MSGID: 101066] [graph.c:409:glusterfs_graph_init] 0-patchy-index: initializing translator failed</div><div>[2019-04-10 19:54:09.557900] E [MSGID: 101176] [graph.c:772:glusterfs_graph_activate] 0-graph: init failed</div><div>[2019-04-10 19:54:09.564154] I [io-stats.c:4033:fini] 0-patchy-io-stats: io-stats translator unloaded</div><div>[2019-04-10 19:54:09.564748] W [glusterfsd.c:1592:cleanup_and_exit] (-->/build/install/sbin/glusterfsd(mgmt_getspec_cbk+0x806) [0x411f32] -->/build/install/sbin/glusterfsd(glusterfs_process_volfp+0x272) [0x40b9b</div><div>9] -->/build/install/sbin/glusterfsd(cleanup_and_exit+0x88) [0x4093a5] ) 0-: received signum (-1), shutting down</div></div><div><br></div><div><br></div><div>And this is from the cmd_history.log file of the 2nd iteration uss.t from another jenkins run of uss.t</div><div><br></div><div><div>[2019-04-10 15:35:51.927343]:++++++++++ G_LOG:./tests/basic/uss.t: TEST: 39 gluster --mode=script --wignore volume set patchy nfs.disable false ++++++++++</div><div>[2019-04-10 15:35:52.038072] : volume set patchy nfs.disable false : SUCCESS</div><div>[2019-04-10 15:35:52.057582]:++++++++++ G_LOG:./tests/basic/uss.t: TEST: 42 gluster --mode=script --wignore volume start patchy ++++++++++</div><div>[2019-04-10 15:35:52.104288] : volume start patchy : FAILED : Failed to find brick directory /d/backends/1/patchy_snap_mnt for volume patchy. Reason : No such file or directory =========> (export directory is not present)</div><div>[2019-04-10 15:35:52.117735]:++++++++++ G_LOG:./tests/basic/uss.t: TEST: 44 _GFS --attribute-timeout=0 --entry-timeout=0 --volfile-server=<a href="http://builder205.int.aws.gluster.org">builder205.int.aws.gluster.org</a> --volfile-id=patchy /mnt/glusterfs/0 ++++++++++</div></div><div><br></div><div><br></div><div>I suspect something wrong with the cleanup sequence which causes the timeout of the test in the 1st iteration and the export directory issues in the next iteration causes the failure of uss.t in the 2nd iteration.</div><div><br></div><div><br></div><div>Regards,</div><div>Raghavendra</div><div><br></div><div><br></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 10, 2019 at 4:07 PM FNU Raghavendra Manjunath <<a href="mailto:rabhat@redhat.com">rabhat@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 10, 2019 at 9:59 AM Atin Mukherjee <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>And now for last 15 days:</div><div><br></div><div><a href="https://fstat.gluster.org/summary?start_date=2019-03-25&end_date=2019-04-10" target="_blank">https://fstat.gluster.org/summary?start_date=2019-03-25&end_date=2019-04-10</a></div><div><br></div><div>./tests/bitrot/bug-1373520.t 18 ==> Fixed through <a href="https://review.gluster.org/#/c/glusterfs/+/22481/" target="_blank">https://review.gluster.org/#/c/glusterfs/+/22481/</a>, I don't see this failing in brick mux post 5th April<br></div></div></div></div></div></blockquote><div><br></div><div>The above patch has been sent to fix the failure with brick mux enabled.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>./tests/bugs/ec/bug-1236065.t 17 ==> happens only in brick mux, needs analysis.<br>./tests/basic/uss.t 15 ==> happens in both brick mux and non brick mux runs, test just simply times out. Needs urgent analysis.<br></div></div></div></div></div></blockquote><div><br></div><div>Nothing has changed in <span class="gmail-m_-1539886695993058310gmail-gr_ gmail-m_-1539886695993058310gmail-gr_86 gmail-m_-1539886695993058310gmail-gr-alert gmail-m_-1539886695993058310gmail-gr_spell gmail-m_-1539886695993058310gmail-gr_inline_cards gmail-m_-1539886695993058310gmail-gr_run_anim gmail-m_-1539886695993058310gmail-ContextualSpelling gmail-m_-1539886695993058310gmail-ins-del gmail-m_-1539886695993058310gmail-multiReplace" id="gmail-m_-1539886695993058310gmail-86" style="display:inline;border-bottom:2px solid transparent;background-repeat:no-repeat">snapview</span>-server and <span class="gmail-m_-1539886695993058310gmail-gr_ gmail-m_-1539886695993058310gmail-gr_87 gmail-m_-1539886695993058310gmail-gr-alert gmail-m_-1539886695993058310gmail-gr_spell gmail-m_-1539886695993058310gmail-gr_inline_cards gmail-m_-1539886695993058310gmail-gr_run_anim gmail-m_-1539886695993058310gmail-ContextualSpelling gmail-m_-1539886695993058310gmail-ins-del gmail-m_-1539886695993058310gmail-multiReplace" id="gmail-m_-1539886695993058310gmail-87" style="display:inline;border-bottom:2px solid transparent;background-repeat:no-repeat">snapview</span>-client recently. Looking into it.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>./tests/basic/ec/ec-fix-openfd.t 13 ==> Fixed through <a href="https://review.gluster.org/#/c/22508/" target="_blank">https://review.gluster.org/#/c/22508/</a> , patch merged today. <br>./tests/basic/volfile-sanity.t 8 ==> Some race, though this succeeds in second attempt every time.<br><br>There're plenty more with 5 instances of failure from many tests. We need all maintainers/owners to look through these failures and fix them, we certainly don't want to get into a stage where master is unstable and we have to lock down the merges till all these failures are resolved. So please help.</div><div><br></div><div>(Please note fstat stats show up the retries as failures too which in a way is right)<br></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 26, 2019 at 5:27 PM Atin Mukherjee <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>[1] captures the test failures report since last 30 days and we'd need volunteers/component owners to see why the number of failures are so high against few tests.</div><div><br></div><div>[1] <a href="https://fstat.gluster.org/summary?start_date=2019-01-26&end_date=2019-02-25&job=all" target="_blank">https://fstat.gluster.org/summary?start_date=2019-01-26&end_date=2019-02-25&job=all</a><br></div></div></div>
</blockquote></div>
</div></div>_______________________________________________<br>Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a></blockquote></div></div>
</blockquote></div>