<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jun 3, 2019 at 11:57 AM Xie Changlong <<a href="mailto:zgrep@139.com">zgrep@139.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div style="overflow-wrap: break-word;"><div style="color:rgb(0,0,0);font-family:微软雅黑"><span style="font-size:10pt">Hi all</span></div><div style="color:rgb(0,0,0);font-family:微软雅黑"><span style="font-size:10pt"><br></span></div><div style="overflow-wrap: break-word;"><span style="color:rgb(0,0,0);font-family:微软雅黑;font-size:10pt">Test gluster 3.8.4-54.15 gnfs, i saw a write request hung in write-behind followed by 1545 FLUSH requests. I found a similar</span></div><div style="overflow-wrap: break-word;"><span style="color:rgb(0,0,0);font-family:微软雅黑;font-size:10pt">bugfix </span><span style="font-size:13px"><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1626787" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1626787</a></span><span style="font-size:10pt">, but not sure if it's the right one. </span></div><div style="overflow-wrap: break-word;"><span style="font-size:10pt"><br></span></div><div style="overflow-wrap: break-word;"><div style="font-size:16px"><span style="font-size:10pt">[xlator.performance.write-behind.wb_inode]</span></div><div style="font-size:16px"><span style="font-size:10pt">path=/575/1e/5751e318f21f605f2aac241bf042e7a8.jpg</span></div><div style="font-size:16px"><span style="font-size:10pt">inode=0x7f51775b71a0</span></div><div style="font-size:16px"><span style="font-size:10pt">window_conf=1073741824</span></div><div style="font-size:16px"><span style="font-size:10pt">window_current=293822</span></div><div style="font-size:16px"><span style="font-size:10pt">transit-size=293822</span></div><div style="font-size:16px"><span style="font-size:10pt">dontsync=0</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt">[.WRITE]</span></div><div style="font-size:16px"><span style="font-size:10pt">request-ptr=0x7f516eec2060</span></div><div style="font-size:16px"><span style="font-size:10pt">refcount=1</span></div><div style="font-size:16px"><span style="font-size:10pt">wound=yes</span></div><div style="font-size:16px"><span style="font-size:10pt">generation-number=1</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_ret=293822</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_errno=0</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-attempts=1</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-in-progress=yes</span></div></div></div></div></blockquote><div><br></div><div>Note that the sync is still in progress. This means, write-behind has wound the write-request to its children and yet to receive the response (unless there is a bug in accounting of sync-in-progress). So, its likely that there are callstacks into children of write-behind, which are not complete yet. Are you sure the deepest hung call-stack is in write-behind? Can you check for frames with "complete=0"? </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div style="overflow-wrap: break-word;"><div style="overflow-wrap: break-word;"><div style="font-size:16px"><span style="font-size:10pt">size=293822</span></div><div style="font-size:16px"><span style="font-size:10pt">offset=1048576</span></div><div style="font-size:16px"><span style="font-size:10pt">lied=-1</span></div><div style="font-size:16px"><span style="font-size:10pt">append=0</span></div><div style="font-size:16px"><span style="font-size:10pt">fulfilled=0</span></div><div style="font-size:16px"><span style="font-size:10pt">go=-1</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt">[.FLUSH]</span></div><div style="font-size:16px"><span style="font-size:10pt">request-ptr=0x7f517c2badf0</span></div><div style="font-size:16px"><span style="font-size:10pt">refcount=1</span></div><div style="font-size:16px"><span style="font-size:10pt">wound=no</span></div><div style="font-size:16px"><span style="font-size:10pt">generation-number=2</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_ret=-1</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_errno=116</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-attempts=0</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt">[.FLUSH]</span></div><div style="font-size:16px"><span style="font-size:10pt">request-ptr=0x7f5173e9f7b0</span></div><div style="font-size:16px"><span style="font-size:10pt">refcount=1</span></div><div style="font-size:16px"><span style="font-size:10pt">wound=no</span></div><div style="font-size:16px"><span style="font-size:10pt">generation-number=2</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_ret=0</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_errno=0</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-attempts=0</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt">[.FLUSH]</span></div><div style="font-size:16px"><span style="font-size:10pt">request-ptr=0x7f51640b8ca0</span></div><div style="font-size:16px"><span style="font-size:10pt">refcount=1</span></div><div style="font-size:16px"><span style="font-size:10pt">wound=no</span></div><div style="font-size:16px"><span style="font-size:10pt">generation-number=2</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_ret=0</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_errno=0</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-attempts=0</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt">[.FLUSH]</span></div><div style="font-size:16px"><span style="font-size:10pt">request-ptr=0x7f516f3979d0</span></div><div style="font-size:16px"><span style="font-size:10pt">refcount=1</span></div><div style="font-size:16px"><span style="font-size:10pt">wound=no</span></div><div style="font-size:16px"><span style="font-size:10pt">generation-number=2</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_ret=0</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_errno=0</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-attempts=0</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt">[.FLUSH]</span></div><div style="font-size:16px"><span style="font-size:10pt">request-ptr=0x7f516f6ac8d0</span></div><div style="font-size:16px"><span style="font-size:10pt">refcount=1</span></div><div style="font-size:16px"><span style="font-size:10pt">wound=no</span></div><div style="font-size:16px"><span style="font-size:10pt">generation-number=2</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_ret=0</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_errno=0</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-attempts=0</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div></div><div style="overflow-wrap: break-word;"><span style="font-size:10pt">Any comments would be appreciated!</span></div><div style="color:rgb(0,0,0);font-family:微软雅黑"><br></div><div style="overflow-wrap: break-word;"><div style="color:rgb(0,0,0);font-family:微软雅黑"><span style="font-size:10pt">Thanks</span></div><div style="color:rgb(0,0,0);font-family:微软雅黑"><span style="font-size:10pt"><span class="gmail-m_672640851088131096Apple-tab-span" style="white-space:pre-wrap"> </span>-Xie</span></div></div><div style="color:rgb(0,0,0);font-family:微软雅黑"><span style="font-size:10pt"><br></span></div><div style="color:rgb(0,0,0);font-family:微软雅黑;font-size:12pt"><br></div></div><div id="gmail-m_672640851088131096SIGNNAME5481"></div><div><span id="gmail-m_672640851088131096_THINKMAILNAME5481"></span></div></div>
</blockquote></div></div>