<html><head></head><body><div class="rich_html_content" style="word-wrap: break-word; font-family: 微软雅黑; color: rgb(0, 0, 0);"><span style="font-size: 10pt;">Firstly i correct myself, </span><span style="font-size: 13px;">write request followed by 771</span><span style="font-size: 13px;">(not 1545)</span><span style="font-size: 13px;"> FLUSH requests</span><span style="font-size: 10pt;">. I've attach gnfs dump file, totally 774 pending call-stacks,</span></div><div class="rich_html_content" style="word-wrap: break-word; font-family: 微软雅黑; color: rgb(0, 0, 0);"><span style="font-size: 10pt;">771 of them pending on write-behind and </span><span style="font-size: 10pt;">the deepest call-stack is afr.</span></div><div class="rich_html_content" style="color: rgb(0, 0, 0); font-family: 微软雅黑; word-wrap: break-word;"><span style="font-size: 10pt;"><br></span></div><div class="rich_html_content" style="word-wrap: break-word;"><div class="rich_html_content" style="word-wrap: break-word;"><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">[global.callpool.stack.771]</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">stack=0x7f517f557f60</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">uid=0</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">gid=0</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">pid=0</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">unique=0</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">lk-owner=</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">op=stack</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">type=0</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">cnt=3</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;"><br></span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">[global.callpool.stack.771.frame.1]</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">frame=0x7f517f655880</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">ref_count=0</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">translator=cl35vol01-replicate-7</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">complete=0</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">parent=cl35vol01-dht</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">wind_from=dht_writev</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">wind_to=subvol->fops->writev</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">unwind_to=dht_writev_cbk</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;"><br></span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">[global.callpool.stack.771.frame.2]</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">frame=0x7f518ed90340</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">ref_count=1</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">translator=cl35vol01-dht</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">complete=0</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">parent=cl35vol01-write-behind</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">wind_from=wb_fulfill_head</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">wind_to=FIRST_CHILD (frame->this)->fops->writev</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">unwind_to=wb_fulfill_cbk</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;"><br></span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">[global.callpool.stack.771.frame.3]</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">frame=0x7f516d3baf10</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">ref_count=1</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">translator=cl35vol01-write-behind</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">complete=0</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;"><br></span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">[global.callpool.stack.772]</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">stack=0x7f51607a5a20</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">uid=0</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">gid=0</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">pid=0</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">unique=0</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">lk-owner=a0715b77517f0000</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">op=stack</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">type=0</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">cnt=1</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;"><br></span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">[global.callpool.stack.772.frame.1]</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">frame=0x7f516ca2d1b0</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">ref_count=0</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">translator=cl35vol01-replicate-7</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">complete=0</span></font></div><div><br></div><div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">[root@rhel-201 35]# grep -rn "global.callpool.stack.*.frame.1" -A 5 glusterdump.20106.dump.1559038081 |grep translator | wc -l</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">774</span></font></div></div></div></div><div class="rich_html_content" style="word-wrap: break-word;"><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">[root@rhel-201 35]# grep -rn "global.callpool.stack.*.frame.1" -A 5 glusterdump.20106.dump.1559038081 |grep complete |wc -l</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">774</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">[root@rhel-201 35]# grep -rn "global.callpool.stack.*.frame.1" -A 5 glusterdump.20106.dump.1559038081 |grep -E "complete=0" |wc -l</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">774</span></font></div></div><div class="rich_html_content" style="word-wrap: break-word;"><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">[root@rhel-201 35]# grep -rn "global.callpool.stack.*.frame.1" -A 5 glusterdump.20106.dump.1559038081 |grep translator | grep write-behind |wc -l</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">771</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">[root@rhel-201 35]# grep -rn "global.callpool.stack.*.frame.1" -A 5 glusterdump.20106.dump.1559038081 |grep translator | grep replicate-7 | wc -l</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">2</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">[root@rhel-201 35]# grep -rn "global.callpool.stack.*.frame.1" -A 5 glusterdump.20106.dump.1559038081 |grep translator | grep glusterfs | wc -l</span></font></div><div class="rich_html_content" style="word-wrap: break-word;"><font face="微软雅黑"><span style="font-size: 13px;">1</span></font></div><div style="color: rgb(0, 0, 0); font-family: 微软雅黑; font-size: 10pt;"><br></div></div><div class="rich_html_content" style="color:#000000; font-Size:12pt; font-family:微软雅黑; word-wrap:break-word;x-overflow:hidden;"><br></div><div class="rich_html_content" style="color:#000000; font-Size:12pt; font-family:微软雅黑; word-wrap:break-word;x-overflow:hidden;"><br></div><div id="SIGNNAME5488"></div><div><span id="_THINKMAILNAME5488" font-size:12pt;font-family:microsoft="" yahei;=""></span></div><blockquote style="margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em;"><div> </div><div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"><meta charset="UTF-8"><div style="color:#333333; font-Size:12pt;font-family:Microsoft YaHei;">发件人: <a href="mailto:rgowdapp@redhat.com">Raghavendra Gowdappa</a></div><div style="color:#333333; font-Size:12pt;font-family:Microsoft YaHei;">时间: 2019/06/03(星期一)14:46</div><div style="color:#333333; font-Size:12pt;font-family:Microsoft YaHei;">收件人: <a href="mailto:zgrep@139.com">Xie Changlong</a>;</div><div style="color:#333333; font-Size:12pt;font-family:Microsoft YaHei;">抄送人: <a href="mailto:gluster-users@gluster.org">gluster-users</a>;</div><div style="color:#333333; font-Size:12pt;font-family:Microsoft YaHei;">主题: Re: write request hung in write-behind</div></div></blockquote><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jun 3, 2019 at 11:57 AM Xie Changlong <<a href="mailto:zgrep@139.com">zgrep@139.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div style="overflow-wrap: break-word;"><div style="color:rgb(0,0,0);font-family:微软雅黑"><span style="font-size:10pt">Hi all</span></div><div style="color:rgb(0,0,0);font-family:微软雅黑"><span style="font-size:10pt"><br></span></div><div style="overflow-wrap: break-word;"><span style="color:rgb(0,0,0);font-family:微软雅黑;font-size:10pt">Test gluster 3.8.4-54.15 gnfs, i saw a write request hung in write-behind followed by 1545 FLUSH requests. I found a similar</span></div><div style="overflow-wrap: break-word;"><span style="color:rgb(0,0,0);font-family:微软雅黑;font-size:10pt">bugfix </span><span style="font-size:13px"><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1626787" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1626787</a></span><span style="font-size:10pt">, but not sure if it's the right one. </span></div><div style="overflow-wrap: break-word;"><span style="font-size:10pt"><br></span></div><div style="overflow-wrap: break-word;"><div style="font-size:16px"><span style="font-size:10pt">[xlator.performance.write-behind.wb_inode]</span></div><div style="font-size:16px"><span style="font-size:10pt">path=/575/1e/5751e318f21f605f2aac241bf042e7a8.jpg</span></div><div style="font-size:16px"><span style="font-size:10pt">inode=0x7f51775b71a0</span></div><div style="font-size:16px"><span style="font-size:10pt">window_conf=1073741824</span></div><div style="font-size:16px"><span style="font-size:10pt">window_current=293822</span></div><div style="font-size:16px"><span style="font-size:10pt">transit-size=293822</span></div><div style="font-size:16px"><span style="font-size:10pt">dontsync=0</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt">[.WRITE]</span></div><div style="font-size:16px"><span style="font-size:10pt">request-ptr=0x7f516eec2060</span></div><div style="font-size:16px"><span style="font-size:10pt">refcount=1</span></div><div style="font-size:16px"><span style="font-size:10pt">wound=yes</span></div><div style="font-size:16px"><span style="font-size:10pt">generation-number=1</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_ret=293822</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_errno=0</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-attempts=1</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-in-progress=yes</span></div></div></div></div></blockquote><div><br></div><div>Note that the sync is still in progress. This means, write-behind has wound the write-request to its children and yet to receive the response (unless there is a bug in accounting of sync-in-progress). So, its likely that there are callstacks into children of write-behind, which are not complete yet. Are you sure the deepest hung call-stack is in write-behind? Can you check for frames with "complete=0"? </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;"><div style="overflow-wrap: break-word;"><div style="overflow-wrap: break-word;"><div style="font-size:16px"><span style="font-size:10pt">size=293822</span></div><div style="font-size:16px"><span style="font-size:10pt">offset=1048576</span></div><div style="font-size:16px"><span style="font-size:10pt">lied=-1</span></div><div style="font-size:16px"><span style="font-size:10pt">append=0</span></div><div style="font-size:16px"><span style="font-size:10pt">fulfilled=0</span></div><div style="font-size:16px"><span style="font-size:10pt">go=-1</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt">[.FLUSH]</span></div><div style="font-size:16px"><span style="font-size:10pt">request-ptr=0x7f517c2badf0</span></div><div style="font-size:16px"><span style="font-size:10pt">refcount=1</span></div><div style="font-size:16px"><span style="font-size:10pt">wound=no</span></div><div style="font-size:16px"><span style="font-size:10pt">generation-number=2</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_ret=-1</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_errno=116</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-attempts=0</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt">[.FLUSH]</span></div><div style="font-size:16px"><span style="font-size:10pt">request-ptr=0x7f5173e9f7b0</span></div><div style="font-size:16px"><span style="font-size:10pt">refcount=1</span></div><div style="font-size:16px"><span style="font-size:10pt">wound=no</span></div><div style="font-size:16px"><span style="font-size:10pt">generation-number=2</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_ret=0</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_errno=0</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-attempts=0</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt">[.FLUSH]</span></div><div style="font-size:16px"><span style="font-size:10pt">request-ptr=0x7f51640b8ca0</span></div><div style="font-size:16px"><span style="font-size:10pt">refcount=1</span></div><div style="font-size:16px"><span style="font-size:10pt">wound=no</span></div><div style="font-size:16px"><span style="font-size:10pt">generation-number=2</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_ret=0</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_errno=0</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-attempts=0</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt">[.FLUSH]</span></div><div style="font-size:16px"><span style="font-size:10pt">request-ptr=0x7f516f3979d0</span></div><div style="font-size:16px"><span style="font-size:10pt">refcount=1</span></div><div style="font-size:16px"><span style="font-size:10pt">wound=no</span></div><div style="font-size:16px"><span style="font-size:10pt">generation-number=2</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_ret=0</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_errno=0</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-attempts=0</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt">[.FLUSH]</span></div><div style="font-size:16px"><span style="font-size:10pt">request-ptr=0x7f516f6ac8d0</span></div><div style="font-size:16px"><span style="font-size:10pt">refcount=1</span></div><div style="font-size:16px"><span style="font-size:10pt">wound=no</span></div><div style="font-size:16px"><span style="font-size:10pt">generation-number=2</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_ret=0</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_errno=0</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-attempts=0</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div></div><div style="overflow-wrap: break-word;"><span style="font-size:10pt">Any comments would be appreciated!</span></div><div style="color:rgb(0,0,0);font-family:微软雅黑"><br></div><div style="overflow-wrap: break-word;"><div style="color:rgb(0,0,0);font-family:微软雅黑"><span style="font-size:10pt">Thanks</span></div><div style="color:rgb(0,0,0);font-family:微软雅黑"><span style="font-size:10pt"><span class="gmail-m_672640851088131096Apple-tab-span" style="white-space:pre-wrap"> </span>-Xie</span></div></div><div style="color:rgb(0,0,0);font-family:微软雅黑"><span style="font-size:10pt"><br></span></div><div style="color:rgb(0,0,0);font-family:微软雅黑;font-size:12pt"><br></div></div><div id="gmail-m_672640851088131096SIGNNAME5481"></div><div><span id="gmail-m_672640851088131096_THINKMAILNAME5481"></span></div></div>
</blockquote></div></div>
</body></html>