<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 4, 2019 at 7:36 AM Xie Changlong <<a href="mailto:zgrep@139.com">zgrep@139.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-m_-8522292196379778152rich_html_content" style="color:rgb(0,0,0);font-size:12pt;font-family:微软雅黑">To me, all 'df' commands on specific(not all) nfs client hung forever. The temporary solution is <span style="font-size:12pt">disable performance.nfs.write-behind and cluster.eager-lock. </span></div><div class="gmail-m_-8522292196379778152rich_html_content" style="color:rgb(0,0,0);font-size:12pt;font-family:微软雅黑"><br></div><div class="gmail-m_-8522292196379778152rich_html_content" style="color:rgb(0,0,0);font-size:12pt;font-family:微软雅黑">I'll try to get more <span style="font-size:12pt">info back if encounter this problem again .</span></div></div></blockquote><div><br></div><div>If you observe this issue again, take successive (at least a minute apart) statedumps of the processes and run <a href="https://github.com/gluster/glusterfs/blob/master/extras/identify-hangs.sh">https://github.com/gluster/glusterfs/blob/master/extras/identify-hangs.sh</a> on it which will give the information about the hangs.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-m_-8522292196379778152rich_html_content" style="color:rgb(0,0,0);font-size:12pt;font-family:微软雅黑"><br></div><div class="gmail-m_-8522292196379778152rich_html_content" style="color:rgb(0,0,0);font-size:12pt;font-family:微软雅黑"><br></div><div id="gmail-m_-8522292196379778152SIGNNAME5511"></div><div><span id="gmail-m_-8522292196379778152_THINKMAILNAME5511"></span></div><blockquote style="margin-top:0px;margin-bottom:0px;margin-left:0.5em"><div> </div><div style="border-color:rgb(181,196,223) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm"><div style="color:rgb(51,51,51);font-size:12pt;font-family:Microsoft YaHei">发件人: <a href="mailto:rgowdapp@redhat.com" target="_blank">Raghavendra Gowdappa</a></div><div style="color:rgb(51,51,51);font-size:12pt;font-family:Microsoft YaHei">时间: 2019/06/04(星期二)09:55</div><div style="color:rgb(51,51,51);font-size:12pt;font-family:Microsoft YaHei">收件人: <a href="mailto:zgrep@139.com" target="_blank">Xie Changlong</a>;<a href="mailto:ranaraya@redhat.com" target="_blank">Ravishankar Narayanankutty</a>;<a href="mailto:pkarampu@redhat.com" target="_blank">Karampuri, Pranith</a>;</div><div style="color:rgb(51,51,51);font-size:12pt;font-family:Microsoft YaHei">抄送人: <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users</a>;</div><div style="color:rgb(51,51,51);font-size:12pt;font-family:Microsoft YaHei">主题: Re: Re: write request hung in write-behind</div></div></blockquote><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jun 3, 2019 at 1:11 PM Xie Changlong <<a href="mailto:zgrep@139.com" target="_blank">zgrep@139.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content" style="font-family:微软雅黑;color:rgb(0,0,0)"><span style="font-size:10pt">Firstly i correct myself, </span><span style="font-size:13px">write request followed by 771</span><span style="font-size:13px">(not 1545)</span><span style="font-size:13px"> FLUSH requests</span><span style="font-size:10pt">. I've attach gnfs dump file, totally 774 pending call-stacks,</span></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content" style="font-family:微软雅黑;color:rgb(0,0,0)"><span style="font-size:10pt">771 of them pending on write-behind and </span><span style="font-size:10pt">the deepest call-stack is afr.</span></div></div></blockquote><div><br></div><div><a class="gmail_plusreply" id="gmail-m_-8522292196379778152plusReplyChip-0" href="mailto:ranaraya@redhat.com" target="_blank">+Ravishankar Narayanankutty</a> <a class="gmail_plusreply" id="gmail-m_-8522292196379778152plusReplyChip-1" href="mailto:pkarampu@redhat.com" target="_blank">+Karampuri, Pranith</a> <br></div><div><br></div><div>Are you sure these were not call-stacks of in-progress ops? One way of confirming that would be to take statedumps periodically (say 3 min apart). Hung call stacks will be common to all the statedumps.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content" style="color:rgb(0,0,0);font-family:微软雅黑"><span style="font-size:10pt"><br></span></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">[global.callpool.stack.771]</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">stack=0x7f517f557f60</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">uid=0</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">gid=0</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">pid=0</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">unique=0</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">lk-owner=</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">op=stack</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">type=0</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">cnt=3</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px"><br></span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">[global.callpool.stack.771.frame.1]</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">frame=0x7f517f655880</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">ref_count=0</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">translator=cl35vol01-replicate-7</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">complete=0</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">parent=cl35vol01-dht</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">wind_from=dht_writev</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">wind_to=subvol->fops->writev</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">unwind_to=dht_writev_cbk</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px"><br></span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">[global.callpool.stack.771.frame.2]</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">frame=0x7f518ed90340</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">ref_count=1</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">translator=cl35vol01-dht</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">complete=0</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">parent=cl35vol01-write-behind</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">wind_from=wb_fulfill_head</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">wind_to=FIRST_CHILD (frame->this)->fops->writev</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">unwind_to=wb_fulfill_cbk</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px"><br></span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">[global.callpool.stack.771.frame.3]</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">frame=0x7f516d3baf10</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">ref_count=1</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">translator=cl35vol01-write-behind</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">complete=0</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px"><br></span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">[global.callpool.stack.772]</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">stack=0x7f51607a5a20</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">uid=0</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">gid=0</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">pid=0</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">unique=0</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">lk-owner=a0715b77517f0000</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">op=stack</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">type=0</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">cnt=1</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px"><br></span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">[global.callpool.stack.772.frame.1]</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">frame=0x7f516ca2d1b0</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">ref_count=0</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">translator=cl35vol01-replicate-7</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">complete=0</span></font></div><div><br></div><div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">[root@rhel-201 35]# grep -rn "global.callpool.stack.*.frame.1" -A 5 glusterdump.20106.dump.1559038081 |grep translator | wc -l</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">774</span></font></div></div></div></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">[root@rhel-201 35]# grep -rn "global.callpool.stack.*.frame.1" -A 5 glusterdump.20106.dump.1559038081 |grep complete |wc -l</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">774</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">[root@rhel-201 35]# grep -rn "global.callpool.stack.*.frame.1" -A 5 glusterdump.20106.dump.1559038081 |grep -E "complete=0" |wc -l</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">774</span></font></div></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">[root@rhel-201 35]# grep -rn "global.callpool.stack.*.frame.1" -A 5 glusterdump.20106.dump.1559038081 |grep translator | grep write-behind |wc -l</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">771</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">[root@rhel-201 35]# grep -rn "global.callpool.stack.*.frame.1" -A 5 glusterdump.20106.dump.1559038081 |grep translator | grep replicate-7 | wc -l</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">2</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">[root@rhel-201 35]# grep -rn "global.callpool.stack.*.frame.1" -A 5 glusterdump.20106.dump.1559038081 |grep translator | grep glusterfs | wc -l</span></font></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content"><font face="微软雅黑"><span style="font-size:13px">1</span></font></div><div style="color:rgb(0,0,0);font-family:微软雅黑;font-size:10pt"><br></div></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content" style="color:rgb(0,0,0);font-size:12pt;font-family:微软雅黑"><br></div><div class="gmail-m_-8522292196379778152gmail-m_5197581941865454911rich_html_content" style="color:rgb(0,0,0);font-size:12pt;font-family:微软雅黑"><br></div><div id="gmail-m_-8522292196379778152gmail-m_5197581941865454911SIGNNAME5488"></div><div><span id="gmail-m_-8522292196379778152gmail-m_5197581941865454911_THINKMAILNAME5488"></span></div><blockquote style="margin-top:0px;margin-bottom:0px;margin-left:0.5em"><div> </div><div style="border-color:rgb(181,196,223) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm"><div style="color:rgb(51,51,51);font-size:12pt;font-family:Microsoft YaHei">发件人: <a href="mailto:rgowdapp@redhat.com" target="_blank">Raghavendra Gowdappa</a></div><div style="color:rgb(51,51,51);font-size:12pt;font-family:Microsoft YaHei">时间: 2019/06/03(星期一)14:46</div><div style="color:rgb(51,51,51);font-size:12pt;font-family:Microsoft YaHei">收件人: <a href="mailto:zgrep@139.com" target="_blank">Xie Changlong</a>;</div><div style="color:rgb(51,51,51);font-size:12pt;font-family:Microsoft YaHei">抄送人: <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users</a>;</div><div style="color:rgb(51,51,51);font-size:12pt;font-family:Microsoft YaHei">主题: Re: write request hung in write-behind</div></div></blockquote><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jun 3, 2019 at 11:57 AM Xie Changlong <<a href="mailto:zgrep@139.com" target="_blank">zgrep@139.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div style="color:rgb(0,0,0);font-family:微软雅黑"><span style="font-size:10pt">Hi all</span></div><div style="color:rgb(0,0,0);font-family:微软雅黑"><span style="font-size:10pt"><br></span></div><div><span style="color:rgb(0,0,0);font-family:微软雅黑;font-size:10pt">Test gluster 3.8.4-54.15 gnfs, i saw a write request hung in write-behind followed by 1545 FLUSH requests. I found a similar</span></div><div><span style="color:rgb(0,0,0);font-family:微软雅黑;font-size:10pt">bugfix </span><span style="font-size:13px"><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1626787" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1626787</a></span><span style="font-size:10pt">, but not sure if it's the right one. </span></div><div><span style="font-size:10pt"><br></span></div><div><div style="font-size:16px"><span style="font-size:10pt">[xlator.performance.write-behind.wb_inode]</span></div><div style="font-size:16px"><span style="font-size:10pt">path=/575/1e/5751e318f21f605f2aac241bf042e7a8.jpg</span></div><div style="font-size:16px"><span style="font-size:10pt">inode=0x7f51775b71a0</span></div><div style="font-size:16px"><span style="font-size:10pt">window_conf=1073741824</span></div><div style="font-size:16px"><span style="font-size:10pt">window_current=293822</span></div><div style="font-size:16px"><span style="font-size:10pt">transit-size=293822</span></div><div style="font-size:16px"><span style="font-size:10pt">dontsync=0</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt">[.WRITE]</span></div><div style="font-size:16px"><span style="font-size:10pt">request-ptr=0x7f516eec2060</span></div><div style="font-size:16px"><span style="font-size:10pt">refcount=1</span></div><div style="font-size:16px"><span style="font-size:10pt">wound=yes</span></div><div style="font-size:16px"><span style="font-size:10pt">generation-number=1</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_ret=293822</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_errno=0</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-attempts=1</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-in-progress=yes</span></div></div></div></div></blockquote><div><br></div><div>Note that the sync is still in progress. This means, write-behind has wound the write-request to its children and yet to receive the response (unless there is a bug in accounting of sync-in-progress). So, its likely that there are callstacks into children of write-behind, which are not complete yet. Are you sure the deepest hung call-stack is in write-behind? Can you check for frames with "complete=0"? </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div><div style="font-size:16px"><span style="font-size:10pt">size=293822</span></div><div style="font-size:16px"><span style="font-size:10pt">offset=1048576</span></div><div style="font-size:16px"><span style="font-size:10pt">lied=-1</span></div><div style="font-size:16px"><span style="font-size:10pt">append=0</span></div><div style="font-size:16px"><span style="font-size:10pt">fulfilled=0</span></div><div style="font-size:16px"><span style="font-size:10pt">go=-1</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt">[.FLUSH]</span></div><div style="font-size:16px"><span style="font-size:10pt">request-ptr=0x7f517c2badf0</span></div><div style="font-size:16px"><span style="font-size:10pt">refcount=1</span></div><div style="font-size:16px"><span style="font-size:10pt">wound=no</span></div><div style="font-size:16px"><span style="font-size:10pt">generation-number=2</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_ret=-1</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_errno=116</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-attempts=0</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt">[.FLUSH]</span></div><div style="font-size:16px"><span style="font-size:10pt">request-ptr=0x7f5173e9f7b0</span></div><div style="font-size:16px"><span style="font-size:10pt">refcount=1</span></div><div style="font-size:16px"><span style="font-size:10pt">wound=no</span></div><div style="font-size:16px"><span style="font-size:10pt">generation-number=2</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_ret=0</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_errno=0</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-attempts=0</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt">[.FLUSH]</span></div><div style="font-size:16px"><span style="font-size:10pt">request-ptr=0x7f51640b8ca0</span></div><div style="font-size:16px"><span style="font-size:10pt">refcount=1</span></div><div style="font-size:16px"><span style="font-size:10pt">wound=no</span></div><div style="font-size:16px"><span style="font-size:10pt">generation-number=2</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_ret=0</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_errno=0</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-attempts=0</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt">[.FLUSH]</span></div><div style="font-size:16px"><span style="font-size:10pt">request-ptr=0x7f516f3979d0</span></div><div style="font-size:16px"><span style="font-size:10pt">refcount=1</span></div><div style="font-size:16px"><span style="font-size:10pt">wound=no</span></div><div style="font-size:16px"><span style="font-size:10pt">generation-number=2</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_ret=0</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_errno=0</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-attempts=0</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt">[.FLUSH]</span></div><div style="font-size:16px"><span style="font-size:10pt">request-ptr=0x7f516f6ac8d0</span></div><div style="font-size:16px"><span style="font-size:10pt">refcount=1</span></div><div style="font-size:16px"><span style="font-size:10pt">wound=no</span></div><div style="font-size:16px"><span style="font-size:10pt">generation-number=2</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_ret=0</span></div><div style="font-size:16px"><span style="font-size:10pt">req->op_errno=0</span></div><div style="font-size:16px"><span style="font-size:10pt">sync-attempts=0</span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div><div style="font-size:16px"><span style="font-size:10pt"><br></span></div></div><div><span style="font-size:10pt">Any comments would be appreciated!</span></div><div style="color:rgb(0,0,0);font-family:微软雅黑"><br></div><div><div style="color:rgb(0,0,0);font-family:微软雅黑"><span style="font-size:10pt">Thanks</span></div><div style="color:rgb(0,0,0);font-family:微软雅黑"><span style="font-size:10pt"><span class="gmail-m_-8522292196379778152gmail-m_5197581941865454911gmail-m_672640851088131096Apple-tab-span" style="white-space:pre-wrap"> </span>-Xie</span></div></div><div style="color:rgb(0,0,0);font-family:微软雅黑"><span style="font-size:10pt"><br></span></div><div style="color:rgb(0,0,0);font-family:微软雅黑;font-size:12pt"><br></div></div><div id="gmail-m_-8522292196379778152gmail-m_5197581941865454911gmail-m_672640851088131096SIGNNAME5481"></div><div><span id="gmail-m_-8522292196379778152gmail-m_5197581941865454911gmail-m_672640851088131096_THINKMAILNAME5481"></span></div></div>
</blockquote></div></div>
</div>
</blockquote></div></div>
</div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Pranith<br></div></div></div>