<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 31, 2017 at 12:29 PM, Amar Tumballi <span dir="ltr"><<a href="mailto:atumball@redhat.com" target="_blank">atumball@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div>Hi Alvin,<br><br></div>Thanks for the dump output. It helped a bit.<br><br></div>For now, recommend turning off open-behind and read-ahead performance translators for you to get rid of this situation, As I noticed hung FLUSH operations from these translators.<br></div></div></blockquote><div><br></div><div>Looks like I gave wrong advise by looking at below snippet:<br><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="font-family:monospace,monospace">[global.callpool.stack.61]</span><br><span style="font-family:monospace,monospace">stack=0x7f6c6f628f04</span><br><span style="font-family:monospace,monospace">uid=48</span><br><span style="font-family:monospace,monospace">gid=48</span><br><span style="font-family:monospace,monospace">pid=11077</span><br><span style="font-family:monospace,monospace">unique=10048797</span><br><span style="font-family:monospace,monospace">lk-owner=a73ae5bdb5fcd0d2</span><br><span style="font-family:monospace,monospace">op=FLUSH</span><br><span style="font-family:monospace,monospace">type=1</span><br><span style="font-family:monospace,monospace">cnt=5</span><br><span style="font-family:monospace,monospace"></span><br><span style="font-family:monospace,monospace">[global.callpool.stack.61.frame.1]</span><br><span style="font-family:monospace,monospace">frame=0x7f6c6f793d88</span><br><span style="font-family:monospace,monospace">ref_count=0</span><br><span style="font-family:monospace,monospace">translator=edocs-production-write-behind</span><br><span style="font-family:monospace,monospace">complete=0</span><br><span style="font-family:monospace,monospace">parent=edocs-production-read-ahead</span><br><span style="font-family:monospace,monospace">wind_from=ra_flush</span><br><span style="font-family:monospace,monospace">wind_to=FIRST_CHILD (this)->fops->flush</span><br><span style="font-family:monospace,monospace">unwind_to=ra_flush_cbk</span><br><span style="font-family:monospace,monospace"></span><br><span style="font-family:monospace,monospace">[global.callpool.stack.61.frame.2]</span><br><span style="font-family:monospace,monospace">frame=0x7f6c6f796c90</span><br><span style="font-family:monospace,monospace">ref_count=1</span><br><span style="font-family:monospace,monospace">translator=edocs-production-read-ahead</span><br><span style="font-family:monospace,monospace">complete=0</span><br><span style="font-family:monospace,monospace">parent=edocs-production-open-behind</span><br><span style="font-family:monospace,monospace">wind_from=default_flush_resume</span><br><span style="font-family:monospace,monospace">wind_to=FIRST_CHILD(this)->fops->flush</span><br><span style="font-family:monospace,monospace">unwind_to=default_flush_cbk</span><br><span style="font-family:monospace,monospace"></span><br><span style="font-family:monospace,monospace">[global.callpool.stack.61.frame.3]</span><br><span style="font-family:monospace,monospace">frame=0x7f6c6f79b724</span><br><span style="font-family:monospace,monospace">ref_count=1</span><br><span style="font-family:monospace,monospace">translator=edocs-production-open-behind</span><br><span style="font-family:monospace,monospace">complete=0</span><br><span style="font-family:monospace,monospace">parent=edocs-production</span><br><span style="font-family:monospace,monospace">wind_from=io_stats_flush</span><br><span style="font-family:monospace,monospace">wind_to=FIRST_CHILD(this)->fops->flush</span><br><span style="font-family:monospace,monospace">unwind_to=io_stats_flush_cbk</span><br><span style="font-family:monospace,monospace"></span><br><span style="font-family:monospace,monospace">[global.callpool.stack.61.frame.4]</span><br><span style="font-family:monospace,monospace">frame=0x7f6c6f79b474</span><br><span style="font-family:monospace,monospace">ref_count=1</span><br><span style="font-family:monospace,monospace">translator=edocs-production</span><br><span style="font-family:monospace,monospace">complete=0</span><br><span style="font-family:monospace,monospace">parent=fuse</span><br><span style="font-family:monospace,monospace">wind_from=fuse_flush_resume</span><br><span style="font-family:monospace,monospace">wind_to=FIRST_CHILD(this)->fops->flush</span><br><span style="font-family:monospace,monospace">unwind_to=fuse_err_cbk</span><br><span style="font-family:monospace,monospace"></span><br><span style="font-family:monospace,monospace">[global.callpool.stack.61.frame.5]</span><br><span style="font-family:monospace,monospace">frame=0x7f6c6f796684</span><br><span style="font-family:monospace,monospace">ref_count=1</span><br><span style="font-family:monospace,monospace">translator=fuse</span><br><span style="font-family:monospace,monospace">complete=0</span><br></blockquote></div><div> <br></div><div>Mos probably, issue is with write-behind's flush. So please turn off write-behind and test. If you don't have any hung httpd processes, please let us know.<br><br></div><div>-Amar<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div>-Amar<br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="gmail-h5">On Wed, Mar 29, 2017 at 6:56 AM, Alvin Starr <span dir="ltr"><<a href="mailto:alvin@netvel.net" target="_blank">alvin@netvel.net</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-h5">We are running gluster 3.8.9-1 on Centos 7.3.1611 for the servers and on the clients 3.7.11-2 on Centos 6.8<br>
<br>
We are seeing httpd processes hang in fuse_request_send or sync_page.<br>
<br>
These calls are from PHP 5.3.3-48 scripts<br>
<br>
I am attaching a tgz file that contains the process dump from glusterfsd and the hung pids along with the offending pid's stacks from /proc/{pid}/stack.<br>
<br>
This has been a low level annoyance for a while but it has become a much bigger issue because the number of hung processes went from a few a week to a few hundred a day.<span class="gmail-m_-1298849780530122127HOEnZb"><font color="#888888"><br>
<br>
<br>
-- <br>
Alvin Starr || voice: (905)513-7688<br>
Netvel Inc. || Cell: (416)806-0133<br>
<a href="mailto:alvin@netvel.net" target="_blank">alvin@netvel.net</a> ||<br>
<br>
</font></span><br></div></div>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><span class="gmail-HOEnZb"><font color="#888888"><br></font></span></blockquote></div><span class="gmail-HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="gmail-m_-1298849780530122127gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>
</div></div>