<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=big5">
</head>
<body>
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.gmail-m-1298849780530122127hoenzb
{mso-style-name:gmail-m_-1298849780530122127hoenzb;}
span.gmail-hoenzb
{mso-style-name:gmail-hoenzb;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal">Thanks Amar, I¡¦ll consider your recommendations. But why performance is totally different on two nodes? Will data be written to both nodes at the same time?</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="mso-element:para-border-div;border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="border:none;padding:0in"><b>From: </b><a href="mailto:atumball@redhat.com">Amar Tumballi</a><br>
<b>Sent: </b>Friday, March 31, 2017 3:14 PM<br>
<b>To: </b><a href="mailto:alvin@netvel.net">Alvin Starr</a><br>
<b>Cc: </b><a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org List</a><br>
<b>Subject: </b>Re: [Gluster-users] hanging httpd processes.</p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Mar 31, 2017 at 12:29 PM, Amar Tumballi <span dir="ltr">
<<a href="mailto:atumball@redhat.com" target="_blank">atumball@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>Hi Alvin,<br>
<br>
</div>
Thanks for the dump output. It helped a bit.<br>
<br>
</div>
For now, recommend turning off open-behind and read-ahead performance translators for you to get rid of this situation, As I noticed hung FLUSH operations from these translators.<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Looks like I gave wrong advise by looking at below snippet:<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span style="font-family:monospace,monospace">[global.callpool.stack.61]</span><br>
<span style="font-family:monospace,monospace">stack=0x7f6c6f628f04</span><br>
<span style="font-family:monospace,monospace">uid=48</span><br>
<span style="font-family:monospace,monospace">gid=48</span><br>
<span style="font-family:monospace,monospace">pid=11077</span><br>
<span style="font-family:monospace,monospace">unique=10048797</span><br>
<span style="font-family:monospace,monospace">lk-owner=a73ae5bdb5fcd0d2</span><br>
<span style="font-family:monospace,monospace">op=FLUSH</span><br>
<span style="font-family:monospace,monospace">type=1</span><br>
<span style="font-family:monospace,monospace">cnt=5</span><br>
<span style="font-family:monospace,monospace"></span><br>
<span style="font-family:monospace,monospace">[global.callpool.stack.61.frame.1]</span><br>
<span style="font-family:monospace,monospace">frame=0x7f6c6f793d88</span><br>
<span style="font-family:monospace,monospace">ref_count=0</span><br>
<span style="font-family:monospace,monospace">translator=edocs-production-write-behind</span><br>
<span style="font-family:monospace,monospace">complete=0</span><br>
<span style="font-family:monospace,monospace">parent=edocs-production-read-ahead</span><br>
<span style="font-family:monospace,monospace">wind_from=ra_flush</span><br>
<span style="font-family:monospace,monospace">wind_to=FIRST_CHILD (this)->fops->flush</span><br>
<span style="font-family:monospace,monospace">unwind_to=ra_flush_cbk</span><br>
<span style="font-family:monospace,monospace"></span><br>
<span style="font-family:monospace,monospace">[global.callpool.stack.61.frame.2]</span><br>
<span style="font-family:monospace,monospace">frame=0x7f6c6f796c90</span><br>
<span style="font-family:monospace,monospace">ref_count=1</span><br>
<span style="font-family:monospace,monospace">translator=edocs-production-read-ahead</span><br>
<span style="font-family:monospace,monospace">complete=0</span><br>
<span style="font-family:monospace,monospace">parent=edocs-production-open-behind</span><br>
<span style="font-family:monospace,monospace">wind_from=default_flush_resume</span><br>
<span style="font-family:monospace,monospace">wind_to=FIRST_CHILD(this)->fops->flush</span><br>
<span style="font-family:monospace,monospace">unwind_to=default_flush_cbk</span><br>
<span style="font-family:monospace,monospace"></span><br>
<span style="font-family:monospace,monospace">[global.callpool.stack.61.frame.3]</span><br>
<span style="font-family:monospace,monospace">frame=0x7f6c6f79b724</span><br>
<span style="font-family:monospace,monospace">ref_count=1</span><br>
<span style="font-family:monospace,monospace">translator=edocs-production-open-behind</span><br>
<span style="font-family:monospace,monospace">complete=0</span><br>
<span style="font-family:monospace,monospace">parent=edocs-production</span><br>
<span style="font-family:monospace,monospace">wind_from=io_stats_flush</span><br>
<span style="font-family:monospace,monospace">wind_to=FIRST_CHILD(this)->fops->flush</span><br>
<span style="font-family:monospace,monospace">unwind_to=io_stats_flush_cbk</span><br>
<span style="font-family:monospace,monospace"></span><br>
<span style="font-family:monospace,monospace">[global.callpool.stack.61.frame.4]</span><br>
<span style="font-family:monospace,monospace">frame=0x7f6c6f79b474</span><br>
<span style="font-family:monospace,monospace">ref_count=1</span><br>
<span style="font-family:monospace,monospace">translator=edocs-production</span><br>
<span style="font-family:monospace,monospace">complete=0</span><br>
<span style="font-family:monospace,monospace">parent=fuse</span><br>
<span style="font-family:monospace,monospace">wind_from=fuse_flush_resume</span><br>
<span style="font-family:monospace,monospace">wind_to=FIRST_CHILD(this)->fops->flush</span><br>
<span style="font-family:monospace,monospace">unwind_to=fuse_err_cbk</span><br>
<span style="font-family:monospace,monospace"></span><br>
<span style="font-family:monospace,monospace">[global.callpool.stack.61.frame.5]</span><br>
<span style="font-family:monospace,monospace">frame=0x7f6c6f796684</span><br>
<span style="font-family:monospace,monospace">ref_count=1</span><br>
<span style="font-family:monospace,monospace">translator=fuse</span><br>
<span style="font-family:monospace,monospace">complete=0</span><br>
</blockquote>
</div>
<div> <br>
</div>
<div>Mos probably, issue is with write-behind's flush. So please turn off write-behind and test. If you don't have any hung httpd processes, please let us know.<br>
<br>
</div>
<div>-Amar<br>
</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
-Amar<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">
<div>
<div class="gmail-h5">On Wed, Mar 29, 2017 at 6:56 AM, Alvin Starr <span dir="ltr">
<<a href="mailto:alvin@netvel.net" target="_blank">alvin@netvel.net</a>></span> wrote:<br>
</div>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div class="gmail-h5">We are running gluster 3.8.9-1 on Centos 7.3.1611 for the servers and on the clients 3.7.11-2 on Centos 6.8<br>
<br>
We are seeing httpd processes hang in fuse_request_send or sync_page.<br>
<br>
These calls are from PHP 5.3.3-48 scripts<br>
<br>
I am attaching a tgz file that contains the process dump from glusterfsd and the hung pids along with the offending pid's stacks from /proc/{pid}/stack.<br>
<br>
This has been a low level annoyance for a while but it has become a much bigger issue because the number of hung processes went from a few a week to a few hundred a day.<span class="gmail-m_-1298849780530122127HOEnZb"><font color="#888888"><br>
<br>
<br>
-- <br>
Alvin Starr || voice: (905)513-7688<br>
Netvel Inc. || Cell: (416)806-0133<br>
<a href="mailto:alvin@netvel.net" target="_blank">alvin@netvel.net</a> ||<br>
<br>
</font></span><br>
</div>
</div>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><span class="gmail-HOEnZb"><font color="#888888"><br>
</font></span></blockquote>
</div>
<span class="gmail-HOEnZb"><font color="#888888"><br>
<br clear="all">
<br>
-- <br>
<div class="gmail-m_-1298849780530122127gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>Amar Tumballi (amarts)<br>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>Amar Tumballi (amarts)<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>