<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI Emoji";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:宋体;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:宋体;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:宋体;}
span.m5678744621743138388m2919162071098184476m-131719470587259559gmail-
{mso-style-name:m_5678744621743138388m2919162071098184476m-131719470587259559gmail-;}
span.m5678744621743138388m2919162071098184476m-131719470587259559gmail-hoenzb
{mso-style-name:m_5678744621743138388m2919162071098184476m-131719470587259559gmail-hoenzb;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:DengXian;
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:宋体;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">So I suppose ctime is enough to consider it whether a good iatt or not.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">And why we also include ia_nlink in function
</span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">gf_zero_fill_stat and gf_is_zero_filled_stat ?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">From my investigation, if set ia_nlink to 0, if kernel read the attr with flag RCU, kernel will check the ia_nlink field, when do LINK operation, it will lead to error of
</span><span lang="EN-US" style="font-size:10.5pt">“</span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">files is not exist</span><span lang="EN-US" style="font-size:10.5pt">”</span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">.<o:p></o:p></span></p>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0cm 0cm 1.0pt 0cm">
<p class="MsoNormal" style="border:none;padding:0cm"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New";color:black">if (<span style="background:red;mso-highlight:red">inode->i_nlink == 0</span> && !(inode->i_state & I_LINKABLE))<o:p></o:p></span></p>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0cm 0cm 1.0pt 0cm">
<p class="MsoNormal" style="border:none;padding:0cm"><span lang="EN-US" style="font-family:"Courier New";color:black">
<span style="background:red;mso-highlight:red">error = -ENOENT</span>;<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Best Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">George</span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> gluster-devel-bounces@gluster.org [mailto:gluster-devel-bounces@gluster.org]
<b>On Behalf Of </b>Pranith Kumar Karampuri<br>
<b>Sent:</b> Wednesday, January 24, 2018 4:15 PM<br>
<b>To:</b> Lian, George (NSB - CN/Hangzhou) <george.lian@nokia-sbell.com><br>
<b>Cc:</b> Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou@nokia-sbell.com>; Gluster-devel@gluster.org; Li, Deqian (NSB - CN/Hangzhou) <deqian.li@nokia-sbell.com>; Sun, Ping (NSB - CN/Hangzhou) <ping.sun@nokia-sbell.com><br>
<b>Subject:</b> Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">If ctime is zero, no xlator should consider it as a good iatt. The fact that this is happening means some xlator is not doing proper checks in code. We need to find what that xlator is and fix it. Internet in our new
office is not working so I'm not able to have call today with you guys. What I would do is to put logs in lookup, link, fstat, stat calls to see if anyone unwound iatt with ia_nlink count as zero but ctime as nonzero. <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On 24 Jan 2018 1:03 pm, "Lian, George (NSB - CN/Hangzhou)" <<a href="mailto:george.lian@nokia-sbell.com">george.lian@nokia-sbell.com</a>> wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Hi,
</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Pranith Kumar,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Can you tell me while need set buf->ia_nlink to
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">0</span>”<span lang="EN-US">in function gf_zero_fill_stat(), which API or Application will care it?</span></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">If I remove this line and also update corresponding in function gf_is_zero_filled_stat,
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">The issue seems gone, but I can</span><span style="font-size:10.5pt;font-family:DengXian">’<span lang="EN-US">t
confirm will lead to other issues.</span></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">So could you please double check it and give your comments?</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">My change is as the below:</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">gf_boolean_t</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">gf_is_zero_filled_stat (struct iatt *buf)</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">{</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> if (!buf)</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> return 1;</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> /* Do not use st_dev because it is transformed to store
the xlator id</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> * in place of the device number. Do not use st_ino because
by this time</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> * we've already mapped the root ino to 1 so it is not
guaranteed to be</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> * 0.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> */</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">//</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
if ((buf->ia_nlink == 0) && (buf->ia_ctime == 0))</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> if (buf->ia_ctime == )</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">return 1;</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> return 0;</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">}</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">void</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">gf_zero_fill_stat (struct iatt *buf)</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">{</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">//</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
buf->ia_nlink = 0;</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> buf->ia_ctime = 0;</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">}</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Thanks & Best Regards</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">George</span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Lian,
George (NSB - CN/Hangzhou) <br>
<b>Sent:</b> Friday, January 19, 2018 10:03 AM<br>
<b>To:</b> Pranith Kumar Karampuri <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>>; Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>><br>
<b>Cc:</b> Li, Deqian (NSB - CN/Hangzhou) <<a href="mailto:deqian.li@nokia-sbell.com" target="_blank">deqian.li@nokia-sbell.com</a>>;
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>; Sun, Ping (NSB - CN/Hangzhou) <<a href="mailto:ping.sun@nokia-sbell.com" target="_blank">ping.sun@nokia-sbell.com</a>></span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><br>
<b>Subject:</b> RE: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"<o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Hi,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">>>></span><span lang="EN-US"> Cool, this works for me too. Send me a mail off-list once you are available and we
can figure out a way to get into a call and work on this.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Have you reproduced the issue per the step I listed in
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1531457" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1531457</a> and last mail?<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">If not, I would like you could try it yourself , which the difference between yours and mine is just create only 2 bricks instead of 6 bricks.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">And Cynthia could have a session with you if you needed when I am not available in next Monday and Tuesday.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Thanks & Best Regards,<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">George<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@gluster.org</a> [<a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">mailto:gluster-devel-bounces@gluster.org</a>]
<b>On Behalf Of </b>Pranith Kumar Karampuri<br>
<b>Sent:</b> Thursday, January 18, 2018 6:03 PM<br>
<b>To:</b> Lian, George (NSB - CN/Hangzhou) <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>><br>
<b>Cc:</b> Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>; Li, Deqian (NSB - CN/Hangzhou) <<a href="mailto:deqian.li@nokia-sbell.com" target="_blank">deqian.li@nokia-sbell.com</a>>;
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>; Sun, Ping (NSB - CN/Hangzhou) <<a href="mailto:ping.sun@nokia-sbell.com" target="_blank">ping.sun@nokia-sbell.com</a>><br>
<b>Subject:</b> Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">On Thu, Jan 18, 2018 at 12:17 PM, Lian, George (NSB - CN/Hangzhou) <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>>
wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Hi,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">>>>I actually tried it with replica-2 and replica-3 and then distributed replica-2 before replying to the earlier mail. We can have a debugging session if you
are okay with it.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">It is fine if you can</span><span style="font-size:10.5pt;font-family:DengXian">’<span lang="EN-US">t reproduce
the issue in your ENV.</span></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">And I has attached the detail reproduce log in the Bugzilla FYI</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">But I am sorry I maybe OOO at Monday and Tuesday next week, so debug session will be fine to me at next Wednesday.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Cool, this works for me too. Send me a mail off-list once you are available and we can figure out a way to get into a call and work on this.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Paste the detail reproduce log FYI here:</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">root@ubuntu:~# gluster peer probe ubuntu</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">peer probe: success. Probe on localhost not needed</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">root@ubuntu:~# gluster v create test replica 2 ubuntu:/home/gfs/b1 ubuntu:/home/gfs/b2 force</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">volume create: test: success: please start the volume to access data</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">root@ubuntu:~# gluster v start test</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">volume start: test: success</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">root@ubuntu:~# gluster v info test</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black"> </span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">Volume Name: test</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">Type: Replicate</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">Volume ID: fef5fca3-81d9-46d3-8847-74cde6f701a5</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">Status: Started</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">Snapshot Count: 0</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">Number of Bricks: 1 x 2 = 2</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">Transport-type: tcp</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">Bricks:</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">Brick1: ubuntu:/home/gfs/b1</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">Brick2: ubuntu:/home/gfs/b2</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">Options Reconfigured:</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">transport.address-family: inet</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">nfs.disable: on</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">performance.client-io-threads: off</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">root@ubuntu:~# gluster v status</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">Status of volume: test</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">Gluster process TCP Port RDMA Port Online Pid</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">------------------------------------------------------------------------------</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">Brick ubuntu:/home/gfs/b1 49152 0 Y 7798</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">Brick ubuntu:/home/gfs/b2 49153 0 Y 7818</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">Self-heal Daemon on localhost N/A N/A Y 7839</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black"> </span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">Task Status of Volume test</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">------------------------------------------------------------------------------</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">There are no active volume tasks</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black"> </span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black"> </span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">root@ubuntu:~# gluster v set test cluster.consistent-metadata on</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">volume set: success</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black"> </span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">root@ubuntu:~# ls /mnt/test</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">ls: cannot access '/mnt/test': No such file or directory</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">root@ubuntu:~# mkdir -p /mnt/test</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">root@ubuntu:~# mount -t glusterfs ubuntu:/test /mnt/test</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black"> </span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">root@ubuntu:~# cd /mnt/test</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">root@ubuntu:/mnt/test# echo "abc">aaa</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">root@ubuntu:/mnt/test# cp aaa bbb;link bbb ccc</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black"> </span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">root@ubuntu:/mnt/test# kill -9 7818</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">root@ubuntu:/mnt/test# cp aaa ddd;link ddd eee</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i><span lang="EN-US" style="font-family:"Courier New";color:black">link: cannot create link 'eee' to 'ddd': No such file or directory</span></i><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Best Regards,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">George</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@gluster.org</a> [mailto:<a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@gluster.org</a>]
<b>On Behalf Of </b>Pranith Kumar Karampuri<br>
<b>Sent:</b> Thursday, January 18, 2018 2:40 PM</span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"><br>
<b>To:</b> Lian, George (NSB - CN/Hangzhou) <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>><br>
<b>Cc:</b> Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>;
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>; Li, Deqian (NSB - CN/Hangzhou) <<a href="mailto:deqian.li@nokia-sbell.com" target="_blank">deqian.li@nokia-sbell.com</a>>; Sun, Ping (NSB - CN/Hangzhou) <<a href="mailto:ping.sun@nokia-sbell.com" target="_blank">ping.sun@nokia-sbell.com</a>><br>
<b>Subject:</b> Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">On Thu, Jan 18, 2018 at 6:33 AM, Lian, George (NSB - CN/Hangzhou) <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>>
wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Hi,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">I suppose the brick numbers in your testing is six, and you just shut down the 3 process.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">When I reproduce the issue, I only create a replicate volume with 2 bricks, only let ONE brick working and set cluster.consistent-metadata
on, </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">With this 2 test condition, the issue could 100% reproducible.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Hi,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> I actually tried it with replica-2 and replica-3 and then distributed replica-2 before replying to the earlier mail. We can have a debugging session if
you are okay with it.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">I am in the middle of a customer issue myself(That is the reason for this delay :-( ) and thinking of wrapping it up early next week. Would that be fine with
you?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">16:44:28 :)
</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI Emoji",serif">⚡</span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> gluster v status</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Status of volume: r2</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Gluster process</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
TCP Port</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> RDMA Port</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
Online</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> Pid</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">------------------------------------------------------------------------------</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Brick localhost.localdomain:/home/gfs/r2_0</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
49152</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> 0</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
Y</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> 5309
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Brick localhost.localdomain:/home/gfs/r2_1</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
49154</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> 0</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
Y</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> 5330
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Brick localhost.localdomain:/home/gfs/r2_2</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
49156</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> 0</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
Y</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> 5351
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Brick localhost.localdomain:/home/gfs/r2_3</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
49158</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> 0</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
Y</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> 5372
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Brick localhost.localdomain:/home/gfs/r2_4</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
49159</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> 0</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
Y</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> 5393
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Brick localhost.localdomain:/home/gfs/r2_5</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
49160</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> 0</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
Y</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> 5414
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Self-heal Daemon on localhost</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">N/A</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
N/A</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> Y</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
5436 </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Task Status of Volume r2</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">------------------------------------------------------------------------------</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">There are no active volume tasks</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">root@dhcp35-190 - ~
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">16:44:38 :)
</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Segoe UI Emoji",serif">⚡</span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> kill -9 5309 5351 5393</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Best Regards,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">George</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@gluster.org</a> [mailto:<a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@gluster.org</a>]
<b>On Behalf Of </b>Pranith Kumar Karampuri<br>
<b>Sent:</b> Wednesday, January 17, 2018 7:27 PM<br>
<b>To:</b> Lian, George (NSB - CN/Hangzhou) <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>><br>
<b>Cc:</b> Li, Deqian (NSB - CN/Hangzhou) <<a href="mailto:deqian.li@nokia-sbell.com" target="_blank">deqian.li@nokia-sbell.com</a>>;
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>; Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>; Sun, Ping (NSB - CN/Hangzhou) <<a href="mailto:ping.sun@nokia-sbell.com" target="_blank">ping.sun@nokia-sbell.com</a>></span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"><br>
<b>Subject:</b> Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">On Mon, Jan 15, 2018 at 1:55 PM, Pranith Kumar Karampuri <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>> wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span class="m5678744621743138388m2919162071098184476m-131719470587259559gmail-"><span lang="EN-US">On Mon, Jan 15, 2018 at 8:46 AM, Lian, George (NSB - CN/Hangzhou) <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>>
wrote:</span></span><span lang="EN-US"><o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Hi,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Have you reproduced this issue? If yes, could you please confirm whether it is an issue or not?</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Hi,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> I tried recreating this on my laptop and on both master and 3.12 and I am not able to recreate the issue :-(.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Here is the execution log:
<a href="https://paste.fedoraproject.org/paste/-csXUKrwsbrZAVW1KzggQQ" target="_blank">
https://paste.fedoraproject.org/paste/-csXUKrwsbrZAVW1KzggQQ</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Since I was doing this on my laptop, I changed shutting down of the replica to killing the brick process to simulate this test.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Let me know if I missed something.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Sorry, I am held up with some issue at work, so I think I will get some time day after tomorrow to look at this. In the mean time I am adding more people who
know about afr to see if they get a chance to work on this before me.<o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">And if it is an issue,</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
do you have any solution for this issue?</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Thanks & Best Regards,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">George</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0cm 0cm 0cm;border-color:currentcolor currentcolor">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Lian,
George (NSB - CN/Hangzhou) <br>
<b>Sent:</b> Thursday, January 11, 2018 2:01 PM<br>
<b>To:</b> Pranith Kumar Karampuri <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>><br>
<b>Cc:</b> Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>;
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>; Li, Deqian (NSB - CN/Hangzhou) <<a href="mailto:deqian.li@nokia-sbell.com" target="_blank">deqian.li@nokia-sbell.com</a>>; Sun, Ping (NSB - CN/Hangzhou) <<a href="mailto:ping.sun@nokia-sbell.com" target="_blank">ping.sun@nokia-sbell.com</a>><br>
<b>Subject:</b> RE: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Hi,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Please see detail test step on
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1531457" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1531457</a></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-family:"Courier New";color:black">How reproducible:</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-family:"Courier New";color:black"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-family:"Courier New";color:black"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-family:"Courier New";color:black">Steps to Reproduce:</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-family:"Courier New";color:black">1.create a volume name "test" with replicated
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-family:"Courier New";color:black">2.set volume option cluster.consistent-metadata with on:
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-family:"Courier New";color:black"> gluster v set test cluster.consistent-metadata on</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-family:"Courier New";color:black">3. mount volume test on client on /mnt/test</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-family:"Courier New";color:black">4. create a file aaa size more than 1 byte</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-family:"Courier New";color:black"> echo "1234567890" >/mnt/test/aaa</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-family:"Courier New";color:black">5. shutdown a replicat node, let's say sn-1, only let sn-0 worked</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-family:"Courier New";color:black">6. cp /mnt/test/aaa /mnt/test/bbb; link /mnt/test/bbb /mnt/test/ccc</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">BRs</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">George</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@gluster.org</a> [<a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">mailto:gluster-devel-bounces@gluster.org</a>]
<b>On Behalf Of </b>Pranith Kumar Karampuri<br>
<b>Sent:</b> Thursday, January 11, 2018 12:39 PM<br>
<b>To:</b> Lian, George (NSB - CN/Hangzhou) <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>><br>
<b>Cc:</b> Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>;
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>; Li, Deqian (NSB - CN/Hangzhou) <<a href="mailto:deqian.li@nokia-sbell.com" target="_blank">deqian.li@nokia-sbell.com</a>>; Sun, Ping (NSB - CN/Hangzhou) <<a href="mailto:ping.sun@nokia-sbell.com" target="_blank">ping.sun@nokia-sbell.com</a>><br>
<b>Subject:</b> Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">On Thu, Jan 11, 2018 at 6:35 AM, Lian, George (NSB - CN/Hangzhou) <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>>
wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Hi,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">>>></span><span lang="EN-US"> In which protocol are you seeing this issue? Fuse/NFS/SMB?<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">It is fuse, within mountpoint by
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">mount -t glusterfs</span></span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
</span><span style="font-size:10.5pt;font-family:DengXian">…“<span lang="EN-US"> command.</span></span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Could you let me know the test you did so that I can try to re-create and see what exactly is going on?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Configuration of the volume and the steps to re-create the issue you are seeing would be helpful in debugging the issue further.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Thanks & Best Regards,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">George</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@gluster.org</a> [mailto:<a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@gluster.org</a>]
<b>On Behalf Of </b>Pranith Kumar Karampuri<br>
<b>Sent:</b> Wednesday, January 10, 2018 8:08 PM<br>
<b>To:</b> Lian, George (NSB - CN/Hangzhou) <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>><br>
<b>Cc:</b> Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>; Zhong, Hua (NSB - CN/Hangzhou) <<a href="mailto:hua.zhong@nokia-sbell.com" target="_blank">hua.zhong@nokia-sbell.com</a>>;
Li, Deqian (NSB - CN/Hangzhou) <<a href="mailto:deqian.li@nokia-sbell.com" target="_blank">deqian.li@nokia-sbell.com</a>>;
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>; Sun, Ping (NSB - CN/Hangzhou) <<a href="mailto:ping.sun@nokia-sbell.com" target="_blank">ping.sun@nokia-sbell.com</a>><br>
<b>Subject:</b> Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"</span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">On Wed, Jan 10, 2018 at 11:09 AM, Lian, George (NSB - CN/Hangzhou) <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>>
wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Hi, Pranith Kumar,<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">I has create a bug on Bugzilla
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1531457" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1531457</a><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">After my investigation for this link issue, I suppose your changes on afr-dir-write.c with issue " Don't let NFS cache stat after writes" , your fix is like:<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">--------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> if (afr_txn_nothing_failed (frame, this)) {<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> /*if it did pre-op, it will do post-op changing ctime*/<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> if (priv->consistent_metadata &&<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> afr_needs_changelog_update (local))<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> afr_zero_fill_stat (local);<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> local->transaction.unwind (frame, this);<o:p></o:p></span></p>
<div style="border:none;border-bottom:solid windowtext 1.0pt;padding:0cm 0cm 1.0pt 0cm;border-color:currentcolor currentcolor windowtext">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> }<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">In the above fix, it set the ia_nlink to
</span>‘<span lang="EN-US">0</span>’<span lang="EN-US"> if option consistent-metadata is set to
</span>“<span lang="EN-US">on</span>”<span lang="EN-US">.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">And hard link a file with which just created will lead to an error, and the error is caused in kernel function
</span>“<span lang="EN-US">vfs_link</span>”<span lang="EN-US">:<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-family:"Courier New";color:black">if (inode->i_nlink == 0 && !(inode->i_state & I_LINKABLE))</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-family:"Courier New";color:black"> error = -ENOENT;</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">could you please have a check and give some comments here?<o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">When stat is "zero filled", understanding is that the higher layer protocol doesn't send stat value to the kernel and a separate lookup is sent by the kernel
to get the latest stat value. In which protocol are you seeing this issue? Fuse/NFS/SMB?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Thanks & Best Regards,<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">George<o:p></o:p></span></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"><br>
<br clear="all">
<br>
-- <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Pranith<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"><br>
<br clear="all">
<br>
-- <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Pranith<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="color:#888888"><br>
<br clear="all">
<br>
<span class="m5678744621743138388m2919162071098184476m-131719470587259559gmail-hoenzb">--
</span></span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="color:#888888">Pranith</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"><br>
<br clear="all">
<br>
-- <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Pranith<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"><br>
<br clear="all">
<br>
-- <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Pranith<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"><br>
<br clear="all">
<br>
-- <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Pranith<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</body>
</html>