<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI Emoji";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:宋体;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:宋体;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:DengXian;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:DengXian;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Hi, GlusterFS expert,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Our fix is as the patch, after test in our ENV, the issue seems gone, but I can’t confirm whether it will lead to other issue or not, And we suppose our change is not so fine.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">So please share your comments , and hope you can give a good solution.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Thanks & Best Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Storage DEV from Nokia<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> gluster-devel-bounces@gluster.org [mailto:gluster-devel-bounces@gluster.org]
<b>On Behalf Of </b>Pranith Kumar Karampuri<br>
<b>Sent:</b> Monday, January 29, 2018 8:23 AM<br>
<b>To:</b> Lian, George (NSB - CN/Hangzhou) <george.lian@nokia-sbell.com><br>
<b>Cc:</b> Li, Deqian (NSB - CN/Hangzhou) <deqian.li@nokia-sbell.com>; Gluster-devel@gluster.org; Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou@nokia-sbell.com>; Sun, Ping (NSB - CN/Hangzhou) <ping.sun@nokia-sbell.com><br>
<b>Subject:</b> Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">+Ravi, +Raghavendra G<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On 25 Jan 2018 8:49 am, "Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>> wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On 25 Jan 2018 8:43 am, "Lian, George (NSB - CN/Hangzhou)" <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>> wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal"><span lang="EN-US">Hi,<br>
<br>
I suppose the zero filled attr is for performance consider to NFS, but for fuse, it will lead issue such like hard LINK FOP,<br>
So I suggest could we add 2 attr field in the endof "struct iatt {", such like ia_fuse_nlink, ia_fuse_ctime,<br>
And in function gf_zero_fill_stat , saved the ia_nlink, ia_ctime to ia_fuse_nlink,ia_fuse_ctime before set its to zero,<br>
And restore it to valued nlink and ctime in function gf_fuse_stat2attr,<br>
So that kernel could get the correct nlink and ctime.<br>
<br>
Is it a considerable solution? Any risk?<br>
<br>
Please share your comments, thanks in advance!<o:p></o:p></span></p>
</blockquote>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Adding csaba for helping with this.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal"><span lang="EN-US"><br>
Best Regards,<br>
George<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><br>
-----Original Message-----<br>
From: <a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@gluster.org</a> [mailto:<a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@gluster.org</a>] On Behalf Of Niels de Vos<br>
Sent: Wednesday, January 24, 2018 7:43 PM<br>
To: Pranith Kumar Karampuri <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Cc: Lian, George (NSB - CN/Hangzhou) <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>>; Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>;
Li, Deqian (NSB - CN/Hangzhou) <<a href="mailto:deqian.li@nokia-sbell.com" target="_blank">deqian.li@nokia-sbell.com</a>>;
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>; Sun, Ping (NSB - CN/Hangzhou) <<a href="mailto:ping.sun@nokia-sbell.com" target="_blank">ping.sun@nokia-sbell.com</a>><br>
Subject: Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"<br>
<br>
On Wed, Jan 24, 2018 at 02:24:06PM +0530, Pranith Kumar Karampuri wrote:<br>
> hi,<br>
> In the same commit you mentioned earlier, there was this code<br>
> earlier:<br>
> -/* Returns 1 if the stat seems to be filled with zeroes. */ -int<br>
> -nfs_zero_filled_stat (struct iatt *buf) -{<br>
> - if (!buf)<br>
> - return 1;<br>
> -<br>
> - /* Do not use st_dev because it is transformed to store the xlator<br>
> id<br>
> - * in place of the device number. Do not use st_ino because by<br>
> this time<br>
> - * we've already mapped the root ino to 1 so it is not guaranteed<br>
> to be<br>
> - * 0.<br>
> - */<br>
> - if ((buf->ia_nlink == 0) && (buf->ia_ctime == 0))<br>
> - return 1;<br>
> -<br>
> - return 0;<br>
> -}<br>
> -<br>
> -<br>
><br>
> I moved this to a common library function that can be used in afr as well.<br>
> Why was it there in NFS? +Niels for answering that question.<br>
<br>
Sorry, I dont know why that was done. It was introduced with the initial gNFS implementation, long before I started to work with Gluster. The only reference I have is this from<br>
xlators/nfs/server/src/nfs3-helpers.c:nfs3_stat_to_post_op_attr()<br>
<br>
371 /* Some performance translators return zero-filled stats when they<br>
372 * do not have up-to-date attributes. Need to handle this by not<br>
373 * returning these zeroed out attrs.<br>
374 */<br>
<br>
This may not be true for the current situation anymore.<br>
<br>
HTH,<br>
Niels<br>
<br>
<br>
><br>
> If I give you a patch which will assert the error condition, would it<br>
> be possible for you to figure out the first xlator which is unwinding<br>
> the iatt with nlink count as zero but ctime as non-zero?<br>
><br>
> On Wed, Jan 24, 2018 at 1:03 PM, Lian, George (NSB - CN/Hangzhou) <<br>
> <a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>> wrote:<br>
><br>
> > Hi, Pranith Kumar,<br>
> ><br>
> ><br>
> ><br>
> > Can you tell me while need set buf->ia_nlink to “0”in function<br>
> > gf_zero_fill_stat(), which API or Application will care it?<br>
> ><br>
> > If I remove this line and also update corresponding in function<br>
> > gf_is_zero_filled_stat,<br>
> ><br>
> > The issue seems gone, but I can’t confirm will lead to other issues.<br>
> ><br>
> ><br>
> ><br>
> > So could you please double check it and give your comments?<br>
> ><br>
> ><br>
> ><br>
> > My change is as the below:<br>
> ><br>
> ><br>
> ><br>
> > gf_boolean_t<br>
> ><br>
> > gf_is_zero_filled_stat (struct iatt *buf)<br>
> ><br>
> > {<br>
> ><br>
> > if (!buf)<br>
> ><br>
> > return 1;<br>
> ><br>
> ><br>
> ><br>
> > /* Do not use st_dev because it is transformed to store the<br>
> > xlator id<br>
> ><br>
> > * in place of the device number. Do not use st_ino because<br>
> > by this time<br>
> ><br>
> > * we've already mapped the root ino to 1 so it is not<br>
> > guaranteed to be<br>
> ><br>
> > * 0.<br>
> ><br>
> > */<br>
> ><br>
> > // if ((buf->ia_nlink == 0) && (buf->ia_ctime == 0))<br>
> ><br>
> > if (buf->ia_ctime == )<br>
> ><br>
> > return 1;<br>
> ><br>
> ><br>
> ><br>
> > return 0;<br>
> ><br>
> > }<br>
> ><br>
> ><br>
> ><br>
> > void<br>
> ><br>
> > gf_zero_fill_stat (struct iatt *buf)<br>
> ><br>
> > {<br>
> ><br>
> > // buf->ia_nlink = 0;<br>
> ><br>
> > buf->ia_ctime = 0;<br>
> ><br>
> > }<br>
> ><br>
> ><br>
> ><br>
> > Thanks & Best Regards<br>
> ><br>
> > George<br>
> ><br>
> > *From:* Lian, George (NSB - CN/Hangzhou)<br>
> > *Sent:* Friday, January 19, 2018 10:03 AM<br>
> > *To:* Pranith Kumar Karampuri <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>>; Zhou, Cynthia<br>
> > (NSB -<br>
> > CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>><br>
> > *Cc:* Li, Deqian (NSB - CN/Hangzhou) <<a href="mailto:deqian.li@nokia-sbell.com" target="_blank">deqian.li@nokia-sbell.com</a>>;<br>
> > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>; Sun, Ping (NSB - CN/Hangzhou) <<br>
> > <a href="mailto:ping.sun@nokia-sbell.com" target="_blank">ping.sun@nokia-sbell.com</a>><br>
> ><br>
> > *Subject:* RE: [Gluster-devel] a link issue maybe introduced in a<br>
> > bug fix " Don't let NFS cache stat after writes"<br>
> ><br>
> ><br>
> ><br>
> > Hi,<br>
> ><br>
> > >>> Cool, this works for me too. Send me a mail off-list once you<br>
> > >>> are<br>
> > available and we can figure out a way to get into a call and work on this.<br>
> ><br>
> ><br>
> ><br>
> > Have you reproduced the issue per the step I listed in<br>
> > <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1531457" target="_blank">
https://bugzilla.redhat.com/show_bug.cgi?id=1531457</a> and last mail?<br>
> ><br>
> ><br>
> ><br>
> > If not, I would like you could try it yourself , which the<br>
> > difference between yours and mine is just create only 2 bricks instead of 6 bricks.<br>
> ><br>
> ><br>
> ><br>
> > And Cynthia could have a session with you if you needed when I am<br>
> > not available in next Monday and Tuesday.<br>
> ><br>
> ><br>
> ><br>
> > Thanks & Best Regards,<br>
> ><br>
> > George<br>
> ><br>
> ><br>
> ><br>
> > *From:* <a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@gluster.org</a><br>
> > [mailto:<a href="mailto:gluster-devel-bounces@" target="_blank">gluster-devel-bounces@</a>
<a href="http://gluster.org" target="_blank">gluster.org</a><br>
> > <<a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@gluster.org</a>>] *On Behalf Of *Pranith Kumar<br>
> > Karampuri<br>
> > *Sent:* Thursday, January 18, 2018 6:03 PM<br>
> > *To:* Lian, George (NSB - CN/Hangzhou) <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>><br>
> > *Cc:* Zhou, Cynthia (NSB - CN/Hangzhou)<br>
> > <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>; Li, Deqian (NSB - CN/Hangzhou)<br>
> > <<a href="mailto:deqian.li@nokia-sbell.com" target="_blank">deqian.li@nokia-sbell.com</a>>;
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>; Sun, Ping<br>
> > (NSB - CN/Hangzhou) < <a href="mailto:ping.sun@nokia-sbell.com" target="_blank">
ping.sun@nokia-sbell.com</a>><br>
> > *Subject:* Re: [Gluster-devel] a link issue maybe introduced in a<br>
> > bug fix " Don't let NFS cache stat after writes"<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Thu, Jan 18, 2018 at 12:17 PM, Lian, George (NSB - CN/Hangzhou) <<br>
> > <a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>> wrote:<br>
> ><br>
> > Hi,<br>
> ><br>
> > >>>I actually tried it with replica-2 and replica-3 and then<br>
> > >>>distributed<br>
> > replica-2 before replying to the earlier mail. We can have a<br>
> > debugging session if you are okay with it.<br>
> ><br>
> ><br>
> ><br>
> > It is fine if you can’t reproduce the issue in your ENV.<br>
> ><br>
> > And I has attached the detail reproduce log in the Bugzilla FYI<br>
> ><br>
> ><br>
> ><br>
> > But I am sorry I maybe OOO at Monday and Tuesday next week, so debug<br>
> > session will be fine to me at next Wednesday.<br>
> ><br>
> ><br>
> ><br>
> > Cool, this works for me too. Send me a mail off-list once you are<br>
> > available and we can figure out a way to get into a call and work on this.<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > Paste the detail reproduce log FYI here:<br>
> ><br>
> > *root@ubuntu:~# gluster peer probe ubuntu*<br>
> ><br>
> > *peer probe: success. Probe on localhost not needed*<br>
> ><br>
> > *root@ubuntu:~# gluster v create test replica 2 ubuntu:/home/gfs/b1<br>
> > ubuntu:/home/gfs/b2 force*<br>
> ><br>
> > *volume create: test: success: please start the volume to access<br>
> > data*<br>
> ><br>
> > *root@ubuntu:~# gluster v start test*<br>
> ><br>
> > *volume start: test: success*<br>
> ><br>
> > *root@ubuntu:~# gluster v info test*<br>
> ><br>
> ><br>
> ><br>
> > *Volume Name: test*<br>
> ><br>
> > *Type: Replicate*<br>
> ><br>
> > *Volume ID: fef5fca3-81d9-46d3-8847-74cde6f701a5*<br>
> ><br>
> > *Status: Started*<br>
> ><br>
> > *Snapshot Count: 0*<br>
> ><br>
> > *Number of Bricks: 1 x 2 = 2*<br>
> ><br>
> > *Transport-type: tcp*<br>
> ><br>
> > *Bricks:*<br>
> ><br>
> > *Brick1: ubuntu:/home/gfs/b1*<br>
> ><br>
> > *Brick2: ubuntu:/home/gfs/b2*<br>
> ><br>
> > *Options Reconfigured:*<br>
> ><br>
> > *transport.address-family: inet*<br>
> ><br>
> > *nfs.disable: on*<br>
> ><br>
> > *performance.client-io-threads: off*<br>
> ><br>
> > *root@ubuntu:~# gluster v status*<br>
> ><br>
> > *Status of volume: test*<br>
> ><br>
> > *Gluster process TCP Port RDMA Port Online<br>
> > Pid*<br>
> ><br>
> ><br>
> > *-------------------------------------------------------------------<br>
> > -----------*<br>
> ><br>
> > *Brick ubuntu:/home/gfs/b1 49152 0 Y<br>
> > 7798*<br>
> ><br>
> > *Brick ubuntu:/home/gfs/b2 49153 0 Y<br>
> > 7818*<br>
> ><br>
> > *Self-heal Daemon on localhost N/A N/A Y<br>
> > 7839*<br>
> ><br>
> ><br>
> ><br>
> > *Task Status of Volume test*<br>
> ><br>
> ><br>
> > *-------------------------------------------------------------------<br>
> > -----------*<br>
> ><br>
> > *There are no active volume tasks*<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > *root@ubuntu:~# gluster v set test cluster.consistent-metadata on*<br>
> ><br>
> > *volume set: success*<br>
> ><br>
> ><br>
> ><br>
> > *root@ubuntu:~# ls /mnt/test*<br>
> ><br>
> > *ls: cannot access '/mnt/test': No such file or directory*<br>
> ><br>
> > *root@ubuntu:~# mkdir -p /mnt/test*<br>
> ><br>
> > *root@ubuntu:~# mount -t glusterfs ubuntu:/test /mnt/test*<br>
> ><br>
> ><br>
> ><br>
> > *root@ubuntu:~# cd /mnt/test*<br>
> ><br>
> > *root@ubuntu:/mnt/test# echo "abc">aaa*<br>
> ><br>
> > *root@ubuntu:/mnt/test# cp aaa bbb;link bbb ccc*<br>
> ><br>
> ><br>
> ><br>
> > *root@ubuntu:/mnt/test# kill -9 7818*<br>
> ><br>
> > *root@ubuntu:/mnt/test# cp aaa ddd;link ddd eee*<br>
> ><br>
> > *link: cannot create link 'eee' to 'ddd': No such file or directory*<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > Best Regards,<br>
> ><br>
> > George<br>
> ><br>
> ><br>
> ><br>
> > *From:* <a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@gluster.org</a><br>
> > [mailto:<a href="mailto:gluster-devel-bounces@" target="_blank">gluster-devel-bounces@</a>
<a href="http://gluster.org" target="_blank">gluster.org</a>] *On Behalf Of *Pranith<br>
> > Kumar Karampuri<br>
> > *Sent:* Thursday, January 18, 2018 2:40 PM<br>
> ><br>
> ><br>
> > *To:* Lian, George (NSB - CN/Hangzhou) <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>><br>
> > *Cc:* Zhou, Cynthia (NSB - CN/Hangzhou)<br>
> > <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>;
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>; Li,<br>
> > Deqian (NSB - CN/Hangzhou) < <a href="mailto:deqian.li@nokia-sbell.com" target="_blank">
deqian.li@nokia-sbell.com</a>>; Sun, Ping<br>
> > (NSB - CN/Hangzhou) < <a href="mailto:ping.sun@nokia-sbell.com" target="_blank">
ping.sun@nokia-sbell.com</a>><br>
> > *Subject:* Re: [Gluster-devel] a link issue maybe introduced in a<br>
> > bug fix " Don't let NFS cache stat after writes"<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Thu, Jan 18, 2018 at 6:33 AM, Lian, George (NSB - CN/Hangzhou) <<br>
> > <a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>> wrote:<br>
> ><br>
> > Hi,<br>
> ><br>
> > I suppose the brick numbers in your testing is six, and you just<br>
> > shut down the 3 process.<br>
> ><br>
> > When I reproduce the issue, I only create a replicate volume with 2<br>
> > bricks, only let ONE brick working and set<br>
> > cluster.consistent-metadata on,<br>
> ><br>
> > With this 2 test condition, the issue could 100% reproducible.<br>
> ><br>
> ><br>
> ><br>
> > Hi,<br>
> ><br>
> > I actually tried it with replica-2 and replica-3 and then<br>
> > distributed replica-2 before replying to the earlier mail. We can<br>
> > have a debugging session if you are okay with it.<br>
> ><br>
> > I am in the middle of a customer issue myself(That is the reason for<br>
> > this delay :-( ) and thinking of wrapping it up early next week.<br>
> > Would that be fine with you?<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > 16:44:28 :) </span><span lang="EN-US" style="font-family:"Segoe UI Emoji",serif">⚡</span><span lang="EN-US"> gluster v status<br>
> ><br>
> > Status of volume: r2<br>
> ><br>
> > Gluster process TCP Port RDMA Port Online<br>
> > Pid<br>
> ><br>
> > ------------------------------------------------------------<br>
> > ------------------<br>
> ><br>
> > Brick localhost.localdomain:/home/gfs/r2_0 49152 0 Y<br>
> > 5309<br>
> ><br>
> > Brick localhost.localdomain:/home/gfs/r2_1 49154 0 Y<br>
> > 5330<br>
> ><br>
> > Brick localhost.localdomain:/home/gfs/r2_2 49156 0 Y<br>
> > 5351<br>
> ><br>
> > Brick localhost.localdomain:/home/gfs/r2_3 49158 0 Y<br>
> > 5372<br>
> ><br>
> > Brick localhost.localdomain:/home/gfs/r2_4 49159 0 Y<br>
> > 5393<br>
> ><br>
> > Brick localhost.localdomain:/home/gfs/r2_5 49160 0 Y<br>
> > 5414<br>
> ><br>
> > Self-heal Daemon on localhost N/A N/A Y<br>
> > 5436<br>
> ><br>
> ><br>
> ><br>
> > Task Status of Volume r2<br>
> ><br>
> > ------------------------------------------------------------<br>
> > ------------------<br>
> ><br>
> > There are no active volume tasks<br>
> ><br>
> ><br>
> ><br>
> > root@dhcp35-190 - ~<br>
> ><br>
> > 16:44:38 :) </span><span lang="EN-US" style="font-family:"Segoe UI Emoji",serif">⚡</span><span lang="EN-US"> kill -9 5309 5351 5393<br>
> ><br>
> ><br>
> ><br>
> > Best Regards,<br>
> ><br>
> > George<br>
> ><br>
> > *From:* <a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@gluster.org</a><br>
> > [mailto:<a href="mailto:gluster-devel-bounces@" target="_blank">gluster-devel-bounces@</a>
<a href="http://gluster.org" target="_blank">gluster.org</a>] *On Behalf Of *Pranith<br>
> > Kumar Karampuri<br>
> > *Sent:* Wednesday, January 17, 2018 7:27 PM<br>
> > *To:* Lian, George (NSB - CN/Hangzhou) <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>><br>
> > *Cc:* Li, Deqian (NSB - CN/Hangzhou) <<a href="mailto:deqian.li@nokia-sbell.com" target="_blank">deqian.li@nokia-sbell.com</a>>;<br>
> > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>; Zhou, Cynthia (NSB - CN/Hangzhou) <<br>
> > <a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>; Sun, Ping (NSB - CN/Hangzhou) <<br>
> > <a href="mailto:ping.sun@nokia-sbell.com" target="_blank">ping.sun@nokia-sbell.com</a>><br>
> ><br>
> ><br>
> > *Subject:* Re: [Gluster-devel] a link issue maybe introduced in a<br>
> > bug fix " Don't let NFS cache stat after writes"<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Mon, Jan 15, 2018 at 1:55 PM, Pranith Kumar Karampuri <<br>
> > <a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>> wrote:<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Mon, Jan 15, 2018 at 8:46 AM, Lian, George (NSB - CN/Hangzhou) <<br>
> > <a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>> wrote:<br>
> ><br>
> > Hi,<br>
> ><br>
> ><br>
> ><br>
> > Have you reproduced this issue? If yes, could you please confirm<br>
> > whether it is an issue or not?<br>
> ><br>
> ><br>
> ><br>
> > Hi,<br>
> ><br>
> > I tried recreating this on my laptop and on both master and<br>
> > 3.12 and I am not able to recreate the issue :-(.<br>
> ><br>
> > Here is the execution log: <a href="https://paste.fedoraproject.org/paste/-" target="_blank">
https://paste.fedoraproject.org/paste/-</a><br>
> > csXUKrwsbrZAVW1KzggQQ<br>
> ><br>
> > Since I was doing this on my laptop, I changed shutting down of the<br>
> > replica to killing the brick process to simulate this test.<br>
> ><br>
> > Let me know if I missed something.<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > Sorry, I am held up with some issue at work, so I think I will get<br>
> > some time day after tomorrow to look at this. In the mean time I am<br>
> > adding more people who know about afr to see if they get a chance to<br>
> > work on this before me.<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > And if it is an issue, do you have any solution for this issue?<br>
> ><br>
> ><br>
> ><br>
> > Thanks & Best Regards,<br>
> ><br>
> > George<br>
> ><br>
> ><br>
> ><br>
> > *From:* Lian, George (NSB - CN/Hangzhou)<br>
> > *Sent:* Thursday, January 11, 2018 2:01 PM<br>
> > *To:* Pranith Kumar Karampuri <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>><br>
> > *Cc:* Zhou, Cynthia (NSB - CN/Hangzhou)<br>
> > <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>;
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>; Li,<br>
> > Deqian (NSB - CN/Hangzhou) < <a href="mailto:deqian.li@nokia-sbell.com" target="_blank">
deqian.li@nokia-sbell.com</a>>; Sun, Ping<br>
> > (NSB - CN/Hangzhou) < <a href="mailto:ping.sun@nokia-sbell.com" target="_blank">
ping.sun@nokia-sbell.com</a>><br>
> > *Subject:* RE: [Gluster-devel] a link issue maybe introduced in a<br>
> > bug fix " Don't let NFS cache stat after writes"<br>
> ><br>
> ><br>
> ><br>
> > Hi,<br>
> ><br>
> ><br>
> ><br>
> > Please see detail test step on <a href="https://bugzilla.redhat.com/" target="_blank">
https://bugzilla.redhat.com/</a><br>
> > show_bug.cgi?id=1531457<br>
> ><br>
> ><br>
> ><br>
> > How reproducible:<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > Steps to Reproduce:<br>
> ><br>
> > 1.create a volume name "test" with replicated<br>
> ><br>
> > 2.set volume option cluster.consistent-metadata with on:<br>
> ><br>
> > gluster v set test cluster.consistent-metadata on<br>
> ><br>
> > 3. mount volume test on client on /mnt/test<br>
> ><br>
> > 4. create a file aaa size more than 1 byte<br>
> ><br>
> > echo "1234567890" >/mnt/test/aaa<br>
> ><br>
> > 5. shutdown a replicat node, let's say sn-1, only let sn-0 worked<br>
> ><br>
> > 6. cp /mnt/test/aaa /mnt/test/bbb; link /mnt/test/bbb /mnt/test/ccc<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > BRs<br>
> ><br>
> > George<br>
> ><br>
> ><br>
> ><br>
> > *From:* <a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@gluster.org</a><br>
> > [mailto:<a href="mailto:gluster-devel-bounces@" target="_blank">gluster-devel-bounces@</a>
<a href="http://gluster.org" target="_blank">gluster.org</a><br>
> > <<a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@gluster.org</a>>] *On Behalf Of *Pranith Kumar<br>
> > Karampuri<br>
> > *Sent:* Thursday, January 11, 2018 12:39 PM<br>
> > *To:* Lian, George (NSB - CN/Hangzhou) <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>><br>
> > *Cc:* Zhou, Cynthia (NSB - CN/Hangzhou)<br>
> > <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>;
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>; Li,<br>
> > Deqian (NSB - CN/Hangzhou) < <a href="mailto:deqian.li@nokia-sbell.com" target="_blank">
deqian.li@nokia-sbell.com</a>>; Sun, Ping<br>
> > (NSB - CN/Hangzhou) < <a href="mailto:ping.sun@nokia-sbell.com" target="_blank">
ping.sun@nokia-sbell.com</a>><br>
> > *Subject:* Re: [Gluster-devel] a link issue maybe introduced in a<br>
> > bug fix " Don't let NFS cache stat after writes"<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Thu, Jan 11, 2018 at 6:35 AM, Lian, George (NSB - CN/Hangzhou) <<br>
> > <a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>> wrote:<br>
> ><br>
> > Hi,<br>
> ><br>
> > >>> In which protocol are you seeing this issue? Fuse/NFS/SMB?<br>
> ><br>
> > It is fuse, within mountpoint by “mount -t glusterfs …“ command.<br>
> ><br>
> ><br>
> ><br>
> > Could you let me know the test you did so that I can try to<br>
> > re-create and see what exactly is going on?<br>
> ><br>
> > Configuration of the volume and the steps to re-create the issue you<br>
> > are seeing would be helpful in debugging the issue further.<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > Thanks & Best Regards,<br>
> ><br>
> > George<br>
> ><br>
> ><br>
> ><br>
> > *From:* <a href="mailto:gluster-devel-bounces@gluster.org" target="_blank">gluster-devel-bounces@gluster.org</a><br>
> > [mailto:<a href="mailto:gluster-devel-bounces@" target="_blank">gluster-devel-bounces@</a>
<a href="http://gluster.org" target="_blank">gluster.org</a>] *On Behalf Of *Pranith<br>
> > Kumar Karampuri<br>
> > *Sent:* Wednesday, January 10, 2018 8:08 PM<br>
> > *To:* Lian, George (NSB - CN/Hangzhou) <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>><br>
> > *Cc:* Zhou, Cynthia (NSB - CN/Hangzhou)<br>
> > <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>; Zhong, Hua (NSB - CN/Hangzhou)<br>
> > <<a href="mailto:hua.zhong@nokia-sbell.com" target="_blank">hua.zhong@nokia-sbell.com</a>>; Li, Deqian (NSB - CN/Hangzhou)<br>
> > <<a href="mailto:deqian.li@nokia-sbell.com" target="_blank">deqian.li@nokia-sbell.com</a>>;
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>; Sun, Ping<br>
> > (NSB - CN/Hangzhou) <<a href="mailto:ping.sun@nokia-sbell.com" target="_blank">ping.sun@nokia-sbell.com</a>><br>
> > *Subject:* Re: [Gluster-devel] a link issue maybe introduced in a<br>
> > bug fix " Don't let NFS cache stat after writes"<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Wed, Jan 10, 2018 at 11:09 AM, Lian, George (NSB - CN/Hangzhou) <<br>
> > <a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>> wrote:<br>
> ><br>
> > Hi, Pranith Kumar,<br>
> ><br>
> ><br>
> ><br>
> > I has create a bug on Bugzilla <a href="https://bugzilla.redhat.com/" target="_blank">
https://bugzilla.redhat.com/</a><br>
> > show_bug.cgi?id=1531457<br>
> ><br>
> > After my investigation for this link issue, I suppose your changes<br>
> > on afr-dir-write.c with issue " Don't let NFS cache stat after<br>
> > writes" , your fix is like:<br>
> ><br>
> > --------------------------------------<br>
> ><br>
> > if (afr_txn_nothing_failed (frame, this)) {<br>
> ><br>
> > /*if it did pre-op, it will do post-op<br>
> > changing ctime*/<br>
> ><br>
> > if (priv->consistent_metadata &&<br>
> ><br>
> > afr_needs_changelog_update (local))<br>
> ><br>
> > afr_zero_fill_stat (local);<br>
> ><br>
> > local->transaction.unwind (frame, this);<br>
> ><br>
> > }<br>
> ><br>
> > In the above fix, it set the ia_nlink to ‘0’ if option<br>
> > consistent-metadata is set to “on”.<br>
> ><br>
> > And hard link a file with which just created will lead to an error,<br>
> > and the error is caused in kernel function “vfs_link”:<br>
> ><br>
> > if (inode->i_nlink == 0 && !(inode->i_state & I_LINKABLE))<br>
> ><br>
> > error = -ENOENT;<br>
> ><br>
> ><br>
> ><br>
> > could you please have a check and give some comments here?<br>
> ><br>
> ><br>
> ><br>
> > When stat is "zero filled", understanding is that the higher layer<br>
> > protocol doesn't send stat value to the kernel and a separate lookup<br>
> > is sent by the kernel to get the latest stat value. In which<br>
> > protocol are you seeing this issue? Fuse/NFS/SMB?<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > Thanks & Best Regards,<br>
> ><br>
> > George<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> ><br>
> > Pranith<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> ><br>
> > Pranith<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> ><br>
> > Pranith<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> ><br>
> > Pranith<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> ><br>
> > Pranith<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> ><br>
> > Pranith<br>
> ><br>
><br>
><br>
><br>
> --<br>
> Pranith<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.gluster.org/mailman/listinfo/gluster-devel</a><o:p></o:p></span></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>