<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"\@等线";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:宋体;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin:0cm;
margin-bottom:.0001pt;
text-indent:21.0pt;
font-size:12.0pt;
font-family:宋体;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:宋体;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:DengXian;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:DengXian;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1549804370;
mso-list-type:hybrid;
mso-list-template-ids:529855824 185252486 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:18.0pt;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-text:"%2\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:42.0pt;
text-indent:-21.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:63.0pt;
text-indent:-21.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:84.0pt;
text-indent:-21.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-text:"%5\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:105.0pt;
text-indent:-21.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:126.0pt;
text-indent:-21.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:147.0pt;
text-indent:-21.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-text:"%8\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:168.0pt;
text-indent:-21.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:189.0pt;
text-indent:-21.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Sorry, there seems still have issue.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">We use
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">dd</span>”<span lang="EN-US"> application of linux tools instead of my demo program, and if the file is not exist before dd, the issue still be there.<o:p></o:p></span></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">The test command is
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">rm -rf /mnt/test/file.txt ; dd if=/dev/zero of=/mnt/test/file.txt bs=512 count=1 oflag=sync;stat /mnt/test/file.txt;tar -czvf /tmp/abc.gz<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2">
<![if !supportLists]><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><span style="mso-list:Ignore">1)<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">If we set md-cache-timeout to 0, the issue will not happen<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2">
<![if !supportLists]><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><span style="mso-list:Ignore">2)<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">If we set md-cache-timeout to 1, the issue will 100% reproduced! (with new patch you mentioned in the mail)<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Please see detail test result as the below:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">bash-4.4# gluster v set export md-cache-timeout 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">volume set: failed: Volume export does not exist<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">bash-4.4# gluster v set test md-cache-timeout 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">volume set: success<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">bash-4.4# dd if=/dev/zero of=/mnt/test/file.txt bs=512 count=1 oflag=sync;stat /mnt/test/file.txt;tar -czvf /tmp/abc.gz /mnt/test/file.txt;stat /mnt/test/file.txt^C<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">bash-4.4# rm /mnt/test/file.txt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">bash-4.4# dd if=/dev/zero of=/mnt/test/file.txt bs=512 count=1 oflag=sync;stat /mnt/test/file.txt;tar -czvf /tmp/abc.gz /mnt/test/file.txt;stat /mnt/test/file.txt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">1+0 records in<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">1+0 records out<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">512 bytes copied, 0.00932571 s, 54.9 kB/s<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> File: /mnt/test/file.txt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> Size: 512 Blocks: 1 IO Block: 131072 regular file<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Device: 33h/51d Inode: 9949244856126716752 Links: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Access: 2018-07-13 17:55:02.756000000 +0000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Modify: 2018-07-13 17:55:02.764000000 +0000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Change: 2018-07-13 17:55:02.768000000 +0000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Birth: -<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">tar: Removing leading `/' from member names<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">/mnt/test/file.txt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> File: /mnt/test/file.txt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> Size: 512 Blocks: 1 IO Block: 131072 regular file<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Device: 33h/51d Inode: 9949244856126716752 Links: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Access: 2018-07-13 17:55:02.776000000 +0000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Modify: 2018-07-13 17:55:02.764000000 +0000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Change: 2018-07-13 17:55:02.768000000 +0000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Birth: -<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">bash-4.4# gluster v set test md-cache-timeout 1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">volume set: success<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">bash-4.4# rm /mnt/test/file.txt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">bash-4.4# dd if=/dev/zero of=/mnt/test/file.txt bs=512 count=1 oflag=sync;stat /mnt/test/file.txt;tar -czvf /tmp/abc.gz /mnt/test/file.txt;stat /mnt/test/file.txt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">1+0 records in<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">1+0 records out<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">512 bytes copied, 0.0107589 s, 47.6 kB/s<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> File: /mnt/test/file.txt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> Size: 512 Blocks: 1 IO Block: 131072 regular file<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Device: 33h/51d Inode: 13569976446871695205 Links: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Access: 2018-07-13 17:55:11.548000000 +0000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Modify: 2018-07-13 17:55:11.560000000 +0000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Change: 2018-07-13 17:55:11.560000000 +0000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Birth: -<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">tar: Removing leading `/' from member names<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">/mnt/test/file.txt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">tar: /mnt/test/file.txt: file changed as we read it<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> File: /mnt/test/file.txt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> Size: 512 Blocks: 1 IO Block: 131072 regular file<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Device: 33h/51d Inode: 13569976446871695205 Links: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Access: 2018-07-13 17:55:11.580000000 +0000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Modify: 2018-07-13 17:55:11.560000000 +0000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Change: 2018-07-13 17:55:11.564000000 +0000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Birth: -<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Best Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">George<o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> gluster-devel-bounces@gluster.org [mailto:gluster-devel-bounces@gluster.org]
<b>On Behalf Of </b>Raghavendra Gowdappa<br>
<b>Sent:</b> Friday, July 20, 2018 4:01 PM<br>
<b>To:</b> Lian, George (NSB - CN/Hangzhou) <george.lian@nokia-sbell.com><br>
<b>Cc:</b> Zhang, Bingxuan (NSB - CN/Hangzhou) <bingxuan.zhang@nokia-sbell.com>; Raghavendra G <raghavendra@gluster.com>; Gluster-devel@gluster.org<br>
<b>Subject:</b> Re: [Gluster-devel] The ctime of fstat is not correct which lead to "tar" utility error<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Fri, Jul 20, 2018 at 1:22 PM, Lian, George (NSB - CN/Hangzhou) <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>> wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">>>>We recently identified an issue with stat-prefetch. Fix can be found at:<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">>>><a href="https://review.gluster.org/#/c/20410/11" target="_blank">https://review.gluster.org/#/c/20410/11</a><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">>>>Can you let us know whether this helps?<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">The patch can resolve this issue, I have verified in Gluster 4.2(master trunk branch) and Gluster 3.12.3!</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thanks we'll merge it.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Thanks & Best Regards,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">George</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">
</span><span lang="EN-US"><a href="mailto:gluster-devel-bounces@gluster.org" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">gluster-devel-bounces@gluster.org</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">
[mailto:</span><span lang="EN-US"><a href="mailto:gluster-devel-bounces@gluster.org" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">gluster-devel-bounces@gluster.org</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">]
<b>On Behalf Of </b>Raghavendra Gowdappa<br>
<b>Sent:</b> Thursday, July 19, 2018 5:06 PM<br>
<b>To:</b> Lian, George (NSB - CN/Hangzhou) <</span><span lang="EN-US"><a href="mailto:george.lian@nokia-sbell.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">george.lian@nokia-sbell.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">><br>
<b>Cc:</b> Zhang, Bingxuan (NSB - CN/Hangzhou) <</span><span lang="EN-US"><a href="mailto:bingxuan.zhang@nokia-sbell.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">bingxuan.zhang@nokia-sbell.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">>;
</span><span lang="EN-US"><a href="mailto:Gluster-devel@gluster.org" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Gluster-devel@gluster.org</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">;
Raghavendra G <</span><span lang="EN-US"><a href="mailto:raghavendra@gluster.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">raghavendra@gluster.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">><br>
<b>Subject:</b> Re: [Gluster-devel] The ctime of fstat is not correct which lead to "tar" utility error</span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">On Thu, Jul 19, 2018 at 2:29 PM, Lian, George (NSB - CN/Hangzhou) <<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>>
wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Hi, Gluster Experts,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">In glusterfs version 3.12.3, There seems a
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">fstat</span>”<span lang="EN-US"> issue for ctime after we use fsync,
</span></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">We have a demo execute binary which write some data and then do fsync for this file, it named as
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">tt</span>”<span lang="EN-US">,</span></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Then run tar command right after
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">tt</span>”<span lang="EN-US"> command, it will always error with
</span>“<span lang="EN-US">tar: /mnt/test/file1.txt: file changed as we read it</span>”</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">The command output is list as the below, the source code and volume info configuration attached FYI,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">This issue will be 100% reproducible! (/mnt/test is mountpoint of glusterfs volume
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">test</span>”<span lang="EN-US"> , which the volume info is attached in mail)</span></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">------------------------------------------------------------------</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">./tt;tar -czvf /tmp/abc.gz /mnt/test/file1.txt</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">mtime:1531247107.272000000</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">ctime:1531247107.272000000</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">tar: Removing leading `/' from member names</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">/mnt/test/file1.txt</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">tar: /mnt/test/file1.txt: file changed as we read it</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">----------------------------------------------------------------------</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">After my investigation, the xattrop for changelog is later than the fsync response , this is mean:</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">In function
</span><span lang="EN-US" style="font-size:10.5pt"> </span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">afr_fsync_cbk</span>”<span lang="EN-US"> will call afr_delayed_changelog_wake_resume (this, local->fd, stub);</span></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">In our case, it always a pending changelog , so glusterfs save the metadata information to stub, and handle pending
changelog first,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">But the changelog will also change the ctime, from the packet captured by tcpdump, the response packet of xattrop
will not include the metadata information,</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> and the wake_resume also not handle this metadata changed case.</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">So in this case, the metadata in mdc_cache is not right, and when cache is valid, the application will get WRONG
metadata!</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">For verify my guess, if I change the configuration for this volume</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-indent:36.75pt">
<span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">gluster v set test md-cache-timeout 0</span>”<span lang="EN-US"> or</span></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-indent:36.75pt">
<span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">gluster v set export stat-prefetch off</span>”</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">This issue will be GONE!</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">We recently identified an issue with stat-prefetch. Fix can be found at:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"><a href="https://review.gluster.org/#/c/20410/11" target="_blank">https://review.gluster.org/#/c/20410/11</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Can you let us know whether this helps?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">And I restore the configuration to default, which mean stat-prefetch is on and md-cache-timeout is 1 second,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">I try invalidate the md-cache in source code as the below in function mdc_fync_cbk on md-cache.c</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">The issue also will be GONE!</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">So GLusterFS Experts,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Could you please verify this issue, and share your comments on my investigation?</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">And your finally solutions is highly appreciated!</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Does the following fix you've posted solves the problem?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">------------------------------------changes in function
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">mdc_fsync_cbk</span>”</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">int</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">mdc_fsync_cbk (call_frame_t *frame, void *cookie, xlator_t *this,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> int32_t op_ret, int32_t op_errno,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> struct iatt *prebuf, struct iatt *postbuf, dict_t
*xdata)</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">{</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> mdc_local_t</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
*local = NULL;</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> local = frame->local;</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> if (op_ret != 0)</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> goto out;</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> if (!local)</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> goto out;</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> mdc_inode_iatt_set_validate(this, local->fd->inode, prebuf,
postbuf,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> _gf_true);</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">/* new added for ctime issue*/</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> mdc_inode_iatt_invalidate(this, local->fd->inode);</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">/* new added end*/</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">out:</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> MDC_STACK_UNWIND (fsync, frame, op_ret, op_errno, prebuf,
postbuf,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> xdata);</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> return 0;</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">}</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">---------------------------------------------------------------------------------</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Best Regards,</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">George</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</div>
</div>
</body>
</html>