<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 19, 2018 at 2:29 PM, Lian, George (NSB - CN/Hangzhou) <span dir="ltr"><<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="ZH-CN">
<div class="gmail-m_-4765632703722540750WordSection1">
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">Hi, Gluster Experts,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">In glusterfs version 3.12.3, There seems a “fstat” issue for ctime after we use fsync,
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">We have a demo execute binary which write some data and then do fsync for this file, it named as “tt”,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">Then run tar command right after “tt” command, it will always error with “tar: /mnt/test/file1.txt: file changed as we read it”<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">The command output is list as the below, the source code and volume info configuration attached FYI,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">This issue will be 100% reproducible! (/mnt/test is mountpoint of glusterfs volume “test” , which the volume info is attached in mail)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">------------------------------<wbr>------------------------------<wbr>------<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">./tt;tar -czvf /tmp/abc.gz /mnt/test/file1.txt<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">mtime:1531247107.272000000<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">ctime:1531247107.272000000<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">tar: Removing leading `/' from member names<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">/mnt/test/file1.txt<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">tar: /mnt/test/file1.txt: file changed as we read it<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">------------------------------<wbr>------------------------------<wbr>----------<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">After my investigation, the xattrop for changelog is later than the fsync response , this is mean:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">In function “afr_fsync_cbk” will call afr_delayed_changelog_wake_<wbr>resume (this, local->fd, stub);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">In our case, it always a pending changelog , so glusterfs save the metadata information to stub, and handle pending changelog first,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">But the changelog will also change the ctime, from the packet captured by tcpdump, the response packet of xattrop will not include the metadata information, and the wake_resume
also not handle this metadata changed case.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">So in this case, the metadata in mdc_cache is not right, and when cache is valid, the application will get WRONG metadata!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">For verify my guess, if I change the configuration for this volume<u></u><u></u></span></p>
<p class="MsoNormal" style="text-indent:36.75pt"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">“gluster v set test md-cache-timeout 0” or<u></u><u></u></span></p>
<p class="MsoNormal" style="text-indent:36.75pt"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">“gluster v set export stat-prefetch off”<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">This issue will be GONE!</span></p></div></div></blockquote><div><br></div><div>We recently identified an issue with stat-prefetch. Fix can be found at:</div><div> <a href="https://review.gluster.org/#/c/20410/11">https://review.gluster.org/#/c/20410/11</a></div><div><br></div><div>Can you let us know whether this helps?</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="ZH-CN"><div class="gmail-m_-4765632703722540750WordSection1"><p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">And I restore the configuration to default, which mean stat-prefetch is on and md-cache-timeout is 1 second,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">I try invalidate the md-cache in source code as the below in function mdc_fync_cbk on md-cache.c<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">The issue also will be GONE!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">So GLusterFS Experts,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">Could you please verify this issue, and share your comments on my investigation?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">And your finally solutions is highly appreciated!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u></span></p></div></div></blockquote><div><br></div><div>Does the following fix you've posted solves the problem?</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="ZH-CN"><div class="gmail-m_-4765632703722540750WordSection1"><p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">------------------------------<wbr>------changes in function “mdc_fsync_cbk”<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">int<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">mdc_fsync_cbk (call_frame_t *frame, void *cookie, xlator_t *this,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> int32_t op_ret, int32_t op_errno,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> struct iatt *prebuf, struct iatt *postbuf, dict_t *xdata)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">{<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> mdc_local_t *local = NULL;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> local = frame->local;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> if (op_ret != 0)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> goto out;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> if (!local)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> goto out;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> mdc_inode_iatt_set_validate(<wbr>this, local->fd->inode, prebuf, postbuf,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> <wbr> _gf_true);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">/* new added for ctime issue*/<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> mdc_inode_iatt_invalidate(<wbr>this, local->fd->inode); <wbr>
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">/* new added end*/<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">out:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> MDC_STACK_UNWIND (fsync, frame, op_ret, op_errno, prebuf, postbuf,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> xdata);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> return 0;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">}<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">------------------------------<wbr>------------------------------<wbr>---------------------<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">Best Regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">George<u></u><u></u></span></p>
</div>
</div>
</blockquote></div><br></div></div>