[Gluster-devel] The ctime of fstat is not correct which lead to "tar" utility error

[Raghavendra Gowdappa]

On Thu, Jul 19, 2018 at 2:29 PM, Lian, George (NSB - CN/Hangzhou) <
george.lian at nokia-sbell.com> wrote:

> Hi, Gluster Experts,
>
>
>
> In glusterfs version 3.12.3, There seems a “fstat” issue for ctime after
> we use fsync,
>
> We have a demo execute binary which write some data and then do fsync for
> this file, it named as “tt”,
>
> Then run tar command right after “tt” command, it will always error with
> “tar: /mnt/test/file1.txt: file changed as we read it”
>
>
>
> The command output is list as the below, the source code and volume info
> configuration attached FYI,
>
> This issue will be 100% reproducible! (/mnt/test is mountpoint of
> glusterfs volume “test” , which the volume info is attached in mail)
>
> ------------------------------------------------------------------
>
> ./tt;tar -czvf /tmp/abc.gz /mnt/test/file1.txt
>
> mtime:1531247107.272000000
>
> ctime:1531247107.272000000
>
> tar: Removing leading `/' from member names
>
> /mnt/test/file1.txt
>
> tar: /mnt/test/file1.txt: file changed as we read it
>
> ----------------------------------------------------------------------
>
>
>
> After my investigation, the xattrop for changelog is later than the fsync
> response , this is mean:
>
> In function  “afr_fsync_cbk” will call afr_delayed_changelog_wake_resume
> (this, local->fd, stub);
>
>
>
> In our case, it always a pending changelog , so glusterfs save the
> metadata information to stub, and handle pending changelog first,
>
> But the changelog will also change the ctime, from the packet captured by
> tcpdump, the response packet of xattrop will not include the metadata
> information,  and the wake_resume also not handle this metadata changed
> case.
>
>
>
> So in this case, the metadata in mdc_cache is not right, and when cache is
> valid, the application will get WRONG metadata!
>
>
>
> For verify my guess, if I change the configuration for this volume
>
> “gluster v set test md-cache-timeout 0” or
>
> “gluster v set export stat-prefetch off”
>
> This issue will be GONE!
>

We recently identified an issue with stat-prefetch. Fix can be found at:
https://review.gluster.org/#/c/20410/11

Can you let us know whether this helps?


>
>
>
> And I restore the configuration to default, which mean stat-prefetch is on
> and md-cache-timeout is 1 second,
>
> I try invalidate the md-cache in source code as the below in function
> mdc_fync_cbk on md-cache.c
>
> The issue also will be GONE!
>
>
>
> So GLusterFS Experts,
>
> Could you please verify this issue, and share your comments on my
> investigation?
>
> And your finally solutions is highly appreciated!
>
>
Does the following fix you've posted solves the problem?


>
> ------------------------------------changes in function “mdc_fsync_cbk”
>
> int
>
> mdc_fsync_cbk (call_frame_t *frame, void *cookie, xlator_t *this,
>
>                int32_t op_ret, int32_t op_errno,
>
>                struct iatt *prebuf, struct iatt *postbuf, dict_t *xdata)
>
> {
>
>         mdc_local_t  *local = NULL;
>
>
>
>         local = frame->local;
>
>
>
>         if (op_ret != 0)
>
>                 goto out;
>
>
>
>         if (!local)
>
>                 goto out;
>
>
>
>         mdc_inode_iatt_set_validate(this, local->fd->inode, prebuf,
> postbuf,
>
>                                      _gf_true);
>
> /* new added for ctime issue*/
>
>         mdc_inode_iatt_invalidate(this, local->fd->inode);
>
>
> /* new added end*/
>
> out:
>
>         MDC_STACK_UNWIND (fsync, frame, op_ret, op_errno, prebuf, postbuf,
>
>                           xdata);
>
>
>
>         return 0;
>
> }
>
> ------------------------------------------------------------
> ---------------------
>
> Best Regards,
>
> George
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20180719/1ad43c7a/attachment.html>