[Bugs] [Bug 1314508] New: tar complains: <fileName>: file changed as we read it
bugzilla at redhat.com
bugzilla at redhat.com
Thu Mar 3 19:26:48 UTC 2016
https://bugzilla.redhat.com/show_bug.cgi?id=1314508
Bug ID: 1314508
Summary: tar complains: <fileName>: file changed as we read it
Product: Red Hat Gluster Storage
Version: 3.1
Component: gluster-dht
Keywords: ZStream
Severity: high
Priority: medium
Assignee: rhs-bugs at redhat.com
Reporter: dblack at redhat.com
QA Contact: storage-qa-internal at redhat.com
CC: bkunal at redhat.com, bugs at gluster.org,
nbalacha at redhat.com, olim at redhat.com,
pkarampu at redhat.com, rgowdapp at redhat.com,
sabansal at redhat.com, smohan at redhat.com,
spalai at redhat.com, storage-qa-internal at redhat.com
Depends On: 1302948, 1312721
Keywords: ZStream
+++ This bug was initially created as a clone of Bug #1312721 +++
+++ This bug was initially created as a clone of Bug #1302948 +++
+++ This bug was initially created as a clone of Bug #1298724 +++
Description of problem:
Encounter 'file changed as we read it' problem, with quota disabled:
# pwd
# tar czvf linux-4.4.tar.gz.test linux-4.4
linux-4.4/
linux-4.4/Kbuild
tar: linux-4.4/Kbuild: file changed as we read it
linux-4.4/block/
linux-4.4/block/blk-lib.c
tar: linux-4.4/block/blk-lib.c: file changed as we read it
linux-4.4/block/blk-settings.c
tar: linux-4.4/block/blk-settings.c: file changed as we read it
linux-4.4/block/scsi_ioctl.c
linux-4.4/block/blk-mq.h
linux-4.4/block/noop-iosched.c
linux-4.4/block/Kconfig.iosched
tar: linux-4.4/block/Kconfig.iosched: file changed as we read it
linux-4.4/block/blk-throttle.c
linux-4.4/block/ioprio.c
linux-4.4/block/t10-pi.c
linux-4.4/block/Kconfig
linux-4.4/block/bio.c
linux-4.4/block/blk-exec.c
linux-4.4/block/blk-ioc.c
linux-4.4/block/blk-iopoll.c
linux-4.4/block/blk-map.c
linux-4.4/block/blk-timeout.c
linux-4.4/block/compat_ioctl.c
linux-4.4/block/deadline-iosched.c
linux-4.4/block/elevator.c
linux-4.4/block/partition-generic.c
linux-4.4/block/Makefile
linux-4.4/block/blk-cgroup.c
linux-4.4/block/blk-core.c
tar: linux-4.4/block/blk-core.c: file changed as we read it
linux-4.4/block/blk-mq-cpu.c
tar: linux-4.4/block/blk-mq-cpu.c: file changed as we read it
linux-4.4/block/bsg.c
tar: linux-4.4/block/bsg.c: file changed as we read it
linux-4.4/block/cfq-iosched.c
tar: linux-4.4/block/cfq-iosched.c: file changed as we read it
linux-4.4/block/cmdline-parser.c
tar: linux-4.4/block/cmdline-parser.c: file changed as we read it
linux-4.4/block/genhd.c
linux-4.4/block/bio-integrity.c
linux-4.4/block/blk-merge.c
linux-4.4/block/blk-softirq.c
linux-4.4/block/blk-tag.c
linux-4.4/block/bounce.c
linux-4.4/block/blk-mq-cpumap.c
tar: linux-4.4/block/blk-mq-cpumap.c: file changed as we read it
linux-4.4/block/blk-mq-sysfs.c
tar: linux-4.4/block/blk-mq-sysfs.c: file changed as we read it
linux-4.4/block/blk-mq-tag.h
tar: linux-4.4/block/blk-mq-tag.h: file changed as we read it
linux-4.4/block/blk-mq.c
linux-4.4/block/blk-sysfs.c
linux-4.4/block/blk.h
linux-4.4/block/partitions/
linux-4.4/block/partitions/amiga.c
linux-4.4/block/partitions/efi.h
linux-4.4/block/partitions/karma.h
linux-4.4/block/partitions/ldm.c
tar: linux-4.4/block/partitions/ldm.c: file changed as we read it
linux-4.4/block/partitions/msdos.c
tar: linux-4.4/block/partitions/msdos.c: file changed as we read it
linux-4.4/block/partitions/osf.h
linux-4.4/block/partitions/atari.c
linux-4.4/block/partitions/cmdline.c
linux-4.4/block/partitions/mac.h
linux-4.4/block/partitions/sgi.h
linux-4.4/block/partitions/sun.h
linux-4.4/block/partitions/Kconfig
linux-4.4/block/partitions/ldm.h
tar: linux-4.4/block/partitions/ldm.h: file changed as we read it
linux-4.4/block/partitions/msdos.h
linux-4.4/block/partitions/sysv68.c
linux-4.4/block/partitions/sysv68.h
linux-4.4/block/partitions/Makefile
linux-4.4/block/partitions/acorn.h
linux-4.4/block/partitions/sgi.c
linux-4.4/block/partitions/atari.h
linux-4.4/block/partitions/efi.c
tar: linux-4.4/block/partitions/efi.c: file changed as we read it
linux-4.4/block/partitions/ibm.c
linux-4.4/block/partitions/aix.h
linux-4.4/block/partitions/check.h
linux-4.4/block/partitions/karma.c
linux-4.4/block/partitions/mac.c
linux-4.4/block/partitions/osf.c
linux-4.4/block/partitions/ultrix.h
linux-4.4/block/partitions/acorn.c
linux-4.4/block/partitions/amiga.h
linux-4.4/block/partitions/check.c
linux-4.4/block/partitions/sun.c
tar: linux-4.4/block/partitions/sun.c: file changed as we read it
linux-4.4/block/partitions/aix.c
linux-4.4/block/partitions/cmdline.h
linux-4.4/block/partitions/ibm.h
linux-4.4/block/partitions/ultrix.c
tar: linux-4.4/block/partitions: file changed as we read it
linux-4.4/block/blk-flush.c
linux-4.4/block/blk-integrity.c
linux-4.4/block/blk-mq-tag.c
linux-4.4/block/bsg-lib.c
linux-4.4/block/ioctl.c
tar: linux-4.4/block: file changed as we read it
linux-4.4/certs/
linux-4.4/certs/system_keyring.c
linux-4.4/certs/system_certificates.S
linux-4.4/certs/.gitignore
linux-4.4/certs/Kconfig
tar: linux-4.4/certs/Kconfig: file changed as we read it
linux-4.4/certs/Makefile
tar: linux-4.4/certs: file changed as we read it
linux-4.4/fs/
linux-4.4/fs/affs/
linux-4.4/fs/affs/inode.c
linux-4.4/fs/affs/symlink.c
tar: linux-4.4/fs/affs/symlink.c: file changed as we read it
linux-4.4/fs/affs/affs.h
tar: linux-4.4/fs/affs/affs.h: file changed as we read it
linux-4.4/fs/affs/amigaffs.c
linux-4.4/fs/affs/dir.c
tar: linux-4.4/fs/affs/dir.c: file changed as we read it
linux-4.4/fs/affs/super.c
linux-4.4/fs/affs/Changes
linux-4.4/fs/affs/Kconfig
linux-4.4/fs/affs/file.c
linux-4.4/fs/affs/Makefile
tar: linux-4.4/fs/affs/Makefile: file changed as we read it
linux-4.4/fs/affs/bitmap.c
linux-4.4/fs/affs/namei.c
tar: linux-4.4/fs/affs: file changed as we read it
linux-4.4/fs/coda/
linux-4.4/fs/coda/dir.c
linux-4.4/fs/coda/upcall.c
linux-4.4/fs/coda/Kconfig
linux-4.4/fs/coda/file.c
linux-4.4/fs/coda/Makefile
linux-4.4/fs/coda/coda_linux.h
linux-4.4/fs/coda/pioctl.c
linux-4.4/fs/coda/coda_cache.h
tar: linux-4.4/fs/coda/coda_cache.h: file changed as we read it
linux-4.4/fs/coda/coda_fs_i.h
linux-4.4/fs/coda/inode.c
linux-4.4/fs/coda/psdev.c
linux-4.4/fs/coda/symlink.c
linux-4.4/fs/coda/sysctl.c
linux-4.4/fs/coda/cache.c
linux-4.4/fs/coda/coda_int.h
linux-4.4/fs/coda/coda_linux.c
linux-4.4/fs/coda/cnode.c
tar: linux-4.4/fs/coda: file changed as we read it
linux-4.4/fs/dcookies.c
tar: linux-4.4/fs/dcookies.c: file changed as we read it
linux-4.4/fs/exec.c
tar: linux-4.4/fs/exec.c: file changed as we read it
linux-4.4/fs/fs-writeback.c
linux-4.4/fs/hfs/
linux-4.4/fs/hfs/Makefile
linux-4.4/fs/hfs/string.c
tar: linux-4.4/fs/hfs/string.c: file changed as we read it
linux-4.4/fs/hfs/bitmap.c
linux-4.4/fs/hfs/bnode.c
linux-4.4/fs/hfs/btree.h
linux-4.4/fs/hfs/hfs.h
linux-4.4/fs/hfs/hfs_fs.h
linux-4.4/fs/hfs/attr.c
tar: linux-4.4/fs/hfs/attr.c: file changed as we read it
linux-4.4/fs/hfs/btree.c
linux-4.4/fs/hfs/inode.c
linux-4.4/fs/hfs/mdb.c
linux-4.4/fs/hfs/trans.c
tar: linux-4.4/fs/hfs/trans.c: file changed as we read it
linux-4.4/fs/hfs/part_tbl.c
linux-4.4/fs/hfs/sysdep.c
linux-4.4/fs/hfs/brec.c
tar: linux-4.4/fs/hfs/brec.c: file changed as we read it
linux-4.4/fs/hfs/catalog.c
linux-4.4/fs/hfs/dir.c
linux-4.4/fs/hfs/super.c
tar: linux-4.4/fs/hfs/super.c: file changed as we read it
linux-4.4/fs/hfs/Kconfig
linux-4.4/fs/hfs/bfind.c
linux-4.4/fs/hfs/extent.c
tar: linux-4.4/fs/hfs: file changed as we read it
linux-4.4/fs/ioctl.c
linux-4.4/fs/jffs2/
linux-4.4/fs/jffs2/README.Locking
linux-4.4/fs/jffs2/acl.h
tar: linux-4.4/fs/jffs2/acl.h: file changed as we read it
linux-4.4/fs/jffs2/compr.h
linux-4.4/fs/jffs2/compr_rubin.c
tar: linux-4.4/fs/jffs2/compr_rubin.c: file changed as we read it
linux-4.4/fs/jffs2/gc.c
linux-4.4/fs/jffs2/jffs2_fs_sb.h
tar: linux-4.4/fs/jffs2/jffs2_fs_sb.h: file changed as we read it
linux-4.4/fs/jffs2/nodelist.c
linux-4.4/fs/jffs2/acl.c
linux-4.4/fs/jffs2/debug.h
linux-4.4/fs/jffs2/os-linux.h
linux-4.4/fs/jffs2/readinode.c
linux-4.4/fs/jffs2/scan.c
linux-4.4/fs/jffs2/xattr_trusted.c
linux-4.4/fs/jffs2/LICENCE
tar: linux-4.4/fs/jffs2/LICENCE: file changed as we read it
linux-4.4/fs/jffs2/build.c
tar: linux-4.4/fs/jffs2/build.c: file changed as we read it
linux-4.4/fs/jffs2/dir.c
tar: linux-4.4/fs/jffs2/dir.c: file changed as we read it
linux-4.4/fs/jffs2/super.c
linux-4.4/fs/jffs2/xattr.c
linux-4.4/fs/jffs2/Kconfig
linux-4.4/fs/jffs2/TODO
linux-4.4/fs/jffs2/file.c
linux-4.4/fs/jffs2/fs.c
linux-4.4/fs/jffs2/nodemgmt.c
linux-4.4/fs/jffs2/Makefile
linux-4.4/fs/jffs2/background.c
linux-4.4/fs/jffs2/compr.c
linux-4.4/fs/jffs2/compr_lzo.c
linux-4.4/fs/jffs2/jffs2_fs_i.h
linux-4.4/fs/jffs2/xattr_user.c
tar: linux-4.4/fs/jffs2/xattr_user.c: file changed as we read it
linux-4.4/fs/jffs2/summary.h
linux-4.4/fs/jffs2/wbuf.c
linux-4.4/fs/jffs2/erase.c
linux-4.4/fs/jffs2/malloc.c
tar: linux-4.4/fs/jffs2/malloc.c: file changed as we read it
linux-4.4/fs/jffs2/nodelist.h
tar: linux-4.4/fs/jffs2/nodelist.h: file changed as we read it
linux-4.4/fs/jffs2/read.c
tar: linux-4.4/fs/jffs2/read.c: file changed as we read it
linux-4.4/fs/jffs2/summary.c
tar: linux-4.4/fs/jffs2/summary.c: file changed as we read it
linux-4.4/fs/jffs2/symlink.c
tar: linux-4.4/fs/jffs2/symlink.c: file changed as we read it
linux-4.4/fs/jffs2/writev.c
linux-4.4/fs/jffs2/compr_rtime.c
linux-4.4/fs/jffs2/compr_zlib.c
linux-4.4/fs/jffs2/debug.c
linux-4.4/fs/jffs2/ioctl.c
linux-4.4/fs/jffs2/security.c
linux-4.4/fs/jffs2/write.c
linux-4.4/fs/jffs2/xattr.h
tar: linux-4.4/fs/jffs2: file changed as we read it
linux-4.4/fs/ncpfs/
linux-4.4/fs/ncpfs/dir.c
linux-4.4/fs/ncpfs/ncplib_kernel.h
tar: linux-4.4/fs/ncpfs/ncplib_kernel.h: file changed as we read it
linux-4.4/fs/ncpfs/ncpsign_kernel.c
linux-4.4/fs/ncpfs/Kconfig
linux-4.4/fs/ncpfs/file.c
linux-4.4/fs/ncpfs/getopt.h
linux-4.4/fs/ncpfs/ncplib_kernel.c
linux-4.4/fs/ncpfs/Makefile
tar: linux-4.4/fs/ncpfs/Makefile: file changed as we read it
linux-4.4/fs/ncpfs/ncp_fs_sb.h
linux-4.4/fs/ncpfs/sock.c
linux-4.4/fs/ncpfs/inode.c
tar: linux-4.4/fs/ncpfs/inode.c: file changed as we read it
linux-4.4/fs/ncpfs/ncp_fs_i.h
tar: linux-4.4/fs/ncpfs/ncp_fs_i.h: file changed as we read it
linux-4.4/fs/ncpfs/symlink.c
linux-4.4/fs/ncpfs/ioctl.c
linux-4.4/fs/ncpfs/getopt.c
linux-4.4/fs/ncpfs/ncp_fs.h
tar: linux-4.4/fs/ncpfs/ncp_fs.h: file changed as we read it
linux-4.4/fs/ncpfs/mmap.c
linux-4.4/fs/ncpfs/ncpsign_kernel.h
tar: linux-4.4/fs/ncpfs: file changed as we read it
linux-4.4/fs/notify/
linux-4.4/fs/notify/Makefile
linux-4.4/fs/notify/notification.c
linux-4.4/fs/notify/dnotify/
linux-4.4/fs/notify/dnotify/Makefile
tar: linux-4.4/fs/notify/dnotify/Makefile: file changed as we read it
linux-4.4/fs/notify/dnotify/Kconfig
linux-4.4/fs/notify/dnotify/dnotify.c
tar: linux-4.4/fs/notify/dnotify: file changed as we read it
linux-4.4/fs/notify/fanotify/
linux-4.4/fs/notify/fanotify/Kconfig
tar: linux-4.4/fs/notify/fanotify/Kconfig: file changed as we read it
linux-4.4/fs/notify/fanotify/Makefile
linux-4.4/fs/notify/fanotify/fanotify_user.c
linux-4.4/fs/notify/fanotify/fanotify.c
tar: linux-4.4/fs/notify/fanotify/fanotify.c: file changed as we read it
linux-4.4/fs/notify/fanotify/fanotify.h
tar: linux-4.4/fs/notify/fanotify: file changed as we read it
linux-4.4/fs/notify/fdinfo.c
linux-4.4/fs/notify/fdinfo.h
linux-4.4/fs/notify/group.c
linux-4.4/fs/notify/fsnotify.c
linux-4.4/fs/notify/fsnotify.h
linux-4.4/fs/notify/mark.c
linux-4.4/fs/notify/inode_mark.c
linux-4.4/fs/notify/inotify/
linux-4.4/fs/notify/inotify/Kconfig
linux-4.4/fs/notify/inotify/Makefile
linux-4.4/fs/notify/inotify/inotify_user.c
linux-4.4/fs/notify/inotify/inotify.h
linux-4.4/fs/notify/inotify/inotify_fsnotify.c
tar: linux-4.4/fs/notify/inotify: file changed as we read it
linux-4.4/fs/notify/vfsmount_mark.c
tar: linux-4.4/fs/notify/vfsmount_mark.c: file changed as we read it
Version-Release number of selected component (if applicable):
glusterfs-3.7.1-11.el7rhgs.x86_64
How reproducible:
Always
Steps to Reproduce:
1.Run 'tar czvf linux-4.4.tar.gz.test linux-4.4'
2.
3.
Actual results:
tar: linux-4.4/fs/notify/inotify: file changed as we read it
Expected results:
tar should not complain
Additional info:
Same symptoms as the below bug which was fixed in glusterfs-3.7.1-6
Bug 1223757 - tar on a glusterfs mount displays "file changed as we read it"
even though the file was not changed
--- Additional comment from Pranith Kumar K on 2016-01-18 20:13:40 EST ---
Anuradha is looking at this bug.
--- Additional comment from Anuradha on 2016-01-20 20:15:57 EST ---
Hi,
I was able to recreate the issue yesterday.
On debugging (with Pranith and Krutika), cause for "file changed as we read it"
for directories was found. Further debugging needs to be done to understand why
the message is displayed for files. I am going to apply a fix in my setup for
the cause found yesterday and see the results.
Will keep you posted.
Thanks,
Anuradha.
--- Additional comment from Anuradha on 2016-01-21 09:20:01 EST ---
Hi,
After testing with the fix, as mentioned in
https://bugzilla.redhat.com/show_bug.cgi?id=1298724#c5, I have not seen the
issue on FUSE mount yet. Will test more.
--- Additional comment from Anuradha on 2016-01-22 11:33:39 EST ---
Hi,
As mentioned in the previous comment, I wasn't able to recreate the issue
on a FUSE mount with the fix applied. But, the issue appeared on NFS mount.
On debugging, the following RCA was found (thanks to Krutika Dhananjay for
finding the root cause):
When a directory is created in a distribute or a distribute-replicate volume,
after creating the directory, dht sets its layout using setxattr call.
So the ctime is updated 2 times for the directory (say, t1 is the first ctime
after mkdir, and t2 is the latest one after setxattr). The ctime to be returned
to the application should be t2. But, dht doesn't update ctime from its
perspective, causing it to return t1 to the upper layers, thus to the
application.
In case of NFS mount in our test setup, it appears that NFS caches this
metadata and provides it to tar during tar's first query. After the archiving
is done, tar makes another query for comparison. At this point, glusterfs is
queried during which dht returns the latest value that it finds (that is t2).
This leads to mismatch in ctime seen by tar application.
The mismatch in these ctimes causes file changed as we read it warning.
Moving the component to dht as the root cause of the issue lies in dht.
Thanks,
Anuradha.
--- Additional comment from Vijay Bellur on 2016-02-04 05:55:50 EST ---
REVIEW: http://review.gluster.org/13352 (dht: mkdir must unwind with latest
ctime) posted (#1) for review on master by Sakshi Bansal
--- Additional comment from Vijay Bellur on 2016-02-24 02:53:55 EST ---
REVIEW: http://review.gluster.org/13352 (dht: mkdir must unwind with latest
ctime) posted (#2) for review on master by Sakshi Bansal
--- Additional comment from Vijay Bellur on 2016-02-26 07:00:32 EST ---
COMMIT: http://review.gluster.org/13352 committed in master by Raghavendra G
(rgowdapp at redhat.com)
------
commit dbf0ca53a5704c7f94257fc4223b41865458301a
Author: Sakshi Bansal <sabansal at redhat.com>
Date: Thu Feb 4 16:02:59 2016 +0530
dht: mkdir must unwind with latest ctime
Currently fops like mkdir used the the ctime it gets
after creating the directory entry. But setting layout
also updates the ctime of a directory. Hence DHT must
get the ctime after the setxattr call and unwind with
the latest ctime to avoid mismatch in time seen by
applications like tar.
Change-Id: Iecbbe3aac5244af5da9788b48ccf299ca56b4bae
BUG: 1302948
Signed-off-by: Sakshi Bansal <sabansal at redhat.com>
Reviewed-on: http://review.gluster.org/13352
Smoke: Gluster Build System <jenkins at build.gluster.com>
CentOS-regression: Gluster Build System <jenkins at build.gluster.com>
NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
Reviewed-by: Raghavendra G <rgowdapp at redhat.com>
--- Additional comment from Vijay Bellur on 2016-02-28 22:54:51 EST ---
REVIEW: http://review.gluster.org/13536 (dht: mkdir must unwind with latest
ctime) posted (#1) for review on release-3.7 by Sakshi Bansal
--- Additional comment from Vijay Bellur on 2016-02-29 23:54:20 EST ---
COMMIT: http://review.gluster.org/13536 committed in release-3.7 by Raghavendra
G (rgowdapp at redhat.com)
------
commit 080af56a02aeec20899ecbc7202a8d293378c1f9
Author: Sakshi Bansal <sabansal at redhat.com>
Date: Thu Feb 4 16:02:59 2016 +0530
dht: mkdir must unwind with latest ctime
Currently fops like mkdir used the the ctime it gets
after creating the directory entry. But setting layout
also updates the ctime of a directory. Hence DHT must
get the ctime after the setxattr call and unwind with
the latest ctime to avoid mismatch in time seen by
applications like tar.
Backport of http://review.gluster.org/#/c/13352/
> Change-Id: Iecbbe3aac5244af5da9788b48ccf299ca56b4bae
> BUG: 1302948
> Signed-off-by: Sakshi Bansal <sabansal at redhat.com>
> Reviewed-on: http://review.gluster.org/13352
> Smoke: Gluster Build System <jenkins at build.gluster.com>
> CentOS-regression: Gluster Build System <jenkins at build.gluster.com>
> NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
> Reviewed-by: Raghavendra G <rgowdapp at redhat.com>
BUG: 1312721
Change-Id: Ie09342d3705b40ce98b2935f05ad4402f74ba069
Signed-off-by: Sakshi Bansal <sabansal at redhat.com>
Reviewed-on: http://review.gluster.org/13536
Smoke: Gluster Build System <jenkins at build.gluster.com>
NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
CentOS-regression: Gluster Build System <jenkins at build.gluster.com>
Reviewed-by: Raghavendra G <rgowdapp at redhat.com>
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1302948
[Bug 1302948] tar complains: <fileName>: file changed as we read it
https://bugzilla.redhat.com/show_bug.cgi?id=1312721
[Bug 1312721] tar complains: <fileName>: file changed as we read it
--
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=8CMLCD2imx&a=cc_unsubscribe
More information about the Bugs
mailing list