[Gluster-devel] Operation reordering and data corruption

[Anand Avati]

On Thu, Sep 8, 2011 at 6:34 PM, Emmanuel Dreyfus <manu at netbsd.org> wrote:

> Hi
>
> I experience a nasty data corruption bug that I can track down to a bad
> interraction between how the NetBSD kernel resize files and an operation
> reordering ability glusterfs seems to have.
>
> Consider the exchanges below that I logged at the FUSE level:
>
> >> SETATTR size = 12178 -> 32394
> >> WRITE size = 12178
> << WRITE size = 12178 -> 32394
> >> GETATTR size = 32394
> << GETATTR size = 32394 -> 12178
> << SETATTR size = 32394 -> 32394
>
> SETATTR si sent before WRITE, but WRITE completes first. Then GETATTR
> kicks in and it seems SETATTR is not yet completed, since the old size
> is obtained.
>
> This is a real problem, because the NetBSD kernel will detect a file
> shrink in the GETATTR vnode method, and it will truncate the file to
> match the new size. The result is that the WRITE that was just done is
> discarded, and data is replaced by a chunk of zeroes.
>
> In what conditions glusterfs performs operation reordering? Is it a bug
> or a feature? Linux does not truncate in the same situation?



GlusterFS does guarantee order of completing operation between two system
calls if the second system call starts _after_ the return of the first
system call. When two system calls are concurrently in progress there is no
expected order of completion. There is no such order of completion in NFS
either. Linux's expectation of ordering of system calls is this as well.
>From what you describe, it looks like NetBSD would have to implement
synchronization in its VFS just the way Linux does (and to live within the
expectations set by NFS and FUSE like filesystems).

Avati
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20110908/cfb06583/attachment-0003.html>