[Gluster-users] How caches are working on AFR?

[Anand Babu Periasamy]

Stas Oskin wrote:
> Hi.
> 
> 2009/3/8 Anand Babu Periasamy <ab at gluster.com <mailto:ab at gluster.com>>
> 
>     Replicate in 2.0 performs atomic writes by default. This means,
>     writes will return control
>     back to application only after both the volumes (or more) are
>     successfully written.
> 
> 
> Ok, so without write-behind cache, only when data physically written to 
> all AFR disk, the app would continue?

Yes. Preciously speaking, when data is handed over to underlying diskfs
and not physically written to disk. It may be written or journaled.

Every parallel write operation is a transaction. It has to complete
atomically on all volumes. If a volume is down, incomplete files
are marked pending. It doesn't block then.

>     To mask the performance penalty of atomic writes, you should load
>     write-behind on top of
>     it. Write-behind returns control as soon as it receives the write
>     call from the
>     application, but it continues to write in background. Write-behind
>     also performs
>     block-aggregation. Smaller writes are aggregated into fewer large
>     writes.
> 
>     POSIX says application should verify the return status of close
>     system call to ensure all
>     writes were successfully written. If they are any pending writes,
>     close call will block to
>      ensure all the data is completely written. There is an option in
>     write-behind to even
>     close in background. It is unsafe and turned off by default.
> 
> 
> So I need to call close() per each file (which should be done 
> nevertheless for correct operations), in order to insure all was written 
> to disk?
> 
> And if the close() fails - this means some of the data was lost?
>
Yes correct. This behavior is expected even for regular disk file systems.

If you want every write to be physically written to disk, you should
either open with O_DIRECT or flush or use appropriate file system APIs
for synchronous writes. GlusterFS respects all the flags/APIs and turns off
write-behind or any such optimizations appropriately.

> 
>     Applications that expect every write to succeed, issues synchronous
>     writes.
> 
> 
> By this you mean that no write-behind should be used, only the default 
> atomic writes behavior?

No, Write-behind is good. Even NFS and regular disk file systems behave
exactly like this.  See the excerpt from GNU Glibc reference manual below.

In GlusterFS, all of the functionalities including basic performance
features are implemented as modules. You will get awful performance
with out these modules loaded. You can only expect GlusterFS to
be functionally right.

--------[ FROM GLIBC DOC ]--------------------------------
for write (..)
      Once `write' returns, the data is enqueued to be written and can be
      read back right away, but it is not necessarily written out to
      permanent storage immediately.  You can use `fsync' when you need
      to be sure your data has been permanently stored before
      continuing.  (It is more efficient for the system to batch up
      consecutive writes and do them all at once when convenient.
      Normally they will always be written to disk within a minute or
      less.)  Modern systems provide another function `fdatasync' which
      guarantees integrity only for the file data and is therefore
      faster.  You can use the `O_FSYNC' open mode to make `write' always
      store the data to disk before returning;

for close (..)
`ENOSPC'
`EIO'
`EDQUOT'
      When the file is accessed by NFS, these errors from `write'
      can sometimes not be detected until `close'.  *Note I/O
      Primitives::, for details on their meaning.
----------------------------------------------------------

-- 
Anand Babu Periasamy
GPG Key ID: 0x62E15A31
Blog [http://ab.multics.org]
GlusterFS [http://www.gluster.org]
The GNU Operating System [http://www.gnu.org]