[Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file

[Lian, George (Nokia - CN/Hangzhou)]

Hi, Dear Expert,

We have use glusterfs as a network filesystem, and syslog store in there, some clients on different host may write the syslog file via "glusterfs" mount point.

Now we encounter an issue when we "tail" the syslog file, it will occasional failed with error "file truncated"
As we study and trace with the "tail" source code, it failed with the following code:
if (S_ISREG (mode) && stats.st_size < f[i].size)
                {
                  error (0, 0, _("%s: file truncated"), quotef (name));
                  /* Assume the file was truncated to 0,
                     and therefore output all "new" data.  */
                  xlseek (fd, 0, SEEK_SET, name);
                  f[i].size = 0;
                }

When stats.st_size < f[i].size, what mean the size report by fstat is less than "tail" had read, it lead to "file truncated", we also use "strace" tools to trace the tail application, the related tail strace log as the below:

nanosleep({1, 0}, NULL)                 = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=192543105, ...}) = 0
nanosleep({1, 0}, NULL)                 = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=192543105, ...}) = 0
nanosleep({1, 0}, NULL)                 = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=192543105, ...}) = 0
nanosleep({1, 0}, NULL)                 = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=192544549, ...}) = 0
read(3, "Data ...-"..., 8192) = 1444
read(3, "Data.."..., 8192) = 720
read(3, "", 8192)                       = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=192544789, ...}) = 0
write(1, "DATA.....") = 2164
write(2, "tail: ", 6tail: )                   = 6
write(2, "/mnt/log/master/syslog: file tru"..., 38/mnt/log/master/syslog: file truncated) = 38

as the above strace log, tail has read 1444+720=2164 bytes,
but fstat tell "tail" 192544789 - 192543105 = 1664 which less than 2164, so it lead to "tail" application "file truncated".

And if we turn off "write-behind" feature, the issue will not be reproduced any more.
So we think it may be related to cache consistence issue due to performance consider, but we still have concern that:

The syslog file is used only with "Append" mode, so the size of file shouldn't be reduced, when a client read the file, why "fstat" can't return the really size match to the cache?
>From current investigation, we doubt that the current implement of "glusterfs" has a bug on "fstat" when cache is on.

Your comments is our highly appreciated!

Thanks & Best Regards

George





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20161013/afae899d/attachment.html>