[Gluster-devel] could you help to check about a glusterfs issue seems to be related to ctime

Ravishankar N ravishankar at redhat.com
Tue Mar 17 09:07:37 UTC 2020


On 17/03/20 12:56 pm, Zhou, Cynthia (NSB - CN/Hangzhou) wrote:
>
> Hi,
>
> I know that this may not be your presumption of this ctime feature, 
> however, our delima is that
>
>  1. product need to support date change scenario.
>  2. if we disable ctime feature, we will meet tar issue.
>  3. if we set consistent-metadata to on to work around this tar issue,
>     there is another even worse problem:
>     https://bugzilla.redhat.com/show_bug.cgi?id=1773476, app just
>     cranshed!
>
> So I have to make this patch.
>
For what it is worth, the 'file changed as we read it' warning from tar 
can be suppressed with the --warning=no-file-changed flag.

For 3, do you see any visible usability/performance issues with the 
work-around mentioned in the bug (comment 5 and 6)? Just curious.

Regards,
Ravi
>
> Maybe I will keep this as internal use just for our local use scenarios.
>
> Cynthia.
>
> *From:* Kotresh Hiremath Ravishankar <khiremat at redhat.com>
> *Sent:* 2020年3月17日 15:17
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* Amar Tumballi <amar at kadalu.io>; Gluster Devel 
> <gluster-devel at gluster.org>
> *Subject:* Re: [Gluster-devel] could you help to check about a 
> glusterfs issue seems to be related to ctime
>
> The whole ctime features relies on time provided by clients which are 
> time synchronized. This patch brings in the time from server to 
> compare against the time sent from client.
> As Amar mentioned, this doesn't fit well into the scheme of how ctime 
> is designed. Definitely keeping it optional and disabling it by 
> default is one way. But is that what your intention
> here?
>
> On Tue, Mar 17, 2020 at 10:56 AM Zhou, Cynthia (NSB - CN/Hangzhou) 
> <cynthia.zhou at nokia-sbell.com <mailto:cynthia.zhou at nokia-sbell.com>> 
> wrote:
>
>     Ok, thanks for your feedback!
>
>     I will do local test to verify this patch first.
>
>     cynthia
>
>     *From:* Amar Tumballi <amar at kadalu.io <mailto:amar at kadalu.io>>
>     *Sent:* 2020年3月17日 13:18
>     *To:* Zhou, Cynthia (NSB - CN/Hangzhou)
>     <cynthia.zhou at nokia-sbell.com <mailto:cynthia.zhou at nokia-sbell.com>>
>     *Cc:* Kotresh Hiremath Ravishankar <khiremat at redhat.com
>     <mailto:khiremat at redhat.com>>; Gluster Devel
>     <gluster-devel at gluster.org <mailto:gluster-devel at gluster.org>>
>     *Subject:* Re: [Gluster-devel] could you help to check about a
>     glusterfs issue seems to be related to ctime
>
>     On Tue, Mar 17, 2020 at 10:18 AM Zhou, Cynthia (NSB - CN/Hangzhou)
>     <cynthia.zhou at nokia-sbell.com
>     <mailto:cynthia.zhou at nokia-sbell.com>> wrote:
>
>         Hi glusterfs expert,
>
>         Our product need to tolerate change date to future and then
>         change back.
>
>         How about change like this ?
>
>         https://review.gluster.org/#/c/glusterfs/+/24229/1/xlators/storage/posix/src/posix-metadata.c
>
>         when time change to future and change back , should still be
>         able to update mdata, so the following changes to file can be
>         populated to other clients.
>
>     We do like to have people integrating with GlusterFS. But this
>     change is not inline with the 'assumptions' we had about the feature.
>
>     If you have verified this change works for you, please add it as
>     an 'option' in posix, which can be changed through volume set, and
>     keep this option disable/off by default. That should be an easier
>     way to get the patch reviewed and take it further. Please make
>     sure to provide the 'description' for the option with details.
>
>     Regards,
>
>     Amar
>
>         cynthia
>
>         *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
>         *Sent:* 2020年3月12日 17:31
>         *To:* 'Kotresh Hiremath Ravishankar' <khiremat at redhat.com
>         <mailto:khiremat at redhat.com>>
>         *Cc:* 'Gluster Devel' <gluster-devel at gluster.org
>         <mailto:gluster-devel at gluster.org>>
>         *Subject:* RE: could you help to check about a glusterfs issue
>         seems to be related to ctime
>
>         Hi,
>
>         One more question, I find each client has the same future time
>         stamp where are those time stamps from, since Since it is
>         different from any brick stored time stamp. And after I modify
>         files  from clients, it remains the same.
>
>         [root at mn-0:/home/robot]
>
>         # stat /mnt/export/testfile
>
>         File: /mnt/export/testfile
>
>         Size: 193             Blocks: 1 IO Block: 131072 regular file
>
>         Device: 28h/40d Inode: 10383279039841136109 Links: 1
>
>         Access: (0644/-rw-r--r--)  Uid: (    0/    root) Gid: ( 
>         615/_nokfsuifileshare)
>
>         Access: 2020-04-11 12:20:22.114365172 +0300
>
>         Modify: 2020-04-11 12:20:22.121552573 +0300
>
>         Change: 2020-04-11 12:20:22.121552573 +0300
>
>         [root at mn-0:/home/robot]
>
>         # date
>
>         Thu Mar 12 11:27:33 EET 2020
>
>         [root at mn-0:/home/robot]
>
>         [root at mn-0:/home/robot]
>
>         # stat /mnt/bricks/export/brick/testfile
>
>         File: /mnt/bricks/export/brick/testfile
>
>         Size: 193             Blocks: 16 IO Block: 4096   regular file
>
>         Device: fc02h/64514d    Inode: 512015      Links: 2
>
>         Access: (0644/-rw-r--r--)  Uid: (    0/    root) Gid: ( 
>         615/_nokfsuifileshare)
>
>         Access: 2020-04-11 12:20:22.100395536 +0300
>
>         Modify: 2020-03-12 11:25:04.095981276 +0200
>
>         Change: 2020-03-12 11:25:04.095981276 +0200
>
>         Birth: 2020-04-11 08:53:26.805163816 +0300
>
>         [root at mn-1:/root]
>
>         # stat /mnt/bricks/export/brick/testfile
>
>         File: /mnt/bricks/export/brick/testfile
>
>         Size: 193             Blocks: 16 IO Block: 4096   regular file
>
>         Device: fc02h/64514d    Inode: 512015      Links: 2
>
>         Access: (0644/-rw-r--r--)  Uid: (    0/    root) Gid: ( 
>         615/_nokfsuifileshare)
>
>         Access: 2020-04-11 12:20:22.100395536 +0300
>
>         Modify: 2020-03-12 11:25:04.094913452 +0200
>
>         Change: 2020-03-12 11:25:04.095913453 +0200
>
>         Birth: 2020-03-12 07:53:26.803783053 +0200
>
>         *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
>         *Sent:* 2020年3月12日 16:09
>         *To:* 'Kotresh Hiremath Ravishankar' <khiremat at redhat.com
>         <mailto:khiremat at redhat.com>>
>         *Cc:* Gluster Devel <gluster-devel at gluster.org
>         <mailto:gluster-devel at gluster.org>>
>         *Subject:* RE: could you help to check about a glusterfs issue
>         seems to be related to ctime
>
>         Hi,
>
>         This is abnormal test case, however, when this happened it
>         will have big impact on the apps using those files. And this
>         can not be restored automatically unless disable some xlator,
>         I think it is unacceptable for the user apps.
>
>         cynthia
>
>         *From:* Kotresh Hiremath Ravishankar <khiremat at redhat.com
>         <mailto:khiremat at redhat.com>>
>         *Sent:* 2020年3月12日 14:37
>         *To:* Zhou, Cynthia (NSB - CN/Hangzhou)
>         <cynthia.zhou at nokia-sbell.com
>         <mailto:cynthia.zhou at nokia-sbell.com>>
>         *Cc:* Gluster Devel <gluster-devel at gluster.org
>         <mailto:gluster-devel at gluster.org>>
>         *Subject:* Re: could you help to check about a glusterfs issue
>         seems to be related to ctime
>
>         All the perf xlators depend on time (mostly mtime I guess). In
>         my setup, only quick read was enabled and hence disabling it
>         worked for me.
>         All perf xlators needs to be disabled to make it work
>         correctly. But I still failed to understand how normal this
>         kind of workload ?
>
>         Thanks,
>         Kotresh
>
>         On Thu, Mar 12, 2020 at 11:20 AM Zhou, Cynthia (NSB -
>         CN/Hangzhou) <cynthia.zhou at nokia-sbell.com
>         <mailto:cynthia.zhou at nokia-sbell.com>> wrote:
>
>             When disable both quick-read and performance.io-cache off
>             everything is back to normal
>
>             I attached the log when only enable quick-read and
>             performance.io-cache is still on glusterfs trace log
>
>             When execute command “cat /mnt/export/testfile”
>
>             Can you help to find why this still to fail to show
>             correct content?
>
>             The file size showed is 141, but actually in brick it is
>             longer than that.
>
>             cynthia
>
>             *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
>             *Sent:* 2020年3月12日 12:53
>             *To:* 'Kotresh Hiremath Ravishankar' <khiremat at redhat.com
>             <mailto:khiremat at redhat.com>>
>             *Cc:* 'Gluster Devel' <gluster-devel at gluster.org
>             <mailto:gluster-devel at gluster.org>>
>             *Subject:* RE: could you help to check about a glusterfs
>             issue seems to be related to ctime
>
>             From my local test only when disable both features.ctime
>             and ctime.noatime this issue is gone.
>
>             Or
>
>             Do echo 3 >/proc/sys/vm/drop_caches after each time when
>             some client change the file , can cat command show correct
>             data(same as brick )
>
>             cynthia
>
>             *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
>             *Sent:* 2020年3月12日 9:53
>             *To:* 'Kotresh Hiremath Ravishankar' <khiremat at redhat.com
>             <mailto:khiremat at redhat.com>>
>             *Cc:* Gluster Devel <gluster-devel at gluster.org
>             <mailto:gluster-devel at gluster.org>>
>             *Subject:* RE: could you help to check about a glusterfs
>             issue seems to be related to ctime
>
>             Hi,
>
>             Thanks for your responding!
>
>             I’ve tried to disable quick-read:
>
>             [root at mn-0:/home/robot]
>
>             # gluster v get export all| grep quick
>
>             performance.quick-read off
>
>             performance.nfs.quick-read off
>
>             however, this issue still exists.
>
>             Two clients see different contents.
>
>             it seems only after I disable utime this issue is
>             completely gone.
>
>             features.ctime off
>
>             ctime.noatime off
>
>             Do you know why is this?
>
>             Cynthia
>
>             Nokia storage team
>
>             *From:* Kotresh Hiremath Ravishankar <khiremat at redhat.com
>             <mailto:khiremat at redhat.com>>
>             *Sent:* 2020年3月11日 22:05
>             *To:* Zhou, Cynthia (NSB - CN/Hangzhou)
>             <cynthia.zhou at nokia-sbell.com
>             <mailto:cynthia.zhou at nokia-sbell.com>>
>             *Cc:* Gluster Devel <gluster-devel at gluster.org
>             <mailto:gluster-devel at gluster.org>>
>             *Subject:* Re: could you help to check about a glusterfs
>             issue seems to be related to ctime
>
>             Hi,
>
>             I figured out what's happening. The issue is that the file
>             has 'c|a|m' time set to future (The file is created after
>             the date is set to +30 days). This
>             is done from client-1. On client-2 with correct date, when
>             data is appended, it doesn't update the mtime and ctime
>             because of both mtime and ctime is less than
>             already set time on the file. This protection is required
>             to keep the latest time when two clients are writing to
>             the same file. We update c|m|a time only if it's greater than
>             existing time. As a result, the perf xlators on client1
>             which relies on mtime doesn't send read to server as it
>             thinks nothing is changed as in this case the times haven't
>             changed.
>
>             Workarounds:
>             1. Disabling quick-read solved the issue for me.
>
>             I don't know how real this kind of workload is? Is this a
>             normal scenario ?
>             The other thing to do is to remove that protection of
>             updating time only if it's greater but that would open up
>             the race when two clients are updating the same file.
>
>             This would result in keeping the older time than the
>             latest. This requires code change and I don't think that
>             should be done.
>
>             Thanks,
>             Kotresh
>
>             On Wed, Mar 11, 2020 at 3:02 PM Kotresh Hiremath
>             Ravishankar <khiremat at redhat.com
>             <mailto:khiremat at redhat.com>> wrote:
>
>                 Exactly, I am also curious about this. I will debug
>                 and update about what's exactly happening.
>
>                 Thanks,
>                 Kotresh
>
>                 On Wed, Mar 11, 2020 at 1:56 PM Zhou, Cynthia (NSB -
>                 CN/Hangzhou) <cynthia.zhou at nokia-sbell.com
>                 <mailto:cynthia.zhou at nokia-sbell.com>> wrote:
>
>                     I used to think the file is cached in some client
>                     side buffer, because I’ve checked from different
>                     sn brick, the file content are all correct. But
>                     when I open client side trace level log, and cat
>                     the file, I only find lookup/open/flush fop from
>                     fuse-bridge side, I am just wondering how is file
>                     content served to client side? Should not there be
>                     readv fop seen from trace log?
>
>                     cynthia
>
>                     *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
>                     *Sent:* 2020年3月11日 15:54
>                     *To:* 'Kotresh Hiremath Ravishankar'
>                     <khiremat at redhat.com <mailto:khiremat at redhat.com>>
>                     *Subject:* RE: could you help to check about a
>                     glusterfs issue seems to be related to ctime
>
>                     Does that require, that for all the time client
>                     should be time synched? What if the client time is
>                     not synched for a while? And then restored?
>
>                     I make a test when time has been restored and then
>                     client change the file, the file’s modify time,
>                     access times remains to be wrong, is that correct?
>
>                     root at mn-0:/home/robot]
>
>                     # echo "fromm mn-0">>/mnt/export/testfile
>
>                     [root at mn-0:/home/robot]
>
>                     # stat /mnt/export/testfile
>
>                       File: /mnt/export/testfile
>
>                       Size: 30 Blocks: 1 IO Block: 131072 regular file
>
>                     Device: 28h/40d Inode: 9855109080001305442 Links: 1
>
>                     Access: (0644/-rw-r--r--) Uid: (    0/ root)  
>                     Gid: ( 615/_nokfsuifileshare)
>
>                     Access: 2020-05-10 09:33:59.713840197 +0300
>
>                     Modify: 2020-05-10 09:33:59.713840197 +0300
>
>                     Change: 2020-05-10 09:33:59.714413772 +0300 
>                     //remains to be future time
>
>                     Birth: -
>
>                     [root at mn-0:/home/robot]
>
>                     # cat /mnt/export/testfil
>
>                     cat: /mnt/export/testfil: No such file or directory
>
>                     [root at mn-0:/home/robot]
>
>                     # cat /mnt/export/testfile
>
>                     from mn0
>
>                     from mn-1
>
>                     fromm mn-0
>
>                     [root at mn-0:/home/robot]
>
>                     # date
>
>                     Wed 11 Mar 2020 09:05:58 AM EET
>
>                     [root at mn-0:/home/robot]
>
>                     cynthia
>
>                     *From:* Kotresh Hiremath Ravishankar
>                     <khiremat at redhat.com <mailto:khiremat at redhat.com>>
>                     *Sent:* 2020年3月11日 15:41
>                     *To:* Zhou, Cynthia (NSB - CN/Hangzhou)
>                     <cynthia.zhou at nokia-sbell.com
>                     <mailto:cynthia.zhou at nokia-sbell.com>>
>                     *Subject:* Re: could you help to check about a
>                     glusterfs issue seems to be related to ctime
>
>                     On Wed, Mar 11, 2020 at 12:46 PM Zhou, Cynthia
>                     (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com
>                     <mailto:cynthia.zhou at nokia-sbell.com>> wrote:
>
>                         But there are times that ntp service went
>                         wrong, and time on two storage nodes may be
>                         not synced.
>
>                         Or do you mean when can not guarantee that the
>                         time on two clients is synched, we should not
>                         enable this ctime feature?
>
>                     Yes, that's correct. The ctime feature relies on
>                     the time generated at the client (that's the utime
>                     xlator loaded in client) and hence
>                     expects all clients to be ntp synced.
>
>                         Without ctime feature, is there some way to
>                         avoid this “file changed as we read it” issue?
>
>                     Unfortunately no. That's the only way as of now.
>
>                         cynthia
>
>                         *From:* Kotresh Hiremath Ravishankar
>                         <khiremat at redhat.com
>                         <mailto:khiremat at redhat.com>>
>                         *Sent:* 2020年3月11日 15:12
>                         *To:* Zhou, Cynthia (NSB - CN/Hangzhou)
>                         <cynthia.zhou at nokia-sbell.com
>                         <mailto:cynthia.zhou at nokia-sbell.com>>
>                         *Subject:* Re: could you help to check about a
>                         glusterfs issue seems to be related to ctime
>
>                         Hi,
>
>                         I have not looked at it. I will take a look
>                         and update you. But one of the pre-requisite
>                         for ctime feature is that the clients should
>                         be time synced (ntp or other means).
>                         Could you try your reproducer by syncing the
>                         time of all clients and update me back ?
>
>                         Thanks,
>
>                         Kotresh HR
>
>                         On Wed, Mar 11, 2020 at 12:37 PM Zhou, Cynthia
>                         (NSB - CN/Hangzhou)
>                         <cynthia.zhou at nokia-sbell.com
>                         <mailto:cynthia.zhou at nokia-sbell.com>> wrote:
>
>                             I make some test
>
>                             After change date to future, and touch a
>                             file write sth, then restore time to normal.
>
>                             Then append sth to the file, the file
>                             modify time access time is still future,
>                             it is not the same with ext4,
>
>                             I think this is wrong.
>
>                             [root at mn-0:/home/robot]
>
>                             # echo "fromm mn-0">>/mnt/export/testfile
>
>                             [root at mn-0:/home/robot]
>
>                             # stat /mnt/export/testfile
>
>                             File: /mnt/export/testfile
>
>                             Size: 30 Blocks: 1          IO Block:
>                             131072 regular file
>
>                             Device: 28h/40d Inode:
>                             9855109080001305442  Links: 1
>
>                             Access: (0644/-rw-r--r--)  Uid: (    0/   
>                             root)   Gid: ( 615/_nokfsuifileshare)
>
>                             Access: 2020-05-10 09:33:59.713840197 +0300
>
>                             Modify: 2020-05-10 09:33:59.713840197 +0300
>
>                             Change: 2020-05-10 09:33:59.714413772 +0300
>
>                             Birth: -
>
>                             [root at mn-0:/home/robot]
>
>                             # cat /mnt/export/testfil
>
>                             cat: /mnt/export/testfil: No such file or
>                             directory
>
>                             [root at mn-0:/home/robot]
>
>                             # cat /mnt/export/testfile
>
>                             from mn0
>
>                             from mn-1
>
>                             fromm mn-0
>
>                             [root at mn-0:/home/robot]
>
>                             # date
>
>                             Wed 11 Mar 2020 09:05:58 AM EET
>
>                             [root at mn-0:/home/robot]
>
>                             cynthia
>
>                             *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
>                             *Sent:* 2020年3月11日 14:41
>                             *To:* 'khiremat at redhat.com
>                             <mailto:khiremat at redhat.com>'
>                             <khiremat at redhat.com
>                             <mailto:khiremat at redhat.com>>
>                             *Subject:* could you help to check about a
>                             glusterfs issue seems to be related to ctime
>
>                             Hi glusterfs expert,
>
>                             Good day!
>
>                             Do you have a time to check the ticket :
>                             https://bugzilla.redhat.com/show_bug.cgi?id=1811907
>                             ?
>
>                             After I disable feature ctime this issue
>                             is gone, however, I meet erro “file
>                             changed as we read it” when using tar command.
>
>                             Have you any idea?
>
>                             Thanks!
>
>                             Cynthia
>
>                             Nokia storage team
>
>
>                         -- 
>
>                         Thanks and Regards,
>
>                         Kotresh H R
>
>
>                     -- 
>
>                     Thanks and Regards,
>
>                     Kotresh H R
>
>
>                 -- 
>
>                 Thanks and Regards,
>
>                 Kotresh H R
>
>
>             -- 
>
>             Thanks and Regards,
>
>             Kotresh H R
>
>
>         -- 
>
>         Thanks and Regards,
>
>         Kotresh H R
>
>         _______________________________________________
>
>         Community Meeting Calendar:
>
>         Schedule -
>         Every Tuesday at 14:30 IST / 09:00 UTC
>         Bridge: https://bluejeans.com/441850968
>
>
>
>
>         Gluster-devel mailing list
>         Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>         https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
>     -- 
>
>     --
>
>     https://kadalu.io
>
>     Container Storage made easy!
>
>
> -- 
>
> Thanks and Regards,
>
> Kotresh H R
>
>
> _______________________________________________
>
> Community Meeting Calendar:
>
> Schedule -
> Every Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
>
>
>
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20200317/e83a531b/attachment-0001.html>


More information about the Gluster-devel mailing list