<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p><br>
</p>
<div class="moz-cite-prefix">On 17/03/20 12:56 pm, Zhou, Cynthia
(NSB - CN/Hangzhou) wrote:<br>
</div>
<blockquote type="cite"
cite="mid:8ac3df79ab954f4499c8ba67e88f4368@nokia-sbell.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:等线;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:微软雅黑;
panose-1:2 11 5 3 2 2 4 2 2 4;}
@font-face
{font-family:"\@等线";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"\@微软雅黑";}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:2138403854;
mso-list-type:hybrid;
mso-list-template-ids:-1324577580 -413370812 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:%1>;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal">I know that this may not be your
presumption of this ctime feature, however, our delima is that<o:p></o:p></p>
<ol style="margin-top:0cm" type="1" start="1">
<li class="MsoListParagraph"
style="margin-left:0cm;mso-list:l0 level1 lfo1">product need
to support date change scenario.<o:p></o:p></li>
<li class="MsoListParagraph"
style="margin-left:0cm;mso-list:l0 level1 lfo1">if we
disable ctime feature, we will meet tar issue.<o:p></o:p></li>
<li class="MsoListParagraph"
style="margin-left:0cm;mso-list:l0 level1 lfo1">if we set
consistent-metadata to on to work around this tar issue,
there is another even worse problem:
<a
href="https://bugzilla.redhat.com/show_bug.cgi?id=1773476"
moz-do-not-send="true">https://bugzilla.redhat.com/show_bug.cgi?id=1773476</a>,
app just cranshed!<o:p></o:p></li>
</ol>
<p class="MsoNormal">So I have to make this patch.</p>
</div>
</blockquote>
<p>For what it is worth, the 'file changed as we read it' warning
from tar can be suppressed with the --warning=no-file-changed
flag.</p>
<p>For 3, do you see any visible usability/performance issues with
the work-around mentioned in the bug (comment 5 and 6)? Just
curious.<br>
</p>
Regards,<br>
Ravi<br>
<blockquote type="cite"
cite="mid:8ac3df79ab954f4499c8ba67e88f4368@nokia-sbell.com">
<div class="WordSection1">
<p class="MsoNormal"><o:p></o:p></p>
<p class="MsoNormal">Maybe I will keep this as internal use just
for our local use scenarios.<o:p></o:p></p>
<p class="MsoNormal">Cynthia.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>From:</b> Kotresh Hiremath Ravishankar
<a class="moz-txt-link-rfc2396E" href="mailto:khiremat@redhat.com"><khiremat@redhat.com></a>
<br>
<b>Sent:</b> 2020<span
style="font-family:"微软雅黑",sans-serif" lang="ZH-CN">年</span>3<span
style="font-family:"微软雅黑",sans-serif" lang="ZH-CN">月</span>17<span
style="font-family:"微软雅黑",sans-serif" lang="ZH-CN">日</span>
15:17<br>
<b>To:</b> Zhou, Cynthia (NSB - CN/Hangzhou)
<a class="moz-txt-link-rfc2396E" href="mailto:cynthia.zhou@nokia-sbell.com"><cynthia.zhou@nokia-sbell.com></a><br>
<b>Cc:</b> Amar Tumballi <a class="moz-txt-link-rfc2396E" href="mailto:amar@kadalu.io"><amar@kadalu.io></a>; Gluster Devel
<a class="moz-txt-link-rfc2396E" href="mailto:gluster-devel@gluster.org"><gluster-devel@gluster.org></a><br>
<b>Subject:</b> Re: [Gluster-devel] could you help to check
about a glusterfs issue seems to be related to ctime<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">The whole ctime features relies on time
provided by clients which are time synchronized. This
patch brings in the time from server to compare against
the time sent from client.<br>
As Amar mentioned, this doesn't fit well into the scheme
of how ctime is designed. Definitely keeping it optional
and disabling it by default is one way. But is that what
your intention<br>
here?<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Mar 17, 2020 at 10:56 AM
Zhou, Cynthia (NSB - CN/Hangzhou) <<a
href="mailto:cynthia.zhou@nokia-sbell.com"
moz-do-not-send="true">cynthia.zhou@nokia-sbell.com</a>>
wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC
1.0pt;padding:0cm 0cm 0cm
6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Ok,
thanks for your feedback!<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I
will do local test to verify this patch first.<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">cynthia<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b>
Amar Tumballi <<a href="mailto:amar@kadalu.io"
target="_blank" moz-do-not-send="true">amar@kadalu.io</a>>
<br>
<b>Sent:</b> 2020<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">年</span>3<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">月</span>17<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">日</span> 13:18<br>
<b>To:</b> Zhou, Cynthia (NSB - CN/Hangzhou) <<a
href="mailto:cynthia.zhou@nokia-sbell.com"
target="_blank" moz-do-not-send="true">cynthia.zhou@nokia-sbell.com</a>><br>
<b>Cc:</b> Kotresh Hiremath Ravishankar <<a
href="mailto:khiremat@redhat.com" target="_blank"
moz-do-not-send="true">khiremat@redhat.com</a>>;
Gluster Devel <<a
href="mailto:gluster-devel@gluster.org"
target="_blank" moz-do-not-send="true">gluster-devel@gluster.org</a>><br>
<b>Subject:</b> Re: [Gluster-devel] could you help
to check about a glusterfs issue seems to be related
to ctime<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On
Tue, Mar 17, 2020 at 10:18 AM Zhou, Cynthia
(NSB - CN/Hangzhou) <<a
href="mailto:cynthia.zhou@nokia-sbell.com"
target="_blank" moz-do-not-send="true">cynthia.zhou@nokia-sbell.com</a>>
wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid
#CCCCCC 1.0pt;padding:0cm 0cm 0cm
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi
glusterfs expert,<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Our
product need to tolerate change date to
future and then change back.<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">How
about change like this ?<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a
href="https://review.gluster.org/#/c/glusterfs/+/24229/1/xlators/storage/posix/src/posix-metadata.c"
target="_blank" moz-do-not-send="true">https://review.gluster.org/#/c/glusterfs/+/24229/1/xlators/storage/posix/src/posix-metadata.c</a><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">when
time change to future and change back ,
should still be able to update mdata, so
the following changes to file can be
populated to other clients.<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We
do like to have people integrating with
GlusterFS. But this change is not inline with
the 'assumptions' we had about the feature.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">If
you have verified this change works for you,
please add it as an 'option' in posix, which
can be changed through volume set, and keep
this option disable/off by default. That
should be an easier way to get the patch
reviewed and take it further. Please make sure
to provide the 'description' for the option
with details.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Amar<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid
#CCCCCC 1.0pt;padding:0cm 0cm 0cm
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">cynthia<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div style="border:none;border-top:solid
#E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b>
Zhou, Cynthia (NSB - CN/Hangzhou)
<br>
<b>Sent:</b> 2020<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">年</span>3<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">月</span>12<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">日</span> 17:31<br>
<b>To:</b> 'Kotresh Hiremath
Ravishankar' <<a
href="mailto:khiremat@redhat.com"
target="_blank"
moz-do-not-send="true">khiremat@redhat.com</a>><br>
<b>Cc:</b> 'Gluster Devel' <<a
href="mailto:gluster-devel@gluster.org"
target="_blank"
moz-do-not-send="true">gluster-devel@gluster.org</a>><br>
<b>Subject:</b> RE: could you help to
check about a glusterfs issue seems to
be related to ctime<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi,<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">One
more question, I find each client has the
same future time stamp where are those
time stamps from, since Since it is
different from any brick stored time
stamp. And after I modify files from
clients, it remains the same.<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[root@mn-0:/home/robot]<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">#
stat /mnt/export/testfile<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">
File: /mnt/export/testfile<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">
Size: 193 Blocks: 1
IO Block: 131072 regular file<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Device:
28h/40d Inode: 10383279039841136109
Links: 1<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Access:
(0644/-rw-r--r--) Uid: ( 0/ root)
Gid: ( 615/_nokfsuifileshare)<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:red">Access: 2020-04-11
12:20:22.114365172 +0300</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:red">Modify: 2020-04-11
12:20:22.121552573 +0300</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="color:red">Change: 2020-04-11
12:20:22.121552573 +0300</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[root@mn-0:/home/robot]<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">#
date<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thu
Mar 12 11:27:33 EET 2020<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[root@mn-0:/home/robot]<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[root@mn-0:/home/robot]<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">#
stat /mnt/bricks/export/brick/testfile<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">
File: /mnt/bricks/export/brick/testfile<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">
Size: 193 Blocks: 16
IO Block: 4096 regular file<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Device:
fc02h/64514d Inode: 512015 Links:
2<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Access:
(0644/-rw-r--r--) Uid: ( 0/ root)
Gid: ( 615/_nokfsuifileshare)<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Access:
2020-04-11 12:20:22.100395536 +0300<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Modify:
2020-03-12 11:25:04.095981276 +0200<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Change:
2020-03-12 11:25:04.095981276 +0200<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Birth:
2020-04-11 08:53:26.805163816 +0300<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[root@mn-1:/root]<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">#
stat /mnt/bricks/export/brick/testfile<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">
File: /mnt/bricks/export/brick/testfile<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">
Size: 193 Blocks: 16
IO Block: 4096 regular file<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Device:
fc02h/64514d Inode: 512015 Links:
2<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Access:
(0644/-rw-r--r--) Uid: ( 0/ root)
Gid: ( 615/_nokfsuifileshare)<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Access:
2020-04-11 12:20:22.100395536 +0300<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Modify:
2020-03-12 11:25:04.094913452 +0200<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Change:
2020-03-12 11:25:04.095913453 +0200<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Birth:
2020-03-12 07:53:26.803783053 +0200<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div style="border:none;border-top:solid
#E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b>
Zhou, Cynthia (NSB - CN/Hangzhou)
<br>
<b>Sent:</b> 2020<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">年</span>3<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">月</span>12<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">日</span> 16:09<br>
<b>To:</b> 'Kotresh Hiremath
Ravishankar' <<a
href="mailto:khiremat@redhat.com"
target="_blank"
moz-do-not-send="true">khiremat@redhat.com</a>><br>
<b>Cc:</b> Gluster Devel <<a
href="mailto:gluster-devel@gluster.org"
target="_blank"
moz-do-not-send="true">gluster-devel@gluster.org</a>><br>
<b>Subject:</b> RE: could you help to
check about a glusterfs issue seems to
be related to ctime<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi,<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This
is abnormal test case, however, when this
happened it will have big impact on the
apps using those files. And this can not
be restored automatically unless disable
some xlator, I think it is unacceptable
for the user apps.<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">cynthia<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b>
Kotresh Hiremath Ravishankar <<a
href="mailto:khiremat@redhat.com"
target="_blank" moz-do-not-send="true">khiremat@redhat.com</a>>
<br>
<b>Sent:</b> 2020<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">年</span>3<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">月</span>12<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">日</span> 14:37<br>
<b>To:</b> Zhou, Cynthia (NSB -
CN/Hangzhou) <<a
href="mailto:cynthia.zhou@nokia-sbell.com"
target="_blank" moz-do-not-send="true">cynthia.zhou@nokia-sbell.com</a>><br>
<b>Cc:</b> Gluster Devel <<a
href="mailto:gluster-devel@gluster.org"
target="_blank" moz-do-not-send="true">gluster-devel@gluster.org</a>><br>
<b>Subject:</b> Re: could you help to
check about a glusterfs issue seems to be
related to ctime<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">All
the perf xlators depend on time (mostly
mtime I guess). In my setup, only quick
read was enabled and hence disabling it
worked for me.<br>
All perf xlators needs to be disabled to
make it work correctly. But I still
failed to understand how normal this
kind of workload ?<o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks,<br>
Kotresh<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On
Thu, Mar 12, 2020 at 11:20 AM Zhou,
Cynthia (NSB - CN/Hangzhou) <<a
href="mailto:cynthia.zhou@nokia-sbell.com"
target="_blank"
moz-do-not-send="true">cynthia.zhou@nokia-sbell.com</a>>
wrote:<o:p></o:p></p>
</div>
<blockquote
style="border:none;border-left:solid
#CCCCCC 1.0pt;padding:0cm 0cm 0cm
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">When
disable both quick-read and
performance.io-cache off
everything is back to normal<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I
attached the log when only enable
quick-read and
performance.io-cache is still on
glusterfs trace log<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">When
execute command “cat
/mnt/export/testfile”<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Can
you help to find why this still to
fail to show correct content?<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The
file size showed is 141, but
actually in brick it is longer
than that.<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">cynthia<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div
style="border:none;border-top:solid
#E1E1E1 1.0pt;padding:3.0pt 0cm
0cm 0cm">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b>
Zhou, Cynthia (NSB -
CN/Hangzhou)
<br>
<b>Sent:</b> 2020<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">年</span>3<span
style="font-family:"微软雅黑",sans-serif" lang="ZH-CN">月</span>12<span
style="font-family:"微软雅黑",sans-serif" lang="ZH-CN">日</span>
12:53<br>
<b>To:</b> 'Kotresh Hiremath
Ravishankar' <<a
href="mailto:khiremat@redhat.com"
target="_blank"
moz-do-not-send="true">khiremat@redhat.com</a>><br>
<b>Cc:</b> 'Gluster Devel'
<<a
href="mailto:gluster-devel@gluster.org"
target="_blank"
moz-do-not-send="true">gluster-devel@gluster.org</a>><br>
<b>Subject:</b> RE: could you
help to check about a
glusterfs issue seems to be
related to ctime<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">From
my local test only when disable
both features.ctime and
ctime.noatime this issue is gone.<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Or<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Do
echo 3
>/proc/sys/vm/drop_caches after
each time when some client change
the file , can cat command show
correct data(same as brick )<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">cynthia<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div
style="border:none;border-top:solid
#E1E1E1 1.0pt;padding:3.0pt 0cm
0cm 0cm">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b>
Zhou, Cynthia (NSB -
CN/Hangzhou)
<br>
<b>Sent:</b> 2020<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">年</span>3<span
style="font-family:"微软雅黑",sans-serif" lang="ZH-CN">月</span>12<span
style="font-family:"微软雅黑",sans-serif" lang="ZH-CN">日</span>
9:53<br>
<b>To:</b> 'Kotresh Hiremath
Ravishankar' <<a
href="mailto:khiremat@redhat.com"
target="_blank"
moz-do-not-send="true">khiremat@redhat.com</a>><br>
<b>Cc:</b> Gluster Devel <<a
href="mailto:gluster-devel@gluster.org" target="_blank"
moz-do-not-send="true">gluster-devel@gluster.org</a>><br>
<b>Subject:</b> RE: could you
help to check about a
glusterfs issue seems to be
related to ctime<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi,<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks
for your responding!<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I’ve
tried to disable quick-read:<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[root@mn-0:/home/robot]<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">#
gluster v get export all| grep
quick<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">performance.quick-read
off<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">performance.nfs.quick-read
off<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">however,
this issue still exists.<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Two
clients see different contents.<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">it
seems only after I disable utime
this issue is completely gone.<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">features.ctime
off<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">ctime.noatime
off<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Do
you know why is this?<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Cynthia<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Nokia
storage team<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b>
Kotresh Hiremath Ravishankar <<a
href="mailto:khiremat@redhat.com" target="_blank" moz-do-not-send="true">khiremat@redhat.com</a>>
<br>
<b>Sent:</b> 2020<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">年</span>3<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">月</span>11<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">日</span> 22:05<br>
<b>To:</b> Zhou, Cynthia (NSB -
CN/Hangzhou) <<a
href="mailto:cynthia.zhou@nokia-sbell.com"
target="_blank"
moz-do-not-send="true">cynthia.zhou@nokia-sbell.com</a>><br>
<b>Cc:</b> Gluster Devel <<a
href="mailto:gluster-devel@gluster.org"
target="_blank"
moz-do-not-send="true">gluster-devel@gluster.org</a>><br>
<b>Subject:</b> Re: could you help
to check about a glusterfs issue
seems to be related to ctime<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi,<br>
<br>
I figured out what's happening.
The issue is that the file has
'c|a|m' time set to future (The
file is created after the date
is set to +30 days). This<br>
is done from client-1. On
client-2 with correct date, when
data is appended, it doesn't
update the mtime and ctime
because of both mtime and ctime
is less than<br>
already set time on the file.
This protection is required to
keep the latest time when two
clients are writing to the same
file. We update c|m|a time only
if it's greater than<br>
existing time. As a result, the
perf xlators on client1 which
relies on mtime doesn't send
read to server as it thinks
nothing is changed as in this
case the times haven't<br>
changed.<o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;margin-bottom:12.0pt">Workarounds:<br>
1. Disabling quick-read solved
the issue for me.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I
don't know how real this kind
of workload is? Is this a
normal scenario ?<br>
The other thing to do is to
remove that protection of
updating time only if it's
greater but that would open up
the race when two clients are
updating the same file.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This
would result in keeping the
older time than the latest.
This requires code change and
I don't think that should be
done.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks,<br>
Kotresh<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On
Wed, Mar 11, 2020 at 3:02 PM
Kotresh Hiremath Ravishankar
<<a
href="mailto:khiremat@redhat.com"
target="_blank"
moz-do-not-send="true">khiremat@redhat.com</a>>
wrote:<o:p></o:p></p>
</div>
<blockquote
style="border:none;border-left:solid
#CCCCCC 1.0pt;padding:0cm 0cm
0cm
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Exactly,
I am also curious about
this. I will debug and
update about what's exactly
happening.<o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks,<br>
Kotresh <o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On
Wed, Mar 11, 2020 at 1:56
PM Zhou, Cynthia (NSB -
CN/Hangzhou) <<a
href="mailto:cynthia.zhou@nokia-sbell.com"
target="_blank"
moz-do-not-send="true">cynthia.zhou@nokia-sbell.com</a>>
wrote:<o:p></o:p></p>
</div>
<blockquote
style="border:none;border-left:solid
#CCCCCC 1.0pt;padding:0cm
0cm 0cm
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I
used to think the file
is cached in some
client side buffer,
because I’ve checked
from different sn
brick, the file
content are all
correct. But when I
open client side trace
level log, and cat the
file, I only find
lookup/open/flush fop
from fuse-bridge side,
I am just wondering
how is file content
served to client side?
Should not there be
readv fop seen from
trace log?<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">cynthia<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div
style="border:none;border-top:solid
#E1E1E1
1.0pt;padding:3.0pt
0cm 0cm 0cm">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b>
Zhou, Cynthia (NSB
- CN/Hangzhou)
<br>
<b>Sent:</b> 2020<span
style="font-family:"微软雅黑",sans-serif" lang="ZH-CN">年</span>3<span
style="font-family:"微软雅黑",sans-serif" lang="ZH-CN">月</span>11<span
style="font-family:"微软雅黑",sans-serif" lang="ZH-CN">日</span>
15:54<br>
<b>To:</b>
'Kotresh Hiremath
Ravishankar' <<a
href="mailto:khiremat@redhat.com" target="_blank" moz-do-not-send="true">khiremat@redhat.com</a>><br>
<b>Subject:</b>
RE: could you help
to check about a
glusterfs issue
seems to be
related to ctime<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Does
that require, that for
all the time client
should be time
synched? What if the
client time is not
synched for a while?
And then restored?<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I
make a test when time
has been restored and
then client change the
file, the file’s
modify time, access
times remains to be
wrong, is that
correct?<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New"">root@mn-0:/home/robot]</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New""># echo
"fromm
mn-0">>/mnt/export/testfile</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New"">[root@mn-0:/home/robot]</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New""># stat
/mnt/export/testfile</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New""> File:
/mnt/export/testfile</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New""> Size:
30
Blocks: 1
IO Block: 131072
regular file</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New"">Device:
28h/40d Inode:
9855109080001305442
Links: 1</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New"">Access:
(0644/-rw-r--r--)
Uid: ( 0/
root) Gid: (
615/_nokfsuifileshare)</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New";color:red">Access:
2020-05-10
09:33:59.713840197
+0300</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New";color:red">Modify:
2020-05-10
09:33:59.713840197
+0300</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New";color:red">Change:
2020-05-10
09:33:59.714413772
+0300 //remains to
be future time</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New"">Birth: -</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New"">[root@mn-0:/home/robot]</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New""># cat
/mnt/export/testfil</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New"">cat:
/mnt/export/testfil:
No such file or
directory</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New"">[root@mn-0:/home/robot]</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New""># cat
/mnt/export/testfile</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New"">from mn0</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New"">from mn-1</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New"">fromm mn-0</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New"">[root@mn-0:/home/robot]</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New""># date</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New"">Wed 11 Mar
2020 09:05:58 AM EET</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:10.0pt;font-family:"Courier New"">[root@mn-0:/home/robot]</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">cynthia<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b>
Kotresh Hiremath
Ravishankar <<a
href="mailto:khiremat@redhat.com"
target="_blank"
moz-do-not-send="true">khiremat@redhat.com</a>>
<br>
<b>Sent:</b> 2020<span
style="font-family:"微软雅黑",sans-serif" lang="ZH-CN">年</span>3<span
style="font-family:"微软雅黑",sans-serif" lang="ZH-CN">月</span>11<span
style="font-family:"微软雅黑",sans-serif" lang="ZH-CN">日</span>
15:41<br>
<b>To:</b> Zhou,
Cynthia (NSB -
CN/Hangzhou) <<a
href="mailto:cynthia.zhou@nokia-sbell.com"
target="_blank"
moz-do-not-send="true">cynthia.zhou@nokia-sbell.com</a>><br>
<b>Subject:</b> Re:
could you help to
check about a
glusterfs issue seems
to be related to ctime<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Wed, Mar
11, 2020 at
12:46 PM Zhou,
Cynthia (NSB -
CN/Hangzhou)
<<a
href="mailto:cynthia.zhou@nokia-sbell.com"
target="_blank" moz-do-not-send="true">cynthia.zhou@nokia-sbell.com</a>>
wrote:<o:p></o:p></p>
</div>
<blockquote
style="border:none;border-left:solid
#CCCCCC
1.0pt;padding:0cm
0cm 0cm
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">But there are
times that ntp
service went
wrong, and
time on two
storage nodes
may be not
synced.<o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Or do you
mean when can
not guarantee
that the time
on two clients
is synched, we
should not
enable this
ctime feature?<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Yes, that's
correct. The
ctime feature
relies on the
time generated
at the client
(that's the
utime xlator
loaded in
client) and
hence<br>
expects all
clients to be
ntp synced. <o:p></o:p></p>
</div>
<blockquote
style="border:none;border-left:solid
#CCCCCC
1.0pt;padding:0cm
0cm 0cm
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Without ctime
feature, is
there some way
to avoid this
“file changed
as we read it”
issue?<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Unfortunately
no. That's the
only way as of
now. <o:p></o:p></p>
</div>
<blockquote
style="border:none;border-left:solid
#CCCCCC
1.0pt;padding:0cm
0cm 0cm
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">cynthia<o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b>
Kotresh
Hiremath
Ravishankar
<<a
href="mailto:khiremat@redhat.com"
target="_blank" moz-do-not-send="true">khiremat@redhat.com</a>>
<br>
<b>Sent:</b>
2020<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">年</span>3<span
style="font-family:"微软雅黑",sans-serif" lang="ZH-CN">月</span>11<span
style="font-family:"微软雅黑",sans-serif" lang="ZH-CN">日</span>
15:12<br>
<b>To:</b>
Zhou, Cynthia
(NSB -
CN/Hangzhou)
<<a
href="mailto:cynthia.zhou@nokia-sbell.com"
target="_blank" moz-do-not-send="true">cynthia.zhou@nokia-sbell.com</a>><br>
<b>Subject:</b>
Re: could you
help to check
about a
glusterfs
issue seems to
be related to
ctime<o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi,<br>
<br>
I have not
looked at it.
I will take a
look and
update you.
But one of the
pre-requisite
for ctime
feature is
that the
clients should
be time synced
(ntp or other
means).<br>
Could you try
your
reproducer by
syncing the
time of all
clients and
update me back
?<br>
<br>
Thanks,<o:p></o:p></p>
<div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Kotresh HR<o:p></o:p></p>
</div>
</div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Wed, Mar
11, 2020 at
12:37 PM Zhou,
Cynthia (NSB -
CN/Hangzhou)
<<a
href="mailto:cynthia.zhou@nokia-sbell.com"
target="_blank" moz-do-not-send="true">cynthia.zhou@nokia-sbell.com</a>>
wrote:<o:p></o:p></p>
</div>
<blockquote
style="border:none;border-left:solid
#CCCCCC
1.0pt;padding:0cm
0cm 0cm
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I make some
test<o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">After change
date to
future, and
touch a file
write sth,
then restore
time to
normal.<o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Then append
sth to the
file, the file
modify time
access time is
still future,
it is not the
same with
ext4,<o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I think this
is wrong.<o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">[root@mn-0:/home/robot]</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">#
echo "fromm
mn-0">>/mnt/export/testfile</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">[root@mn-0:/home/robot]</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">#
stat
/mnt/export/testfile</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">
File:
/mnt/export/testfile</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">
Size:
30
Blocks:
1 IO
Block: 131072
regular file</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">Device:
28h/40d Inode:
9855109080001305442 Links: 1</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">Access:
(0644/-rw-r--r--) Uid: ( 0/ root) Gid: (
615/_nokfsuifileshare)</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">Access:
2020-05-10
09:33:59.713840197
+0300</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">Modify:
2020-05-10
09:33:59.713840197
+0300</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">Change:
2020-05-10
09:33:59.714413772
+0300</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">Birth:
-</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">[root@mn-0:/home/robot]</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">#
cat
/mnt/export/testfil</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">cat:
/mnt/export/testfil: No such file or directory</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">[root@mn-0:/home/robot]</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">#
cat
/mnt/export/testfile</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">from
mn0</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">from
mn-1</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">fromm
mn-0</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">[root@mn-0:/home/robot]</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">#
date</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">Wed
11 Mar 2020
09:05:58 AM
EET</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:8.0pt">[root@mn-0:/home/robot]</span><o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">cynthia<o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div
style="border:none;border-top:solid
#E1E1E1
1.0pt;padding:3.0pt
0cm 0cm 0cm">
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b>
Zhou, Cynthia
(NSB -
CN/Hangzhou)
<br>
<b>Sent:</b>
2020<span
style="font-family:"微软雅黑",sans-serif"
lang="ZH-CN">年</span>3<span
style="font-family:"微软雅黑",sans-serif" lang="ZH-CN">月</span>11<span
style="font-family:"微软雅黑",sans-serif" lang="ZH-CN">日</span>
14:41<br>
<b>To:</b> '<a
href="mailto:khiremat@redhat.com" target="_blank" moz-do-not-send="true">khiremat@redhat.com</a>'
<<a
href="mailto:khiremat@redhat.com"
target="_blank" moz-do-not-send="true">khiremat@redhat.com</a>><br>
<b>Subject:</b>
could you help
to check about
a glusterfs
issue seems to
be related to
ctime<o:p></o:p></p>
</div>
</div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi glusterfs
expert,<o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Good day!<o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Do you have a
time to check
the ticket :
<a
href="https://bugzilla.redhat.com/show_bug.cgi?id=1811907"
target="_blank" moz-do-not-send="true">https://bugzilla.redhat.com/show_bug.cgi?id=1811907</a>
?<o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">After I
disable
feature ctime
this issue is
gone, however,
I meet erro
“file changed
as we read it”
when using tar
command.<o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Have you any
idea?<o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks!<o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Cynthia<o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Nokia storage
team<o:p></o:p></p>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br
clear="all">
<o:p></o:p></p>
<div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--
<o:p></o:p></p>
<div>
<div>
<div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks and
Regards,<o:p></o:p></p>
</div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Kotresh H R<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br
clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--
<o:p></o:p></p>
<div>
<div>
<div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks and
Regards,<o:p></o:p></p>
</div>
<p
class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Kotresh H R<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br
clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--
<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks
and Regards,<o:p></o:p></p>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Kotresh
H R<o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br
clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--
<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks
and Regards,<o:p></o:p></p>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Kotresh
H R<o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br
clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--
<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks
and Regards,<o:p></o:p></p>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Kotresh
H R<o:p></o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;margin-bottom:12.0pt">_______________________________________________<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a
href="https://bluejeans.com/441850968"
target="_blank" moz-do-not-send="true">https://bluejeans.com/441850968</a><br>
<br>
<br>
<br>
<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org"
target="_blank" moz-do-not-send="true">Gluster-devel@gluster.org</a><br>
<a
href="https://lists.gluster.org/mailman/listinfo/gluster-devel"
target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-devel</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br
clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--<o:p></o:p></p>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a
href="https://kadalu.io" target="_blank"
moz-do-not-send="true">https://kadalu.io</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Container
Storage made easy!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">Thanks and Regards,<o:p></o:p></p>
</div>
<p class="MsoNormal">Kotresh H R<o:p></o:p></p>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Community Meeting Calendar:
Schedule -
Every Tuesday at 14:30 IST / 09:00 UTC
Bridge: <a class="moz-txt-link-freetext" href="https://bluejeans.com/441850968">https://bluejeans.com/441850968</a>
Gluster-devel mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>
<a class="moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-devel">https://lists.gluster.org/mailman/listinfo/gluster-devel</a>
</pre>
</blockquote>
</body>
</html>