<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 15, 2017 at 11:31 PM, Soumya Koduri <span dir="ltr"><<a href="mailto:skoduri@redhat.com" target="_blank">skoduri@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Rafi,<br>
<br>
I haven't thoroughly gone through design. But have few comments/queries which I have posted inline for now .<span class=""><br>
<br>
On 02/28/2017 01:11 PM, Mohammed Rafi K C wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Thanks for the reply , Comments are inline<br>
<br>
<br>
<br>
On 02/28/2017 12:50 PM, Niels de Vos wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Tue, Feb 28, 2017 at 11:21:55AM +0530, Mohammed Rafi K C wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi All,<br>
<br>
<br>
We discussed the problem $subject in the mail thread [1]. Based on the<br>
comments and suggestions I will summarize the design (Made as points for<br>
simplicity.)<br>
<br>
<br>
1) As part of each fop, top layer will generate a time stamp and pass it<br>
to the down along with other param.<br>
<br>
1.1) This will bring a dependency for NTP synced clients along with<br>
servers<br>
</blockquote>
What do you mean with "top layer"? Is this on the Gluster client, or<br>
does the time get inserted on the bricks?<br>
</blockquote>
It is the top layer (master xlator) in client graph like fuse, gfapi,<br>
nfs . My mistake I should have mentioned . Sorry for that.<br>
</blockquote>
<br></span>
These clients shouldn't include internal client processes like rebalance, self-heal daemons right? IIUC from [1], we should avoid changing times during rebalance and self-heals.<br>
<br>
Also what about fops generated from the underlying layers - getxattr/setxattr which may modify these time attributes?<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I think we should not require a hard dependency on NTP, but have it<br>
strongly suggested. Having a synced time in a clustered environment is<br>
always helpful for reading and matching logs.<br>
</blockquote>
Agreed, but if we go with option 1 where we generate time from client,<br>
then time will not be in sync if not done with NTP.<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
1.2) There can be a diff in time if the fop stuck in the xlator for<br>
various reason, for ex: because of locks.<br>
</blockquote>
Or just slow networks? Blocking (mandatory?) locks should be handled<br>
correctly. The time a FOP is blocked can be long.<br>
</blockquote>
True, the questions can this be included in timestamp valie, because if<br>
it generated from say fuse then when it reaches to the brick the time<br>
may have moved ahead. what do you think about it ?<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
2) On the server posix layer stores the value in the memory (inode ctx)<br>
and will sync the data periodically to the disk as an extended attr<br>
</blockquote></blockquote></blockquote></span>
Will you use any timer thread for asynchronous update?<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
2.1) of course sync call also will force it. And fop comes for an<br>
inode which is not linked, we do the sync immediately.<br>
</blockquote>
Does it need to be in the posix layer?<br>
</blockquote>
<br>
You mean storing the time attr ? then it need not be , protocol/server<br>
is also another candidate but I feel posix is ahead in the race ;) .<br>
</blockquote>
<br></span>
I agree with Shyam and Niels that posix layer doesn't seem right. Since having this support comes with performance cost, how about a separate xlator (which shall be optional)?<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
3) Each time when inodes are created or initialized it read the data<br>
from disk and store it.<br>
<br>
<br>
4) Before setting to inode_ctx we compare the timestamp stored and the<br>
timestamp received, and only store if the stored value is lesser than<br>
the current value.<br>
</blockquote></blockquote></blockquote></span>
If we choose not to set this attribute for self-heal/rebalance (as stated above) daemons, we would need special handling for the requests sent by them (i.e, to heal this time attribute as well on the destination file/dir).<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
5) So in best case data will be stored and retrieved from the memory. We<br>
replace the values in iatt with the values in inode_ctx.<br>
<br>
<br>
6) File ops that changes the parent directory attr time need to be<br>
consistent across all the distributed directories across the subvolumes.<br>
(for eg: a create call will change ctime and mtime of parent dir)<br>
<br>
6.1) This has to handle separately because we only send the fop to<br>
the hashed subvolume.<br>
<br>
6.2) We can asynchronously send the timeupdate setattr fop to the<br>
other subvoumes and change the values for parent directory if the file<br>
fops is successful on hashed subvolume.<br>
</blockquote></blockquote></blockquote>
<br></span>
The same needs to be handled even during DHT directory healing right?<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
6.3) This will have a window where the times are inconsistent<br>
across dht subvolume (Please provide your suggestions)<br>
</blockquote>
Isn't this the same problem for 'normal' AFR volumes? I guess self-heal<br>
needs to know how to pick the right value for the [cm]time xattr.<br>
</blockquote>
<br>
Yes and need to heal. Both self heal and dht. But till then there can be<br>
difference in values.<br>
</blockquote>
<br></span>
Is this design targetting to synchronize only ctime/mtime? If 'atime' is also considered , as the read/stat done by AFR shall modify atime only on the first subvol, even AFR xlator needs to take care of updating other subvols. Same goes with EC as well.<br></blockquote><div><br></div><div>atime is updated on open which is sent to all subvols in AFR/EC<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Thanks,<br>
Soumya<div class="HOEnZb"><div class="h5"><br>
______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-devel</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>