<div dir="ltr"><div>Ashish, Xavi,<br></div> I think it is better to implement this change as a separate read-after-write caching xlator which we can load between EC and client xlator. That way EC will not get a lot more functionality than necessary and may be this xlator can be used somewhere else in the stack if possible.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 16, 2017 at 4:19 PM, Ashish Pandey <span dir="ltr"><<a href="mailto:aspandey@redhat.com" target="_blank">aspandey@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:times new roman,new york,times,serif;font-size:12pt;color:#000000"><div><br></div><div>I think it should be done as we have agreement on basic design.<br></div><div><br></div><hr id="m_-8936172880582030626zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><b>From: </b>"Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>><br><b>To: </b>"Xavier Hernandez" <<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><br><b>Cc: </b>"Ashish Pandey" <<a href="mailto:aspandey@redhat.com" target="_blank">aspandey@redhat.com</a>>, "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a>><br><b>Sent: </b>Friday, June 16, 2017 3:50:09 PM<br><b>Subject: </b>Re: [Gluster-devel] Disperse volume : Sequential Writes<div><div class="h5"><br><div><br></div><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 16, 2017 at 3:12 PM, Xavier Hernandez <span dir="ltr"><<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On 16/06/17 10:51, Pranith Kumar Karampuri wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>
<br>
<br>
On Fri, Jun 16, 2017 at 12:02 PM, Xavier Hernandez<br></span><span>
<<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a> <mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><wbr>> wrote:<br>
<br>
On 15/06/17 11:50, Pranith Kumar Karampuri wrote:<br>
<br>
<br>
<br>
On Thu, Jun 15, 2017 at 11:51 AM, Ashish Pandey<br>
<<a href="mailto:aspandey@redhat.com" target="_blank">aspandey@redhat.com</a> <mailto:<a href="mailto:aspandey@redhat.com" target="_blank">aspandey@redhat.com</a>><br></span><div><div class="m_-8936172880582030626h5">
<mailto:<a href="mailto:aspandey@redhat.com" target="_blank">aspandey@redhat.com</a> <mailto:<a href="mailto:aspandey@redhat.com" target="_blank">aspandey@redhat.com</a>>>> wrote:<br>
<br>
Hi All,<br>
<br>
We have been facing some issues in disperse (EC) volume.<br>
We know that currently EC is not good for random IO as it<br>
requires<br>
READ-MODIFY-WRITE fop<br>
cycle if an offset and offset+length falls in the middle of<br>
strip size.<br>
<br>
Unfortunately, it could also happen with sequential writes.<br>
Consider an EC volume with configuration 4+2. The stripe<br>
size for<br>
this would be 512 * 4 = 2048. That is, 2048 bytes of user data<br>
stored in one stripe.<br>
Let's say 2048 + 512 = 2560 bytes are already written on this<br>
volume. 512 Bytes would be in second stripe.<br>
Now, if there are sequential writes with offset 2560 and of<br>
size 1<br>
Byte, we have to read the whole stripe, encode it with 1<br>
Byte and<br>
then again have to write it back.<br>
Next, write with offset 2561 and size of 1 Byte will again<br>
READ-MODIFY-WRITE the whole stripe. This is causing bad<br>
performance.<br>
<br>
There are some tools and scenario's where such kind of load is<br>
coming and users are not aware of that.<br>
Example: fio and zip<br>
<br>
Solution:<br>
One possible solution to deal with this issue is to keep<br>
last stripe<br>
in memory.<br>
This way, we need not to read it again and we can save READ fop<br>
going over the network.<br>
Considering the above example, we have to keep last 2048 bytes<br>
(maximum) in memory per file. This should not be a big<br>
deal as we already keep some data like xattr's and size info in<br>
memory and based on that we take decisions.<br>
<br>
Please provide your thoughts on this and also if you have<br>
any other<br>
solution.<br>
<br>
<br>
Just adding more details.<br>
The stripe will be in memory only when lock on the inode is active.<br>
<br>
<br>
I think that's ok.<br>
<br>
One<br>
thing we are yet to decide on is: do we want to read the stripe<br>
everytime we get the lock or just after an extending write is<br>
performed.<br>
I am thinking keeping the stripe in memory just after an<br>
extending write<br>
is better as it doesn't involve extra network operation.<br>
<br>
<br>
I wouldn't read the last stripe unconditionally every time we lock<br>
the inode. There's no benefit at all on random writes (in fact it's<br>
worse) and a sequential write will issue the read anyway when<br>
needed. The only difference is a small delay for the first operation<br>
after a lock.<br>
<br>
<br>
Yes, perfect.<br>
<br>
<br>
<br>
What I would do is to keep the last stripe of every write (we can<br>
consider to do it per fd), even if it's not the last stripe of the<br>
file (to also optimize sequential rewrites).<br>
<br>
<br>
Ah! good point. But if we remember it per fd, one fd's cached data can<br>
be over-written by another fd on the disk so we need to also do cache<br>
invalidation.<br>
</div></div></blockquote>
<br>
We only cache data if we have the inodelk, so all related fd's must be from the same client, and we'll control all its writes so cache invalidation in this case is pretty easy.<br>
<br>
There exists the possibility to have two fd's from the same client writing to the same region. To control this we would need some range checking in the writes, but all this is local, so it's easy to control it.<br>
<br>
Anyway, this is probably not a common case, so we could start by caching only the last stripe of the last write, ignoring the fd.<span><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
May be implementation should consider this possibility.<br>
Yet to think about how to do this. But it is a good point. We should<br>
consider this.<br>
</blockquote>
<br></span>
Maybe we could keep a list of cached stripes sorted by offset in the inode (if the maximum number of entries is small, we could keep the list not sorted). Each fd should store the offset of the last write. Cached stripes should have a ref counter just to account for the case that two fd's point to the same offset.<br>
<br>
When a new write arrives, we check the offset stored in the fd and see if it corresponds to a sequential write. If so, we look at the inode list to find the cached stripe, otherwise we can release the cached stripe.<br>
<br>
We can limit the number of cached entries and release the least recently used when we reach some maximum.<span><br></span></blockquote><div><br></div><div>Yeah, this works :-).<br></div><div>Ashish,<br></div><div> Can all of this be implemented by 3.12?<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
<br>
One thing I've observed is that a 'dd' with block size of 1MB gets<br>
split into multiple 128KB blocks that are sent in parallel and not<br>
necessarily processed in the sequential order. This means that big<br>
block sizes won't benefit much from this optimization since they<br>
will be seen as partially non-sequential writes. Anyway the change<br>
won't hurt.<br>
<br>
<br>
In this case as per the solution we won't cache anything right? Because<br>
we didn't request anything from the disk. We will only keep the data in<br>
cache if it is not aligned write which is at the current EOF. At least<br>
that is what I had in mind.<br>
</blockquote>
<br></span>
Suppose we are writing multiple 1MB blocks at offset 1. If each write is split into 8 blocks of 128KB, all writes will be not aligned, and can be received in any order. Suppose that the first write happens to be at offset 128K + 1. We don't have anything cached, so we read the needed stripes and cache the last one. Now the next write is at offset 1. In this case we won't get any benefit from the previous write, since the stripe we need is not cached. However the write from the user point of view is sequential.<br>
<br>
It won't hurt but it won't take all benefits from the new caching mechanism.<br>
<br>
As a mitigating factor, we could consider to extend the previous solution I've explained to allow caching multiple stripes per fd. A small number like 8 would be enough.<br>
<br>
Xavi<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>
<br>
<br>
<br>
Xavi<br>
<br>
<br>
<br>
<br>
<br>
---<br>
Ashish<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a> <mailto:<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.<wbr>org</a>><br></span>
<mailto:<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.<wbr>org</a><span><br>
<mailto:<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.<wbr>org</a>>><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br>
<<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><wbr>><br>
<<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br>
<<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><wbr>>><br>
<br>
<br>
<br>
<br>
--<br>
Pranith<br>
<br>
<br>
<br>
<br>
<br>
--<br>
Pranith<br>
</span></blockquote>
<br>
</blockquote></div><br><br clear="all"><br>-- <br><div class="m_-8936172880582030626gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>
</div></div></div><div><br></div></div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div>