<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 8, 2018 at 12:05 PM, Raghavendra G <span dir="ltr"><<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Tue, Feb 6, 2018 at 8:15 PM, Vijay Bellur <span dir="ltr"><<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div><br></div><div class="gmail_extra"><div class="gmail_quote"><span>On Sun, Feb 4, 2018 at 3:39 AM, Raghavendra Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">All,<br>
<br>
One of our users pointed out to the documentation that glusterfs is not good for storing "Structured data" [1], while discussing an issue [2]. </blockquote><div><br></div><div><br></div></span><div>As far as I remember, the content around structured data in the Install Guide is from a FAQ that was being circulated in Gluster, Inc. indicating the startup's market positioning. Most of that was based on not wanting to get into performance based comparisons of storage systems that are frequently seen in the structured data space.<br></div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Does any of you have more context on the feasibility of storing "structured data" on Glusterfs? Is one of the reasons for such a suggestion "staleness of metadata" as encountered in bugs like [3]?<br></blockquote><div><br></div><div><br></div></span><div>There are challenges that distributed storage systems face when exposed to applications that were written for a local filesystem interface. We have encountered problems with applications like tar [4] that are not in the realm of "Structured data". If we look at the common theme across all these problems, it is related to metadata & read after write consistency issues with the default translator stack that gets exposed on the client side. While the default stack is optimal for other scenarios, it does seem that a category of applications needing strict metadata consistency is not well served by that. We have observed that disabling a few performance translators and tuning cache timeouts for VFS/FUSE have helped to overcome some of them. The WIP effort on timestamp consistency across the translator stack, patches that have been merged as a result of the bugs that you mention & other fixes for outstanding issues should certainly help in catering to these workloads better with the file interface.</div><div><br></div><div>There are deployments that I have come across where glusterfs is used for storing structured data. gluster-block & qemu-libgfapi overcome the metadata consistency problem by exposing a file as a block device & by disabling most of the performance translators in the default stack. Workloads that have been deemed problematic with the file interface for the reasons alluded above, function well with the block interface. </div></div></div></div></blockquote><div><br></div></span><div>I agree that gluster-block due to its usage of a subset of glusterfs fops (mostly reads/writes I guess), runs into less number of consistency issues. However, as you've mentioned we seem to disable perf xlator stack in our tests/use-cases till now. Note that perf xlator stack is one of worst offenders as far as the metadata consistency is concerned (relatively less scenarios of data inconsistency). So, I wonder,</div><div>* what would be the scenario if we enable perf xlator stack for gluster-block?</div><div>* Is performance on gluster-block satisfactory so that we don't need these xlators?</div><div> - Or is it that these xlators are not useful for the workload usually run on gluster-block (For random read/write workload, read/write caching xlators offer less or no advantage)? <br></div></div></div></div></blockquote><div><br></div><div>Yes. They are not useful. Block/VM files are opened with O_DIRECT, so we don't enable caching at any layer in glusterfs. md-cache could be useful for serving fstat from glusterfs. But apart from that I don't see any other xlator contributing much.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div> - Or theoretically the workload is ought to benefit from perf xlators, but we don't see them in our results (there are open bugs to this effect)?</div><div><br></div><div>I am asking these questions to ascertain priority on fixing perf xlators for (meta)data inconsistencies. If we offer a different solution for these workloads, the need for fixing these issues will be less.<br></div></div></div></div></blockquote><div><br></div><div>My personal opinion is that both block and fs should work correctly. i.e. caching xlators shouldn't lead to inconsistency issues. It would be better if we are in a position where we choose a workload on block vs fs based on their performance for that workload and nothing else. Block/VM usecases change the workload of the application for glusterfs, so for small file operations the kind of performance you see on block can never be achieved by glusterfs with the current architecture/design.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><span class=""><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>I feel that we have come a long way from the time the install guide was written and an update for removing the "staleness of content" might be in order there :-).</div><div><br></div><div>Regards,</div><div>Vijay</div><div><br></div><div>[4] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1058526" target="_blank">https://bugzilla.redhat.co<wbr>m/show_bug.cgi?id=1058526</a></div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
[1] <a href="http://docs.gluster.org/en/latest/Install-Guide/Overview/" rel="noreferrer" target="_blank">http://docs.gluster.org/en/lat<wbr>est/Install-Guide/Overview/</a><br>
[2] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1512691" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1512691</a><br>
[3] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1390050" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1390050</a><br>
<br>
regards,<br>
Raghavendra<br>
______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-devel</a><br>
</blockquote></span></div><br></div></div>
<br>______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-devel</a><br></blockquote></span></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="m_2228948920624986172gmail_signature" data-smartmail="gmail_signature">Raghavendra G<br></div>
</font></span></div></div>
<br>______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>