[Gluster-devel] Glusterfs and Structured data

Thu Feb 15 04:19:28 UTC 2018

On Wed, Feb 14, 2018 at 10:31 PM, Amar Tumballi <atumball at redhat.com> wrote:

> Top posting as it is not exactly about in-consistency of perf layer.
>
> On the performance translators of Gluster, I am more interested to get
> work done on write-back caching layer, specially with using lease feature.
> Mainly because there are way too many usecases where a given directory
> would be used only by one client/application at the time.
>

Can you explain a bit more in detail? Is it the problem of cached writes
reaching bricks after a lease is revoked (but application writes were done
when there was a valid lease)?

> Regards,
> Amar
>
>
> On Wed, Feb 14, 2018 at 8:17 AM, Raghavendra G <raghavendra at gluster.com>
> wrote:
>
>> I've started marking "whiteboard" of bugs in this class with tag
>> "GLUSTERFS_METADATA_INCONSISTENCY". Please add the tag to any bugs which
>> you deem to fit in.
>>
>> On Fri, Feb 9, 2018 at 4:30 PM, Raghavendra Gowdappa <rgowdapp at redhat.com
>> > wrote:
>>
>>>
>>>
>>> ----- Original Message -----
>>> > From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
>>> > To: "Raghavendra G" <raghavendra at gluster.com>
>>> > Cc: "Gluster Devel" <gluster-devel at gluster.org>
>>> > Sent: Friday, February 9, 2018 2:30:59 PM
>>> > Subject: Re: [Gluster-devel] Glusterfs and Structured data
>>> >
>>> >
>>> >
>>> > On Thu, Feb 8, 2018 at 12:05 PM, Raghavendra G <
>>> raghavendra at gluster.com >
>>> > wrote:
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Tue, Feb 6, 2018 at 8:15 PM, Vijay Bellur < vbellur at redhat.com >
>>> wrote:
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Sun, Feb 4, 2018 at 3:39 AM, Raghavendra Gowdappa <
>>> rgowdapp at redhat.com >
>>> > wrote:
>>> >
>>> >
>>> > All,
>>> >
>>> > One of our users pointed out to the documentation that glusterfs is
>>> not good
>>> > for storing "Structured data" [1], while discussing an issue [2].
>>> >
>>> >
>>> > As far as I remember, the content around structured data in the
>>> Install Guide
>>> > is from a FAQ that was being circulated in Gluster, Inc. indicating the
>>> > startup's market positioning. Most of that was based on not wanting to
>>> get
>>> > into performance based comparisons of storage systems that are
>>> frequently
>>> > seen in the structured data space.
>>> >
>>> >
>>> > Does any of you have more context on the feasibility of storing
>>> "structured
>>> > data" on Glusterfs? Is one of the reasons for such a suggestion
>>> "staleness
>>> > of metadata" as encountered in bugs like [3]?
>>> >
>>> >
>>> > There are challenges that distributed storage systems face when
>>> exposed to
>>> > applications that were written for a local filesystem interface. We
>>> have
>>> > encountered problems with applications like tar [4] that are not in the
>>> > realm of "Structured data". If we look at the common theme across all
>>> these
>>> > problems, it is related to metadata & read after write consistency
>>> issues
>>> > with the default translator stack that gets exposed on the client side.
>>> > While the default stack is optimal for other scenarios, it does seem
>>> that a
>>> > category of applications needing strict metadata consistency is not
>>> well
>>> > served by that. We have observed that disabling a few performance
>>> > translators and tuning cache timeouts for VFS/FUSE have helped to
>>> overcome
>>> > some of them. The WIP effort on timestamp consistency across the
>>> translator
>>> > stack, patches that have been merged as a result of the bugs that you
>>> > mention & other fixes for outstanding issues should certainly help in
>>> > catering to these workloads better with the file interface.
>>> >
>>> > There are deployments that I have come across where glusterfs is used
>>> for
>>> > storing structured data. gluster-block & qemu-libgfapi overcome the
>>> metadata
>>> > consistency problem by exposing a file as a block device & by
>>> disabling most
>>> > of the performance translators in the default stack. Workloads that
>>> have
>>> > been deemed problematic with the file interface for the reasons alluded
>>> > above, function well with the block interface.
>>> >
>>> > I agree that gluster-block due to its usage of a subset of glusterfs
>>> fops
>>> > (mostly reads/writes I guess), runs into less number of consistency
>>> issues.
>>> > However, as you've mentioned we seem to disable perf xlator stack in
>>> our
>>> > tests/use-cases till now. Note that perf xlator stack is one of worst
>>> > offenders as far as the metadata consistency is concerned (relatively
>>> less
>>> > scenarios of data inconsistency). So, I wonder,
>>> > * what would be the scenario if we enable perf xlator stack for
>>> > gluster-block?
>>> > * Is performance on gluster-block satisfactory so that we don't need
>>> these
>>> > xlators?
>>> > - Or is it that these xlators are not useful for the workload usually
>>> run on
>>> > gluster-block (For random read/write workload, read/write caching
>>> xlators
>>> > offer less or no advantage)?
>>> >
>>> > Yes. They are not useful. Block/VM files are opened with O_DIRECT, so
>>> we
>>> > don't enable caching at any layer in glusterfs. md-cache could be
>>> useful for
>>> > serving fstat from glusterfs. But apart from that I don't see any other
>>> > xlator contributing much.
>>> >
>>> >
>>> >
>>> > - Or theoretically the workload is ought to benefit from perf xlators,
>>> but we
>>> > don't see them in our results (there are open bugs to this effect)?
>>> >
>>> > I am asking these questions to ascertain priority on fixing perf
>>> xlators for
>>> > (meta)data inconsistencies. If we offer a different solution for these
>>> > workloads, the need for fixing these issues will be less.
>>> >
>>> > My personal opinion is that both block and fs should work correctly.
>>> i.e.
>>> > caching xlators shouldn't lead to inconsistency issues.
>>>
>>> +1. That's my personal opinion too. We'll try to fix these issues.
>>> However, we need to qualify the fixes. It would be helpful if community can
>>> help here. We'll let community know when the fixes are in.
>>>
>>> > It would be better
>>> > if we are in a position where we choose a workload on block vs fs
>>> based on
>>> > their performance for that workload and nothing else. Block/VM usecases
>>> > change the workload of the application for glusterfs, so for small file
>>> > operations the kind of performance you see on block can never be
>>> achieved by
>>> > glusterfs with the current architecture/design.
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > I feel that we have come a long way from the time the install guide was
>>> > written and an update for removing the "staleness of content" might be
>>> in
>>> > order there :-).
>>> >
>>> > Regards,
>>> > Vijay
>>> >
>>> > [4] https://bugzilla.redhat.com/show_bug.cgi?id=1058526
>>> >
>>> >
>>> >
>>> > [1] http://docs.gluster.org/en/latest/Install-Guide/Overview/
>>> > [2] https://bugzilla.redhat.com/show_bug.cgi?id=1512691
>>> > [3] https://bugzilla.redhat.com/show_bug.cgi?id=1390050
>>> >
>>> > regards,
>>> > Raghavendra
>>> > _______________________________________________
>>> > Gluster-devel mailing list
>>> > Gluster-devel at gluster.org
>>> > http://lists.gluster.org/mailman/listinfo/gluster-devel
>>> >
>>> >
>>> > _______________________________________________
>>> > Gluster-devel mailing list
>>> > Gluster-devel at gluster.org
>>> > http://lists.gluster.org/mailman/listinfo/gluster-devel
>>> >
>>> >
>>> >
>>> > --
>>> > Raghavendra G
>>> >
>>> > _______________________________________________
>>> > Gluster-devel mailing list
>>> > Gluster-devel at gluster.org
>>> > http://lists.gluster.org/mailman/listinfo/gluster-devel
>>> >
>>> >
>>> >
>>> > --
>>> > Pranith
>>> >
>>> > _______________________________________________
>>> > Gluster-devel mailing list
>>> > Gluster-devel at gluster.org
>>> > http://lists.gluster.org/mailman/listinfo/gluster-devel
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>>
>>
>>
>>
>> --
>> Raghavendra G
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>
>
>
> --
> Amar Tumballi (amarts)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20180215/9228389a/attachment.html>