[Gluster-users] Elasticsearch facing CorruptIndexException exception with GlusterFs 3.10.1

Mon May 8 17:46:56 UTC 2017

*@Prasanna & @Pranith & @Keithley *Thank you very much for this update, let
me try out this with build frm src and then use with Elasticsearch in
kubernetes environment, i will let you know the update.

My hole intention to use this gluster-block solution is to *avoid
Elasticsearch index health turn RED issue due to CorruptIndex* issue by
using GlusterFS & FUSE, *on this regard if any further pointer or forwards
are there will relay appreciate.*

Regards,
Abhijit

On Mon, May 8, 2017 at 5:26 PM, Prasanna Kalever <pkalever at redhat.com>
wrote:

> On Mon, May 8, 2017 at 3:02 PM, Pranith Kumar Karampuri
> <pkarampu at redhat.com> wrote:
> > Abhijit,
> >          We released gluster-block v0.2 just this Friday for which RHEL
> > packages are yet to be built.
> >
> > +Kaleb,
> >      Could you help with this please?
> >
> > Prasanna,
> >      Could you let Abhijit know the rpm versions for tcmu-runner and
> other
> > packages so that this feature can be used?
>
> Hi Abhijit,
>
> We will soon try to help you with the centos packages, but for time
> being you can try compile from sources, dependencies include
>
> gluster-block [1]: use v0.2, INSTALL guide [2]
> tcmu-runner [3] : >= 1.0.4 ( you can use current head for time being )
> targetcli [4]: >= 2.1.fb43 (try yum install, you should get this in
> RHEL, this pulls the required deps)
>
> [1] https://github.com/gluster/gluster-block
> [2] https://github.com/gluster/gluster-block/blob/master/INSTALL
> [3] https://github.com/open-iscsi/tcmu-runner
> [4] https://github.com/open-iscsi/targetcli-fb
>
>
> Note: gluster-block(server side) will be supported soon on RHEL 7.3 and +
> only.
>
> Cheers!
> --
> prasanna
>
>
>
>
> >
> >
> > On Mon, May 8, 2017 at 2:49 PM, Pranith Kumar Karampuri
> > <pkarampu at redhat.com> wrote:
> >>
> >> Wait, wait we are discussing this issue only. Expect a reply in some
> time
> >> :-)
> >>
> >> On Mon, May 8, 2017 at 2:19 PM, Abhijit Paul <er.abhijitpaul at gmail.com>
> >> wrote:
> >>>
> >>> poking for previous mail reply
> >>>
> >>> On Sun, May 7, 2017 at 1:06 AM, Abhijit Paul <er.abhijitpaul at gmail.com
> >
> >>> wrote:
> >>>>
> >>>>
> >>>> https://pkalever.wordpress.com/2017/03/14/elasticsearch-
> with-gluster-block/
> >>>> here used tested environment is Fedora ,
> >>>> but i am using RHEL based Oracle linux so does gluster-block
> compatible
> >>>> with RHEL as well? What i needs to change & make it work?
> >>>>
> >>>> On Fri, May 5, 2017 at 5:42 PM, Pranith Kumar Karampuri
> >>>> <pkarampu at redhat.com> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Fri, May 5, 2017 at 5:40 PM, Pranith Kumar Karampuri
> >>>>> <pkarampu at redhat.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Fri, May 5, 2017 at 5:36 PM, Abhijit Paul
> >>>>>> <er.abhijitpaul at gmail.com> wrote:
> >>>>>>>
> >>>>>>> So should i start using gluster-block with elasticsearch in
> >>>>>>> kubernetes environment?
> >>>>>>>
> >>>>>>> My expectation from gluster-block is, it should not CorruptIndex of
> >>>>>>> elasticsearch...and issue facing in previous mails.
> >>>>>>>
> >>>>>>> Please let me know whether should i processed with above mentioned
> >>>>>>> combination.
> >>>>>>
> >>>>>>
> >>>>>> We are still in the process of fixing the failure scenarios of
> >>>>>> tcmu-runner dying and failingover in the multipath scenarios.
> >>>>>
> >>>>>
> >>>>> Prasanna did test that elasticsearch itself worked fine in
> >>>>> gluster-block environment when all the machines are up etc i.e.
> success
> >>>>> path. We are doing failure path testing and fixing things at the
> moment.
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, May 5, 2017 at 5:06 PM, Pranith Kumar Karampuri
> >>>>>>> <pkarampu at redhat.com> wrote:
> >>>>>>>>
> >>>>>>>> Abhijit we just started making the efforts to get all of this
> >>>>>>>> stable.
> >>>>>>>>
> >>>>>>>> On Fri, May 5, 2017 at 4:45 PM, Abhijit Paul
> >>>>>>>> <er.abhijitpaul at gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> I yet to try gluster-block with elasticsearch...but carious to
> know
> >>>>>>>>> does this combination plays well in kubernetes environment?
> >>>>>>>>>
> >>>>>>>>> On Fri, May 5, 2017 at 12:14 PM, Abhijit Paul
> >>>>>>>>> <er.abhijitpaul at gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>> thanks Krutika for the alternative.
> >>>>>>>>>>
> >>>>>>>>>> @Prasanna @Pranith
> >>>>>>>>>> I was going thorough the mentioned blog post and saw that used
> >>>>>>>>>> tested environment was Fedora ,
> >>>>>>>>>> but i am using RHEL based Oracle linux so does gluster-block
> >>>>>>>>>> compatible with RHEL as well?
> >>>>>>>>>>
> >>>>>>>>>> On Fri, May 5, 2017 at 12:03 PM, Krutika Dhananjay
> >>>>>>>>>> <kdhananj at redhat.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Yeah, there are a couple of cache consistency issues with
> >>>>>>>>>>> performance translators that are causing these exceptions.
> >>>>>>>>>>> Some of them were fixed by 3.10.1. Some still remain.
> >>>>>>>>>>>
> >>>>>>>>>>> Alternatively you can give gluster-block + elasticsearch a try,
> >>>>>>>>>>> which doesn't require solving all these caching issues.
> >>>>>>>>>>> Here's a blog post on the same -
> >>>>>>>>>>> https://pkalever.wordpress.com/2017/03/14/elasticsearch-
> with-gluster-block/
> >>>>>>>>>>>
> >>>>>>>>>>> Adding Prasanna and Pranith who worked on this, in case you
> need
> >>>>>>>>>>> more info on this.
> >>>>>>>>>>>
> >>>>>>>>>>> -Krutika
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, May 5, 2017 at 12:15 AM, Abhijit Paul
> >>>>>>>>>>> <er.abhijitpaul at gmail.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the reply, i will try it out but i am also facing
> one
> >>>>>>>>>>>> more issue "i.e. replicated volumes returning different
> timestamps"
> >>>>>>>>>>>> so is this because of Bug 1426548 - Openshift Logging
> >>>>>>>>>>>> ElasticSearch FSLocks when using GlusterFS storage backend ?
> >>>>>>>>>>>>
> >>>>>>>>>>>> FYI i am using glusterfs 3.10.1 tar.gz
> >>>>>>>>>>>>
> >>>>>>>>>>>> Regards,
> >>>>>>>>>>>> Abhijit
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, May 4, 2017 at 10:58 PM, Amar Tumballi
> >>>>>>>>>>>> <atumball at redhat.com> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, May 4, 2017 at 10:41 PM, Abhijit Paul
> >>>>>>>>>>>>> <er.abhijitpaul at gmail.com> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Since i am new to gluster, can please provide how to turn
> >>>>>>>>>>>>>> off/disable "perf xlator options"?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> $ gluster volume set <volname> performance.stat-prefetch off
> >>>>>>>>>>>>> $ gluster volume set <volname> performance.read-ahead off
> >>>>>>>>>>>>> $ gluster volume set <volname> performance.write-behind off
> >>>>>>>>>>>>> $ gluster volume set <volname> performance.io-cache off
> >>>>>>>>>>>>> $ gluster volume set <volname> performance.quick-read off
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>> Amar
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, May 3, 2017 at 8:51 PM, Atin Mukherjee
> >>>>>>>>>>>>>>> <amukherj at redhat.com> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I think there is still some pending stuffs in some of the
> >>>>>>>>>>>>>>>> gluster perf xlators to make that work complete. Cced the
> relevant folks for
> >>>>>>>>>>>>>>>> more information. Can you please turn off all the perf
> xlator options as a
> >>>>>>>>>>>>>>>> work around to move forward?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Wed, May 3, 2017 at 8:04 PM, Abhijit Paul
> >>>>>>>>>>>>>>>> <er.abhijitpaul at gmail.com> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Dear folks,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I setup Glusterfs(3.10.1) NFS type as persistence volume
> >>>>>>>>>>>>>>>>> for Elasticsearch(5.1.2) but currently facing issue with
> >>>>>>>>>>>>>>>>> "CorruptIndexException" with Elasticseach logs and due
> to that index health
> >>>>>>>>>>>>>>>>> trued RED in Elasticsearch.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Later found that there was an issue with gluster < 3.10
> >>>>>>>>>>>>>>>>> (https://bugzilla.redhat.com/show_bug.cgi?id=1390050)
> but even after
> >>>>>>>>>>>>>>>>> upgrading to 3.10.1 issue is still there.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> So curios to know what would be the root cause to fix
> this
> >>>>>>>>>>>>>>>>> issue.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>> Abhijit
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>>>>> Gluster-users mailing list
> >>>>>>>>>>>>>>>>> Gluster-users at gluster.org
> >>>>>>>>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>> Gluster-users mailing list
> >>>>>>>>>>>>>> Gluster-users at gluster.org
> >>>>>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> Amar Tumballi (amarts)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>> Gluster-users mailing list
> >>>>>>>>>>>> Gluster-users at gluster.org
> >>>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Pranith
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Pranith
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Pranith
> >>>>
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> Pranith
> >
> >
> >
> >
> > --
> > Pranith
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170508/990be745/attachment.html>