[Gluster-users] Elasticsearch facing CorruptIndexException exception with GlusterFs 3.10.1

Kaleb S. KEITHLEY kkeithle at redhat.com
Mon May 8 18:23:53 UTC 2017


On 05/08/2017 05:32 AM, Pranith Kumar Karampuri wrote:
>           We released gluster-block v0.2 just this Friday for which RHEL 
> packages are yet to be built.
> 
> +Kaleb,
>       Could you help with this please?

Of course this doesn't build in EPEL, where glusterfs-api-devel is not 
available. EPEL does not have packages that exist in RHEL; RHEL has 
glusterfs, but not glusterfs-api-devel, which is in a channel other than 
the base for reasons I don't understand.

There's no way to do scratch builds in CentOS's Storage SIG. We can add 
it to the Storage SIG after it has been accepted into Fedora. I asked 
previously, but didn't get an answer, about the status is of getting 
this into Fedora.

The best option ATM is to do a mockchain build — which anyone can do — 
using the glusterfs src.rpm, e.g. from the Storage SIG, and the 
gluster-block src.rpm.

> 
> Prasanna,
>       Could you let Abhijit know the rpm versions for tcmu-runner and 
> other packages so that this feature can be used?
> 
> 
> On Mon, May 8, 2017 at 2:49 PM, Pranith Kumar Karampuri 
> <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote:
> 
>     Wait, wait we are discussing this issue only. Expect a reply in some
>     time :-)
> 
>     On Mon, May 8, 2017 at 2:19 PM, Abhijit Paul
>     <er.abhijitpaul at gmail.com <mailto:er.abhijitpaul at gmail.com>> wrote:
> 
>         poking for previous mail reply
> 
>         On Sun, May 7, 2017 at 1:06 AM, Abhijit Paul
>         <er.abhijitpaul at gmail.com <mailto:er.abhijitpaul at gmail.com>> wrote:
> 
>             https://pkalever.wordpress.com/2017/03/14/elasticsearch-with-gluster-block/
>             <https://pkalever.wordpress.com/2017/03/14/elasticsearch-with-gluster-block/>
>             here used tested environment is Fedora ,
>             but i am using RHEL based Oracle linux so does gluster-block
>             compatible with RHEL as well? What i needs to change & make
>             it work?
> 
>             On Fri, May 5, 2017 at 5:42 PM, Pranith Kumar Karampuri
>             <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote:
> 
> 
> 
>                 On Fri, May 5, 2017 at 5:40 PM, Pranith Kumar Karampuri
>                 <pkarampu at redhat.com <mailto:pkarampu at redhat.com>> wrote:
> 
> 
> 
>                     On Fri, May 5, 2017 at 5:36 PM, Abhijit Paul
>                     <er.abhijitpaul at gmail.com
>                     <mailto:er.abhijitpaul at gmail.com>> wrote:
> 
>                         So should i start using gluster-block with
>                         elasticsearch in kubernetes environment?
> 
>                         My expectation from gluster-block is, it should
>                         not CorruptIndex of elasticsearch...and issue
>                         facing in previous mails.
> 
>                         Please let me know whether should i
>                         processed with above mentioned combination.
> 
> 
>                     We are still in the process of fixing the failure
>                     scenarios of tcmu-runner dying and failingover in
>                     the multipath scenarios.
> 
> 
>                 Prasanna did test that elasticsearch itself worked fine
>                 in gluster-block environment when all the machines are
>                 up etc i.e. success path. We are doing failure path
>                 testing and fixing things at the moment.
> 
> 
>                         On Fri, May 5, 2017 at 5:06 PM, Pranith Kumar
>                         Karampuri <pkarampu at redhat.com
>                         <mailto:pkarampu at redhat.com>> wrote:
> 
>                             Abhijit we just started making the efforts
>                             to get all of this stable.
> 
>                             On Fri, May 5, 2017 at 4:45 PM, Abhijit Paul
>                             <er.abhijitpaul at gmail.com
>                             <mailto:er.abhijitpaul at gmail.com>> wrote:
> 
>                                 I yet to try gluster-block with
>                                 elasticsearch...but carious to know does
>                                 this combination plays well in
>                                 kubernetes environment?
> 
>                                 On Fri, May 5, 2017 at 12:14 PM, Abhijit
>                                 Paul <er.abhijitpaul at gmail.com
>                                 <mailto:er.abhijitpaul at gmail.com>> wrote:
> 
>                                     thanks Krutika for the alternative.
> 
>                                     @*Prasanna @**Pranith*
>                                     I was going thorough the mentioned
>                                     blog post and saw that used tested
>                                     environment was Fedora ,
>                                     but i am using RHEL based Oracle
>                                     linux so does gluster-block
>                                     compatible with RHEL as well?
> 
>                                     On Fri, May 5, 2017 at 12:03 PM,
>                                     Krutika Dhananjay
>                                     <kdhananj at redhat.com
>                                     <mailto:kdhananj at redhat.com>> wrote:
> 
>                                         Yeah, there are a couple of
>                                         cache consistency issues with
>                                         performance translators that are
>                                         causing these exceptions.
>                                         Some of them were fixed by
>                                         3.10.1. Some still remain.
> 
>                                         Alternatively you can give
>                                         gluster-block + elasticsearch a
>                                         try, which doesn't require
>                                         solving all these caching issues.
>                                         Here's a blog post on the same -
>                                         https://pkalever.wordpress.com/2017/03/14/elasticsearch-with-gluster-block/
>                                         <https://pkalever.wordpress.com/2017/03/14/elasticsearch-with-gluster-block/>
> 
>                                         Adding Prasanna and Pranith who
>                                         worked on this, in case you need
>                                         more info on this.
> 
>                                         -Krutika
> 
>                                         On Fri, May 5, 2017 at 12:15 AM,
>                                         Abhijit Paul
>                                         <er.abhijitpaul at gmail.com
>                                         <mailto:er.abhijitpaul at gmail.com>>
>                                         wrote:
> 
>                                             Thanks for the reply, i will
>                                             try it out but i am also
>                                             facing one more issue "i.e.
>                                             replicated volumes returning
>                                             different timestamps"
>                                             so is this because of Bug
>                                             1426548 - Openshift Logging
>                                             ElasticSearch FSLocks when
>                                             using GlusterFS storage
>                                             backend
>                                             <https://bugzilla.redhat.com/show_bug.cgi?id=1426548> ?
> 
>                                             *FYI i am using glusterfs
>                                             3.10.1 tar.gz*
>                                             *
>                                             *
>                                             Regards,
>                                             Abhijit
> 
> 
> 
>                                             On Thu, May 4, 2017 at 10:58
>                                             PM, Amar Tumballi
>                                             <atumball at redhat.com
>                                             <mailto:atumball at redhat.com>> wrote:
> 
> 
> 
>                                                 On Thu, May 4, 2017 at
>                                                 10:41 PM, Abhijit Paul
>                                                 <er.abhijitpaul at gmail.com <mailto:er.abhijitpaul at gmail.com>>
>                                                 wrote:
> 
>                                                     Since i am new to
>                                                     gluster, can please
>                                                     provide how to turn
>                                                     off/disable "perf
>                                                     xlator options"?
> 
> 
>                                                 $ gluster volume set
>                                                 <volname>
>                                                 performance.stat-prefetch off
>                                                 $ gluster volume set
>                                                 <volname>
>                                                 performance.read-ahead off
>                                                 $ gluster volume set
>                                                 <volname>
>                                                 performance.write-behind off
>                                                 $ gluster volume set
>                                                 <volname>
>                                                 performance.io-cache off
>                                                 $ gluster volume set
>                                                 <volname>
>                                                 performance.quick-read off
> 
> 
>                                                 Regards,
>                                                 Amar
> 
> 
>                                                         On Wed, May 3,
>                                                         2017 at 8:51 PM,
>                                                         Atin Mukherjee
>                                                         <amukherj at redhat.com
>                                                         <mailto:amukherj at redhat.com>>
>                                                         wrote:
> 
>                                                             I think
>                                                             there is
>                                                             still some
>                                                             pending
>                                                             stuffs in
>                                                             some of the
>                                                             gluster perf
>                                                             xlators to
>                                                             make that
>                                                             work
>                                                             complete.
>                                                             Cced the
>                                                             relevant
>                                                             folks for
>                                                             more
>                                                             information.
>                                                             Can you
>                                                             please turn
>                                                             off all the
>                                                             perf xlator
>                                                             options as a
>                                                             work around
>                                                             to move forward?
> 
>                                                             On Wed, May
>                                                             3, 2017 at
>                                                             8:04 PM,
>                                                             Abhijit Paul
>                                                             <er.abhijitpaul at gmail.com
>                                                             <mailto:er.abhijitpaul at gmail.com>>
>                                                             wrote:
> 
>                                                                 Dear folks,
> 
>                                                                 I setup
>                                                                 Glusterfs(3.10.1)
>                                                                 NFS type
>                                                                 as
>                                                                 persistence
>                                                                 volume
>                                                                 for
>                                                                 Elasticsearch(5.1.2)
>                                                                 but
>                                                                 currently facing
>                                                                 issue
>                                                                 with
>                                                                 *"CorruptIndexException"
>                                                                 *with
>                                                                 Elasticseach
>                                                                 logs and
>                                                                 due to
>                                                                 that
>                                                                 index
>                                                                 health
>                                                                 trued
>                                                                 RED in
>                                                                 Elasticsearch.
> 
> 
>                                                                 Later
>                                                                 found
>                                                                 that
>                                                                 there
>                                                                 was an
>                                                                 issue
>                                                                 with
>                                                                 gluster
>                                                                 < 3.10
>                                                                 (https://bugzilla.redhat.com/show_bug.cgi?id=1390050
>                                                                 <https://bugzilla.redhat.com/show_bug.cgi?id=1390050>)
>                                                                 but even
>                                                                 after
>                                                                 *upgrading
>                                                                 to
>                                                                 3.10.1
>                                                                 issue is
>                                                                 still
>                                                                 there.*
>                                                                 *
>                                                                 *
>                                                                 *So
>                                                                 curios
>                                                                 to know
>                                                                 what
>                                                                 would be
>                                                                 the root
>                                                                 cause to
>                                                                 fix this
>                                                                 issue.*
> 
>                                                                 Regards,
>                                                                 Abhijit
> 
>                                                                 _______________________________________________
>                                                                 Gluster-users
>                                                                 mailing list
>                                                                 Gluster-users at gluster.org
>                                                                 <mailto:Gluster-users at gluster.org>
>                                                                 http://lists.gluster.org/mailman/listinfo/gluster-users
>                                                                 <http://lists.gluster.org/mailman/listinfo/gluster-users>
> 
> 
> 
> 
> 
>                                                     _______________________________________________
>                                                     Gluster-users
>                                                     mailing list
>                                                     Gluster-users at gluster.org
>                                                     <mailto:Gluster-users at gluster.org>
>                                                     http://lists.gluster.org/mailman/listinfo/gluster-users
>                                                     <http://lists.gluster.org/mailman/listinfo/gluster-users>
> 
> 
> 
> 
>                                                 -- 
>                                                 Amar Tumballi (amarts)
> 
> 
> 
>                                             _______________________________________________
>                                             Gluster-users mailing list
>                                             Gluster-users at gluster.org
>                                             <mailto:Gluster-users at gluster.org>
>                                             http://lists.gluster.org/mailman/listinfo/gluster-users
>                                             <http://lists.gluster.org/mailman/listinfo/gluster-users>
> 
> 
> 
> 
> 
> 
> 
>                             -- 
>                             Pranith
> 
> 
> 
> 
> 
>                     -- 
>                     Pranith
> 
> 
> 
> 
>                 -- 
>                 Pranith
> 
> 
> 
> 
> 
> 
>     -- 
>     Pranith
> 
> 
> 
> 
> -- 
> Pranith



More information about the Gluster-users mailing list