[Gluster-users] Elasticsearch facing CorruptIndexException exception with GlusterFs 3.10.1

Mon May 8 11:56:18 UTC 2017

On Mon, May 8, 2017 at 3:02 PM, Pranith Kumar Karampuri
<pkarampu at redhat.com> wrote:
> Abhijit,
>          We released gluster-block v0.2 just this Friday for which RHEL
> packages are yet to be built.
>
> +Kaleb,
>      Could you help with this please?
>
> Prasanna,
>      Could you let Abhijit know the rpm versions for tcmu-runner and other
> packages so that this feature can be used?

Hi Abhijit,

We will soon try to help you with the centos packages, but for time
being you can try compile from sources, dependencies include

gluster-block [1]: use v0.2, INSTALL guide [2]
tcmu-runner [3] : >= 1.0.4 ( you can use current head for time being )
targetcli [4]: >= 2.1.fb43 (try yum install, you should get this in
RHEL, this pulls the required deps)

[1] https://github.com/gluster/gluster-block
[2] https://github.com/gluster/gluster-block/blob/master/INSTALL
[3] https://github.com/open-iscsi/tcmu-runner
[4] https://github.com/open-iscsi/targetcli-fb

Note: gluster-block(server side) will be supported soon on RHEL 7.3 and + only.

Cheers!
--
prasanna

>
>
> On Mon, May 8, 2017 at 2:49 PM, Pranith Kumar Karampuri
> <pkarampu at redhat.com> wrote:
>>
>> Wait, wait we are discussing this issue only. Expect a reply in some time
>> :-)
>>
>> On Mon, May 8, 2017 at 2:19 PM, Abhijit Paul <er.abhijitpaul at gmail.com>
>> wrote:
>>>
>>> poking for previous mail reply
>>>
>>> On Sun, May 7, 2017 at 1:06 AM, Abhijit Paul <er.abhijitpaul at gmail.com>
>>> wrote:
>>>>
>>>>
>>>> https://pkalever.wordpress.com/2017/03/14/elasticsearch-with-gluster-block/
>>>> here used tested environment is Fedora ,
>>>> but i am using RHEL based Oracle linux so does gluster-block compatible
>>>> with RHEL as well? What i needs to change & make it work?
>>>>
>>>> On Fri, May 5, 2017 at 5:42 PM, Pranith Kumar Karampuri
>>>> <pkarampu at redhat.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Fri, May 5, 2017 at 5:40 PM, Pranith Kumar Karampuri
>>>>> <pkarampu at redhat.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, May 5, 2017 at 5:36 PM, Abhijit Paul
>>>>>> <er.abhijitpaul at gmail.com> wrote:
>>>>>>>
>>>>>>> So should i start using gluster-block with elasticsearch in
>>>>>>> kubernetes environment?
>>>>>>>
>>>>>>> My expectation from gluster-block is, it should not CorruptIndex of
>>>>>>> elasticsearch...and issue facing in previous mails.
>>>>>>>
>>>>>>> Please let me know whether should i processed with above mentioned
>>>>>>> combination.
>>>>>>
>>>>>>
>>>>>> We are still in the process of fixing the failure scenarios of
>>>>>> tcmu-runner dying and failingover in the multipath scenarios.
>>>>>
>>>>>
>>>>> Prasanna did test that elasticsearch itself worked fine in
>>>>> gluster-block environment when all the machines are up etc i.e. success
>>>>> path. We are doing failure path testing and fixing things at the moment.
>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, May 5, 2017 at 5:06 PM, Pranith Kumar Karampuri
>>>>>>> <pkarampu at redhat.com> wrote:
>>>>>>>>
>>>>>>>> Abhijit we just started making the efforts to get all of this
>>>>>>>> stable.
>>>>>>>>
>>>>>>>> On Fri, May 5, 2017 at 4:45 PM, Abhijit Paul
>>>>>>>> <er.abhijitpaul at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> I yet to try gluster-block with elasticsearch...but carious to know
>>>>>>>>> does this combination plays well in kubernetes environment?
>>>>>>>>>
>>>>>>>>> On Fri, May 5, 2017 at 12:14 PM, Abhijit Paul
>>>>>>>>> <er.abhijitpaul at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> thanks Krutika for the alternative.
>>>>>>>>>>
>>>>>>>>>> @Prasanna @Pranith
>>>>>>>>>> I was going thorough the mentioned blog post and saw that used
>>>>>>>>>> tested environment was Fedora ,
>>>>>>>>>> but i am using RHEL based Oracle linux so does gluster-block
>>>>>>>>>> compatible with RHEL as well?
>>>>>>>>>>
>>>>>>>>>> On Fri, May 5, 2017 at 12:03 PM, Krutika Dhananjay
>>>>>>>>>> <kdhananj at redhat.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Yeah, there are a couple of cache consistency issues with
>>>>>>>>>>> performance translators that are causing these exceptions.
>>>>>>>>>>> Some of them were fixed by 3.10.1. Some still remain.
>>>>>>>>>>>
>>>>>>>>>>> Alternatively you can give gluster-block + elasticsearch a try,
>>>>>>>>>>> which doesn't require solving all these caching issues.
>>>>>>>>>>> Here's a blog post on the same -
>>>>>>>>>>> https://pkalever.wordpress.com/2017/03/14/elasticsearch-with-gluster-block/
>>>>>>>>>>>
>>>>>>>>>>> Adding Prasanna and Pranith who worked on this, in case you need
>>>>>>>>>>> more info on this.
>>>>>>>>>>>
>>>>>>>>>>> -Krutika
>>>>>>>>>>>
>>>>>>>>>>> On Fri, May 5, 2017 at 12:15 AM, Abhijit Paul
>>>>>>>>>>> <er.abhijitpaul at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the reply, i will try it out but i am also facing one
>>>>>>>>>>>> more issue "i.e. replicated volumes returning different timestamps"
>>>>>>>>>>>> so is this because of Bug 1426548 - Openshift Logging
>>>>>>>>>>>> ElasticSearch FSLocks when using GlusterFS storage backend ?
>>>>>>>>>>>>
>>>>>>>>>>>> FYI i am using glusterfs 3.10.1 tar.gz
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Abhijit
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, May 4, 2017 at 10:58 PM, Amar Tumballi
>>>>>>>>>>>> <atumball at redhat.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, May 4, 2017 at 10:41 PM, Abhijit Paul
>>>>>>>>>>>>> <er.abhijitpaul at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Since i am new to gluster, can please provide how to turn
>>>>>>>>>>>>>> off/disable "perf xlator options"?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> $ gluster volume set <volname> performance.stat-prefetch off
>>>>>>>>>>>>> $ gluster volume set <volname> performance.read-ahead off
>>>>>>>>>>>>> $ gluster volume set <volname> performance.write-behind off
>>>>>>>>>>>>> $ gluster volume set <volname> performance.io-cache off
>>>>>>>>>>>>> $ gluster volume set <volname> performance.quick-read off
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Amar
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, May 3, 2017 at 8:51 PM, Atin Mukherjee
>>>>>>>>>>>>>>> <amukherj at redhat.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think there is still some pending stuffs in some of the
>>>>>>>>>>>>>>>> gluster perf xlators to make that work complete. Cced the relevant folks for
>>>>>>>>>>>>>>>> more information. Can you please turn off all the perf xlator options as a
>>>>>>>>>>>>>>>> work around to move forward?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, May 3, 2017 at 8:04 PM, Abhijit Paul
>>>>>>>>>>>>>>>> <er.abhijitpaul at gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Dear folks,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I setup Glusterfs(3.10.1) NFS type as persistence volume
>>>>>>>>>>>>>>>>> for Elasticsearch(5.1.2) but currently facing issue with
>>>>>>>>>>>>>>>>> "CorruptIndexException" with Elasticseach logs and due to that index health
>>>>>>>>>>>>>>>>> trued RED in Elasticsearch.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Later found that there was an issue with gluster < 3.10
>>>>>>>>>>>>>>>>> (https://bugzilla.redhat.com/show_bug.cgi?id=1390050) but even after
>>>>>>>>>>>>>>>>> upgrading to 3.10.1 issue is still there.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> So curios to know what would be the root cause to fix this
>>>>>>>>>>>>>>>>> issue.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> Abhijit
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Amar Tumballi (amarts)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Pranith
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Pranith
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Pranith
>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Pranith
>
>
>
>
> --
> Pranith