[Gluster-users] glusterd crashes on Assertion failed: rsp.op == txn_op_info.op

[Sanju Rakonde]

Thanks for the update Olaf.

You are hitting the bug, which is mentioned by Atin in this mail thread.
I'm not sure whether we can backport the fix to release-5 branch. I will
update you regarding this in early next week.

On Thu, Jun 20, 2019 at 7:30 PM Olaf Buitelaar <olaf.buitelaar at gmail.com>
wrote:

> Hi Sanju,
>
> going through the stacks i noticed that this function was in
> between; glusterd_volume_rebalance_use_rsp_dict
> So it might after all have todo something with the rebalancing logic.
> I've checked the cmd_history.log and exactly on the time of crash time
> command was executed;
> [2019-06-19 07:25:03.108360]  : volume rebalance ovirt-data status :
> SUCCESS preceding a couple of other status checks of rebalancing. The
> complete batch of 2 mins before, all reported success.
> These commands are executed by ovirt about every 2 minutes, to pull for
> the status of gluster.
> I'm sure no actual rebalancing tasks were running, also checked the last
> time that was @2019-06-08 21:13:02 and was completed successfully
> Hopefully this is additional useful info.
>
> Thanks Olaf
>
> Op do 20 jun. 2019 om 14:52 schreef Olaf Buitelaar <
> olaf.buitelaar at gmail.com>:
>
>> Hi Sanju,
>>
>> you can download the coredump here;
>> http://edgecastcdn.net/0004FA/files/core_dump.zip (around 20MB)
>>
>> Thanks Olaf
>>
>> Op do 20 jun. 2019 om 08:35 schreef Sanju Rakonde <srakonde at redhat.com>:
>>
>>> Olaf,
>>>
>>> Can you please paste complete backtrace from the core file, so that we
>>> can analyse what is wrong here.
>>>
>>> On Wed, Jun 19, 2019 at 10:31 PM Olaf Buitelaar <
>>> olaf.buitelaar at gmail.com> wrote:
>>>
>>>> Hi Atin,
>>>>
>>>> Thank you for pointing out this bug report, however no rebalancing task
>>>> was running during this event. So maybe something else is causing this?
>>>> According the report this should be fixed in gluster 6, unfortunate
>>>> ovirt doesn't seem to officially support that version, so i'm stuck on the
>>>> 5 branch for now.
>>>> Any chance this will be back ported?
>>>>
>>>> Thanks Olaf
>>>>
>>>>
>>>> Op wo 19 jun. 2019 om 17:57 schreef Atin Mukherjee <amukherj at redhat.com
>>>> >:
>>>>
>>>>> Please see - https://bugzilla.redhat.com/show_bug.cgi?id=1655827
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jun 19, 2019 at 5:52 PM Olaf Buitelaar <
>>>>> olaf.buitelaar at gmail.com> wrote:
>>>>>
>>>>>> Dear All,
>>>>>>
>>>>>> Has anybody seen this error on gluster 5.6;
>>>>>> [glusterd-rpc-ops.c:1388:__glusterd_commit_op_cbk]
>>>>>> (-->/lib64/libgfrpc.so.0(+0xec60) [0x7fbfb7801c60]
>>>>>> -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x79b7a)
>>>>>> [0x7fbfac50db7a]
>>>>>> -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x77393)
>>>>>> [0x7fbfac50b393] ) 0-: Assertion failed: rsp.op == txn_op_info.op
>>>>>>
>>>>>> checking the code;
>>>>>> https://github.com/gluster/glusterfs/blob/6fd8281ac9af58609979f660ece58c2ed1100e72/xlators/mgmt/glusterd/src/glusterd-rpc-ops.c#L1388
>>>>>>
>>>>>> doesn't seem to reveal much on what could causing this.
>>>>>>
>>>>>> It's the second time this occurs.
>>>>>>
>>>>>> Attached the full stack.
>>>>>>
>>>>>> Thanks Olaf
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Sanju
>>>
>>

-- 
Thanks,
Sanju
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190621/47c3d3f1/attachment.html>