[Gluster-users] glusterd crashes on Assertion failed: rsp.op == txn_op_info.op

[Olaf Buitelaar]

Hi Sanju,

going through the stacks i noticed that this function was in
between; glusterd_volume_rebalance_use_rsp_dict
So it might after all have todo something with the rebalancing logic.
I've checked the cmd_history.log and exactly on the time of crash time
command was executed;
[2019-06-19 07:25:03.108360]  : volume rebalance ovirt-data status :
SUCCESS preceding a couple of other status checks of rebalancing. The
complete batch of 2 mins before, all reported success.
These commands are executed by ovirt about every 2 minutes, to pull for the
status of gluster.
I'm sure no actual rebalancing tasks were running, also checked the last
time that was @2019-06-08 21:13:02 and was completed successfully
Hopefully this is additional useful info.

Thanks Olaf

Op do 20 jun. 2019 om 14:52 schreef Olaf Buitelaar <olaf.buitelaar at gmail.com
>:

> Hi Sanju,
>
> you can download the coredump here;
> http://edgecastcdn.net/0004FA/files/core_dump.zip (around 20MB)
>
> Thanks Olaf
>
> Op do 20 jun. 2019 om 08:35 schreef Sanju Rakonde <srakonde at redhat.com>:
>
>> Olaf,
>>
>> Can you please paste complete backtrace from the core file, so that we
>> can analyse what is wrong here.
>>
>> On Wed, Jun 19, 2019 at 10:31 PM Olaf Buitelaar <olaf.buitelaar at gmail.com>
>> wrote:
>>
>>> Hi Atin,
>>>
>>> Thank you for pointing out this bug report, however no rebalancing task
>>> was running during this event. So maybe something else is causing this?
>>> According the report this should be fixed in gluster 6, unfortunate
>>> ovirt doesn't seem to officially support that version, so i'm stuck on the
>>> 5 branch for now.
>>> Any chance this will be back ported?
>>>
>>> Thanks Olaf
>>>
>>>
>>> Op wo 19 jun. 2019 om 17:57 schreef Atin Mukherjee <amukherj at redhat.com
>>> >:
>>>
>>>> Please see - https://bugzilla.redhat.com/show_bug.cgi?id=1655827
>>>>
>>>>
>>>>
>>>> On Wed, Jun 19, 2019 at 5:52 PM Olaf Buitelaar <
>>>> olaf.buitelaar at gmail.com> wrote:
>>>>
>>>>> Dear All,
>>>>>
>>>>> Has anybody seen this error on gluster 5.6;
>>>>> [glusterd-rpc-ops.c:1388:__glusterd_commit_op_cbk]
>>>>> (-->/lib64/libgfrpc.so.0(+0xec60) [0x7fbfb7801c60]
>>>>> -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x79b7a)
>>>>> [0x7fbfac50db7a]
>>>>> -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x77393)
>>>>> [0x7fbfac50b393] ) 0-: Assertion failed: rsp.op == txn_op_info.op
>>>>>
>>>>> checking the code;
>>>>> https://github.com/gluster/glusterfs/blob/6fd8281ac9af58609979f660ece58c2ed1100e72/xlators/mgmt/glusterd/src/glusterd-rpc-ops.c#L1388
>>>>>
>>>>> doesn't seem to reveal much on what could causing this.
>>>>>
>>>>> It's the second time this occurs.
>>>>>
>>>>> Attached the full stack.
>>>>>
>>>>> Thanks Olaf
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>> --
>> Thanks,
>> Sanju
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190620/47d3f0aa/attachment.html>