<div dir="ltr">Thanks for the update Olaf.<div><br></div><div>You are hitting the bug, which is mentioned by Atin in this mail thread. I'm not sure whether we can backport the fix to release-5 branch. I will update you regarding this in early next week.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 20, 2019 at 7:30 PM Olaf Buitelaar <<a href="mailto:olaf.buitelaar@gmail.com">olaf.buitelaar@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Sanju,<div><br></div><div>going through the stacks i noticed that this function was in between; glusterd_volume_rebalance_use_rsp_dict</div><div>So it might after all have todo something with the rebalancing logic.</div><div>I've checked the cmd_history.log and exactly on the time of crash time command was executed;</div><div>[2019-06-19 07:25:03.108360] : volume rebalance ovirt-data status : SUCCESS preceding a couple of other status checks of rebalancing. The complete batch of 2 mins before, all reported success.<br></div><div>These commands are executed by ovirt about every 2 minutes, to pull for the status of gluster.</div><div>I'm sure no actual rebalancing tasks were running, also checked the last time that was @2019-06-08 21:13:02 and was completed successfully</div><div>Hopefully this is additional useful info.</div><div><br></div><div>Thanks Olaf</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Op do 20 jun. 2019 om 14:52 schreef Olaf Buitelaar <<a href="mailto:olaf.buitelaar@gmail.com" target="_blank">olaf.buitelaar@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Sanju,<div><br></div><div>you can download the coredump here; <a href="http://edgecastcdn.net/0004FA/files/core_dump.zip" target="_blank">http://edgecastcdn.net/0004FA/files/core_dump.zip</a> (around 20MB)</div><div><br></div><div>Thanks Olaf</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Op do 20 jun. 2019 om 08:35 schreef Sanju Rakonde <<a href="mailto:srakonde@redhat.com" target="_blank">srakonde@redhat.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Olaf,<div><br></div><div>Can you please paste complete backtrace from the core file, so that we can analyse what is wrong here.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 19, 2019 at 10:31 PM Olaf Buitelaar <<a href="mailto:olaf.buitelaar@gmail.com" target="_blank">olaf.buitelaar@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Atin,<div><br></div><div>Thank you for pointing out this bug report, however no rebalancing task was running during this event. So maybe something else is causing this?</div><div>According the report this should be fixed in gluster 6, unfortunate ovirt doesn't seem to officially support that version, so i'm stuck on the 5 branch for now. </div><div>Any chance this will be back ported? </div><div><br></div><div>Thanks Olaf</div><div> </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Op wo 19 jun. 2019 om 17:57 schreef Atin Mukherjee <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Please see - <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1655827" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1655827</a></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 19, 2019 at 5:52 PM Olaf Buitelaar <<a href="mailto:olaf.buitelaar@gmail.com" target="_blank">olaf.buitelaar@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Dear All,<div><br></div><div>Has anybody seen this error on gluster 5.6;</div><div>[glusterd-rpc-ops.c:1388:__glusterd_commit_op_cbk] (-->/lib64/libgfrpc.so.0(+0xec60) [0x7fbfb7801c60] -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x79b7a) [0x7fbfac50db7a] -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x77393) [0x7fbfac50b393] ) 0-: Assertion failed: rsp.op == txn_op_info.op<br></div><div><br></div><div>checking the code; <a href="https://github.com/gluster/glusterfs/blob/6fd8281ac9af58609979f660ece58c2ed1100e72/xlators/mgmt/glusterd/src/glusterd-rpc-ops.c#L1388" target="_blank">https://github.com/gluster/glusterfs/blob/6fd8281ac9af58609979f660ece58c2ed1100e72/xlators/mgmt/glusterd/src/glusterd-rpc-ops.c#L1388</a> </div><div>doesn't seem to reveal much on what could causing this.</div><div><br></div><div>It's the second time this occurs. </div><div><br></div><div>Attached the full stack.</div><div><br></div><div>Thanks Olaf</div></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div>
</blockquote></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_8023491083303005622gmail-m_9167453745802894500gmail-m_-4170331012233077658gmail_signature"><div dir="ltr"><div>Thanks,<br></div>Sanju<br></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>Thanks,<br></div>Sanju<br></div></div>