[Gluster-devel] Cores generated with ./tests/geo-rep/georep-basic-dr-tarssh.t

[Kotresh Hiremath Ravishankar]

Hi Soumya,

I analysed the issue and found out that crash has happened because of the patch [1].

The patch doesn't set transport object to NULL in 'rpc_clnt_disable' but instead does it on
'rpc_clnt_trigger_destroy'. So if there are pending rpc invocations on the rpc object that
is disabled (those instances are possible as happening now in changelog), it will trigger a
CONNECT notify again with 'mydata' that is freed causing a crash. This happens because
'rpc_clnt_submit' reconnects if rpc is not connected.

 rpc_clnt_submit (...) {
   ...
                if (conn->connected == 0) {
                        ret = rpc_transport_connect (conn->trans,
                                                     conn->config.remote_port);
                }
   ...
 }

Without your patch, conn->trans was set NULL and hence CONNECT fails not resulting with
CONNECT notify call. And also the cleanup happens in failure path.

So the memory leak can happen, if there is no try for rpc invocation after DISCONNECT.
It will be cleaned up otherwise.


[1] http://review.gluster.org/#/c/13507/

Thanks and Regards,
Kotresh H R

----- Original Message -----
> From: "Kotresh Hiremath Ravishankar" <khiremat at redhat.com>
> To: "Soumya Koduri" <skoduri at redhat.com>
> Cc: avishwan at redhat.com, "Gluster Devel" <gluster-devel at gluster.org>
> Sent: Monday, February 29, 2016 4:15:22 PM
> Subject: Re: Cores generated with ./tests/geo-rep/georep-basic-dr-tarssh.t
> 
> Hi Soumya,
> 
> I just tested that it is reproducible only with your patch both in master and
> 3.76 branch.
> The geo-rep test cases are marked bad in master. So it's not hit in master.
> rpc is introduced
> in changelog xlator to communicate to applications via libgfchangelog.
> Venky/Me will check
> why is the crash happening and will update.
> 
> 
> Thanks and Regards,
> Kotresh H R
> 
> ----- Original Message -----
> > From: "Soumya Koduri" <skoduri at redhat.com>
> > To: avishwan at redhat.com, "kotresh" <khiremat at redhat.com>
> > Cc: "Gluster Devel" <gluster-devel at gluster.org>
> > Sent: Monday, February 29, 2016 2:10:51 PM
> > Subject: Cores generated with ./tests/geo-rep/georep-basic-dr-tarssh.t
> > 
> > Hi Aravinda/Kotresh,
> > 
> > With [1], I consistently see cores generated with the test
> > './tests/geo-rep/georep-basic-dr-tarssh.t' in release-3.7 branch. From
> > the cores, looks like we are trying to dereference a freed
> > changelog_rpc_clnt_t(crpc) object in changelog_rpc_notify(). Strangely
> > this was not reported in master branch.
> > 
> > I tried debugging but couldn't find any possible suspects. I request you
> > to take a look and let me know if [1] caused any regression.
> > 
> > Thanks,
> > Soumya
> > 
> > [1] http://review.gluster.org/#/c/13507/
> > 
>