<div dir="ltr"><div dir="ltr"><div>Looks like using BIO_free() is not right. Here's what the SSL_set_bio() man page says ...</div><div><br></div><div><span style="background-color:rgb(255,242,204)">SSL_set_bio() is similar to SSL_set0_rbio() and SSL_set0_wbio() except that it connects both the rbio and the wbio at the same time, and <b><span style="color:rgb(255,0,0)">transfers the ownership of rbio and wbio to ssl</span></b> according to the following set of rules:</span></div><div><br></div><div>So, I think you were right about SSL_free() doing the job for the bio. However, SSL_free() has no reason to set the priv->ssl_sbio pointer to NULL. I think priv->ssl_sbio should be set to NULL immediately after the call to SSL_set_bio() is successful. And we need to add a comment while setting priv->ssl_sbio to NULL that the ownership of the bio has now been transferred to SSL and SSL will free the related memory appropriately.<br></div><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Apr 22, 2019 at 11:37 AM Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com">cynthia.zhou@nokia-sbell.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="ZH-CN">
<div class="gmail-m_2788971008235605304WordSection1">
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">I tried to print priv->ssl_sbio after SSL_free() find this pointer is not null, so I add free ssl_sbio with BIO_free, however this cause glusterfd coredump<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">(gdb) bt<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">#0 0x00007f3047867f9b in raise () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">#1 0x00007f3047869351 in abort () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">#2 0x00007f30478aa8c7 in __libc_message () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">#3 0x00007f30478b0e6a in malloc_printerr () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">#4 0x00007f30478b2835 in _int_free () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">#5 0x00007f3047c5bbbd in CRYPTO_free () from /lib64/libcrypto.so.10<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">#6 0x00007f3047d07582 in BIO_free () from /lib64/libcrypto.so.10<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">#7 0x00007f3043f9ba4b in __socket_reset (this=0x7f303c1ae710) at socket.c:1032<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">#8 0x00007f3043f9c4aa in socket_event_poll_err (this=0x7f303c1ae710, gen=1, idx=17) at socket.c:1232<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">#9 0x00007f3043fa1b7d in socket_event_handler (fd=26, idx=17, gen=1, data=0x7f303c1ae710, poll_in=1, poll_out=0, poll_err=0) at socket.c:2669<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">#10 0x00007f3049307984 in event_dispatch_epoll_handler (event_pool=0x1035610, event=0x7f3043b14e84) at event-epoll.c:587<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">#11 0x00007f3049307c5b in event_dispatch_epoll_worker (data=0x107e3e0) at event-epoll.c:663<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">#12 0x00007f30480535da in start_thread () from /lib64/libpthread.so.0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">#13 0x00007f3047929eaf in clone () from /lib64/libc.so.6<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">@@ -1019,7 +1019,20 @@ static void __socket_reset(rpc_transport_t *this) {<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> memset(&priv->incoming, 0, sizeof(priv->incoming));<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">-<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ if(priv->use_ssl&& priv->ssl_ssl)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ {<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ gf_log(this->name, GF_LOG_INFO,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ "clear and reset for socket(%d), free ssl ",<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ priv->sock);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ SSL_shutdown(priv->ssl_ssl);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ SSL_clear(priv->ssl_ssl);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ SSL_free(priv->ssl_ssl);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ gf_log(this->name, GF_LOG_INFO,"priv->ssl_sbio of socket(%d)is %p ",priv->sock,priv->ssl_sbio);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ if(priv->ssl_sbio != NULL)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ BIO_free(priv->ssl_sbio);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ priv->ssl_ssl = NULL;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ priv->ssl_sbio = NULL;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ }<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> priv->sock = -1;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> priv->idx = -1;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> priv->connected = -1;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">@@ -4238,6 +4251,20 @@ void fini(rpc_transport_t *this) {<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> pthread_mutex_destroy(&priv->out_lock);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> pthread_mutex_destroy(&priv->cond_lock);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> pthread_cond_destroy(&priv->cond);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ if(priv->use_ssl&& priv->ssl_ssl)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ {<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ gf_log(this->name, GF_LOG_INFO,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ "clear and reset for socket(%d), free ssl ",<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ priv->sock);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ SSL_shutdown(priv->ssl_ssl);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ SSL_clear(priv->ssl_ssl);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ SSL_free(priv->ssl_ssl);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ gf_log(this->name, GF_LOG_INFO,"priv->ssl_sbio of socket(%d)is %p ",priv->sock,priv->ssl_sbio);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ if(priv->ssl_sbio != NULL)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ BIO_free(priv->ssl_sbio);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ priv->ssl_ssl = NULL;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ priv->ssl_sbio = NULL;<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ }<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> if (priv->ssl_private_key) {<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> GF_FREE(priv->ssl_private_key);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US"> Milind Changire <<a href="mailto:mchangir@redhat.com" target="_blank">mchangir@redhat.com</a>>
<br>
<b>Sent:</b> Monday, April 22, 2019 1:35 PM<br>
<b>To:</b> Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>><br>
<b>Cc:</b> Atin Mukherjee <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>>; <a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">This probably went unnoticed until now.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">On Mon, Apr 22, 2019 at 10:45 AM Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>> wrote:<u></u><u></u></span></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">Why there is no bio_free called in ssl_teardown_connection then?
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">cynthia</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US"> Milind
Changire <</span><span lang="EN-US"><a href="mailto:mchangir@redhat.com" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">mchangir@redhat.com</span></a></span><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">>
<br>
<b>Sent:</b> Monday, April 22, 2019 10:21 AM<br>
<b>To:</b> Zhou, Cynthia (NSB - CN/Hangzhou) <</span><span lang="EN-US"><a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">cynthia.zhou@nokia-sbell.com</span></a></span><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">><br>
<b>Cc:</b> Atin Mukherjee <</span><span lang="EN-US"><a href="mailto:amukherj@redhat.com" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">amukherj@redhat.com</span></a></span><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">>;
</span><span lang="EN-US"><a href="mailto:gluster-devel@gluster.org" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">gluster-devel@gluster.org</span></a></span><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US"><br>
<b>Subject:</b> Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">After patch
<a href="https://review.gluster.org/c/glusterfs/+/22334" target="_blank">22334</a>, the
</span><span style="font-family:"Courier New"" lang="EN-US">priv->ssl_ctx</span><span lang="EN-US"> is now maintained per socket connection and is no longer shared.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">So you might want to
</span><span style="font-family:"Courier New"" lang="EN-US">SSL_free(priv->ssl_ctx)</span><span lang="EN-US"> as well and set
</span><span style="font-family:"Courier New"" lang="EN-US">priv->ssl_ctx</span><span lang="EN-US"> to
</span><span style="font-family:"Courier New"" lang="EN-US">NULL</span><span lang="EN-US">.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">There might be some strings that are duplicated (</span><span style="font-family:"Courier New"" lang="EN-US">gf_strdup()</span><span lang="EN-US">) via the
</span><span style="font-family:"Courier New"" lang="EN-US">socket_init()</span><span lang="EN-US"> code path. Please take a look at those as well.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Sorry about that. I missed it.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">On Mon, Apr 22, 2019 at 7:25 AM Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>
wrote:<u></u><u></u></span></p>
</div>
<blockquote style="border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">Hi,</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">From my code study it seems priv->ssl_ssl is not properly released, I made a patch and the glusterfsd memory leak
is alleviated with my patch, but some otherwhere is still leaking, I have no clue about the other leak points.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">--- a/rpc/rpc-transport/socket/src/socket.c</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+++ b/rpc/rpc-transport/socket/src/socket.c</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">@@ -1019,7 +1019,16 @@ static void __socket_reset(rpc_transport_t *this) {</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> memset(&priv->incoming, 0, sizeof(priv->incoming));</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx);</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">-</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ if(priv->use_ssl&& priv->ssl_ssl)</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ {</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ gf_log(this->name, GF_LOG_TRACE,</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ "clear and reset for socket(%d), free ssl ",</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ priv->sock);</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ SSL_shutdown(priv->ssl_ssl);</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ SSL_clear(priv->ssl_ssl);</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ SSL_free(priv->ssl_ssl);</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ priv->ssl_ssl = NULL;</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ }</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> priv->sock = -1;</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> priv->idx = -1;</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> priv->connected = -1;</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">@@ -4238,6 +4250,16 @@ void fini(rpc_transport_t *this) {</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> pthread_mutex_destroy(&priv->out_lock);</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> pthread_mutex_destroy(&priv->cond_lock);</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> pthread_cond_destroy(&priv->cond);</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ if(priv->use_ssl&& priv->ssl_ssl)</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ {</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ gf_log(this->name, GF_LOG_TRACE,</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ "clear and reset for socket(%d), free ssl ",</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ priv->sock);</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ SSL_shutdown(priv->ssl_ssl);</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ SSL_clear(priv->ssl_ssl);</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ SSL_free(priv->ssl_ssl);</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ priv->ssl_ssl = NULL;</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">+ }</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> if (priv->ssl_private_key) {</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> GF_FREE(priv->ssl_private_key);</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> }</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> </span><span lang="EN-US"><u></u><u></u></span></p>
<div>
<div style="border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm;border-color:currentcolor">
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US"> Zhou,
Cynthia (NSB - CN/Hangzhou) <br>
<b>Sent:</b> Thursday, April 18, 2019 5:31 PM<br>
<b>To:</b> 'Atin Mukherjee' <</span><span lang="EN-US"><a href="mailto:amukherj@redhat.com" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">amukherj@redhat.com</span></a></span><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">><br>
<b>Cc:</b> 'Raghavendra Gowdappa' <</span><span lang="EN-US"><a href="mailto:rgowdapp@redhat.com" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">rgowdapp@redhat.com</span></a></span><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">>;
'</span><span lang="EN-US"><a href="mailto:gluster-devel@gluster.org" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">gluster-devel@gluster.org</span></a></span><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">'
<</span><span lang="EN-US"><a href="mailto:gluster-devel@gluster.org" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">gluster-devel@gluster.org</span></a></span><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">><br>
<b>Subject:</b> RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">We scan it use memory-leak tool, there are following prints. We doubt some open ssl lib malloc is is not properly
freed by glusterfs code.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US">er+0x2af [</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">libglusterfs.so.0.0.1]\n\t\tstart_thread+0xda</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US">
[</span><span lang="EN-US"><a href="http://libpthread-2.27.so" target="_blank"><span style="font-size:10pt;font-family:"Segoe UI",sans-serif">libpthread-2.27.so</span></a></span><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US">]'<br>
</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">13580</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US"> bytes in 175 allocations from stack<br>
b'CRYPTO_malloc+0x58 [</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">libcrypto.so.1.0.2p</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US">]'<br>
</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">232904</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US"> bytes in 14 allocations from stack<br>
b'CRYPTO_malloc+0x58 [</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">libcrypto.so.1.0.2p]\n\t\t[unknown</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US">]'<br>
[15:41:56] Top 10 stacks with outstanding allocations:<br>
</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">8792</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US"> bytes in 14 allocations from stack<br>
b'CRYPTO_malloc+0x58 [</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">libcrypto.so.1.0.2p]\n\t\t[unknown</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US">]'<br>
</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">9408</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US"> bytes in 42 allocations from stack<br>
b'CRYPTO_realloc+0x4d [</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">libcrypto.so.1.0.2p</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US">]'<br>
</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">9723</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US"> bytes in 14 allocations from stack<br>
b'CRYPTO_malloc+0x58 [</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">libcrypto.so.1.0.2p]\n\t\t[unknown</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US">]'<br>
</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">10696</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US"> bytes in 21 allocations from stack<br>
b'CRYPTO_malloc+0x58 [</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">libcrypto.so.1.0.2p]\n\t\t[unknown</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US">]'<br>
</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">11319</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US"> bytes in 602 allocations from stack<br>
b'CRYPTO_malloc+0x58 [</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">libcrypto.so.1.0.2p]\n\t\t[unknown</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US">]'<br>
</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">11431</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US"> bytes in 518 allocations from stack<br>
b'CRYPTO_malloc+0x58 [</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">libcrypto.so.1.0.2p]\n\t\t[unknown</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US">]'<br>
</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">11704</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US"> bytes in 371 allocations from stack<br>
b'CRYPTO_malloc+0x58 [</span><u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:blue" lang="EN-US">libcrypto.so.1.0.2p]\n\t\t[unknown</span></u><span style="font-size:10pt;font-family:"Segoe UI",sans-serif;color:rgb(26,26,26)" lang="EN-US">]'</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">cynthia</span><span lang="EN-US"><u></u><u></u></span></p>
<div>
<div style="border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm;border-color:currentcolor">
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US"> Zhou,
Cynthia (NSB - CN/Hangzhou) <br>
<b>Sent:</b> Thursday, April 18, 2019 5:25 PM<br>
<b>To:</b> 'Atin Mukherjee' <</span><span lang="EN-US"><a href="mailto:amukherj@redhat.com" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">amukherj@redhat.com</span></a></span><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">><br>
<b>Cc:</b> Raghavendra Gowdappa <</span><span lang="EN-US"><a href="mailto:rgowdapp@redhat.com" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">rgowdapp@redhat.com</span></a></span><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">>;
</span><span lang="EN-US"><a href="mailto:gluster-devel@gluster.org" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">gluster-devel@gluster.org</span></a></span><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US"><br>
<b>Subject:</b> RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">I</span><span style="font-size:10.5pt;font-family:DengXian">’<span lang="EN-US">ve test on glusterfs3.12.15 and
glusterfs5.5 all have this issue, after enable tls ssl socket, when execute gluster v heal <vol-name> info, will trigger glfshel to connect glusterfsd process, and cause glusterfsd process memory leak. Could you please try in your env?
</span></span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US">cynthia</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:DengXian" lang="EN-US"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US"> Atin
Mukherjee <</span><span lang="EN-US"><a href="mailto:amukherj@redhat.com" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">amukherj@redhat.com</span></a></span><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">>
<br>
<b>Sent:</b> Thursday, April 18, 2019 1:19 PM<br>
<b>To:</b> Zhou, Cynthia (NSB - CN/Hangzhou) <</span><span lang="EN-US"><a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">cynthia.zhou@nokia-sbell.com</span></a></span><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">><br>
<b>Cc:</b> Raghavendra Gowdappa <</span><span lang="EN-US"><a href="mailto:rgowdapp@redhat.com" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">rgowdapp@redhat.com</span></a></span><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">>;
</span><span lang="EN-US"><a href="mailto:gluster-devel@gluster.org" target="_blank"><span style="font-size:11pt;font-family:"Calibri",sans-serif">gluster-devel@gluster.org</span></a></span><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US"><br>
<b>Subject:</b> Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">On Wed, 17 Apr 2019 at 10:53, Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>
wrote:<u></u><u></u></span></p>
</div>
<blockquote style="border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">In my recent test, I found that there are very severe glusterfsd memory leak when enable socket ssl option<u></u><u></u></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">What gluster version are you testing? Would you be able to continue your investigation and share the root cause? <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US">--
<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">- Atin (atinm)<u></u><u></u></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US">_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a><u></u><u></u></span></p>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US"><br clear="all">
<br>
-- <u></u><u></u></span></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span lang="EN-US">Milind<u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US"><br clear="all">
<br>
-- <u></u><u></u></span></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span lang="EN-US">Milind<u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr">Milind<br><br></div></div></div></div>