[Gluster-devel] Should it be possible to disable own-thread for encrypted RPC?

Jeff Darcy jdarcy at redhat.com
Thu Apr 21 12:07:38 UTC 2016


> I've recently become aware of another problem with own-threads. The
> threads launched are not reaped, pthread_joined, after a TLS
> connection disconnects.  This is especially problematic with GlusterD
> as it launches a lot of threads to handle generally short lived
> connections (volfile fetch, portmapper).  This causes GlusterDs mem
> usage to continually grow, and finally lead to other failures due to
> memory shortage.  I've recently seen a setup with GlusterD memory
> usage in 10s of GBs of reserved mem and TBs of virt mem. This is
> easily reproducible as well.  I'm still working out a solution for
> this.
> 
> While allowing TLS connections with own-threads only will lead to a
> more stable experience, this is a really bad in terms of our memory
> consumption.  This will badly affect our chances of having 1000s of
> clients. Making TLS work with epoll would fix this, but I'm not very
> sure of the effort involved.  Could we fix this for 3.8? For 4.0, if
> we want to default to TLS, we definitely need to fix this.

Maybe it's just my not-so-humble opinion, but reaping threads seems
like a pretty easy thing to implement.  By contrast, the prospects
of making TLS (specifically OpenSSL) work reliably with epoll seem
murky at best.  Nothing has been easy with epoll so far, and I don't
see why we'd expect making it work reliably with OpenSSL's horrible
API would be the first exception.  Fixing one small issue with
own-thread still seems like the quickest route to a stable TLS
implementation.


More information about the Gluster-devel mailing list