[Gluster-devel] Memory management and friends

[Oleksandr Natalenko]

Hello.

As a result of today's community meeting I start dedicated ML thread for 
gathering memory management issues together to make it possible to 
summarize them and construct some plan what to do next.

Very important notice: I'm not the active GlusterFS developer, but 
gained excessive experience with GlusterFS in the past at previous work, 
and the main issue that was chasing me all the time was memory leaking. 
Consider this as a request for action from GlusterFS customer, 
apparently approved by Kaushal and Amye during last meeting :).

So, here go key points.

1) Almost all nasty and obvious memory leaks have been successfully 
fixed during the last year, and that allowed me to run GlusterFS in 
production at previous work for almost all types of workload except one 
— dovecot mail storage. The specific of this workload is that it 
involved huge amount of files, and I assume this to be kinda of edge 
case unhiding some dark corners of GlusterFS memory management. I was 
able to provide Nithya with Valgrind+Massif memory profiling results and 
test case, and that helped her to prepare at least 1 extra fix (and more 
to come AFAIK), which has some deal with readdirp-related code. 
Nevertheless, it is reported that this is not the major source of 
leaking. Nithya suspect that memory gets fragmented heavily due to lots 
of small allocations, and memory pools cannot cope with this kind of 
fragmentation under constant load.

Related BZs:

   * https://bugzilla.redhat.com/show_bug.cgi?id=1369364
   * https://bugzilla.redhat.com/show_bug.cgi?id=1380249

People involved:

   * nbalacha, could you please provide more info on your findings?

2) Meanwhile, Jeff goes on with brick multiplexing feature, facing some 
issue with memory management too and blaming memory pools for that.

Related ML email:

   * 
http://www.gluster.org/pipermail/gluster-devel/2016-October/051118.html
   * 
http://www.gluster.org/pipermail/gluster-devel/2016-October/051160.html

People involved:

   * jdarcy, have you discussed this outside of ML? It seems your email 
didn't get proper attention.

3) We had brief discussion with obnox and anoopcs on #gluster-meeting 
and #gluster-dev regarding jemalloc and talloc. obnox believes that we 
may use both, jemalloc for substituting malloc/free, talloc for 
rewriting memory management for GlusterFS properly.

Related logs:

   * 
https://botbot.me/freenode/gluster-dev/2016-10-26/?msg=75501394&page=2

People involved:

   * obnox, could you share your ideas on this?

To summarize:

1) we need key devs involved in memory management to share their ideas;
2) using production-proven memory allocators and memory pools 
implementation is desired;
3) someone should manage the workflow of reconstructing memory 
management.

Feel free to add anything I've missed.

Regards,
   Oleksandr