<div dir="ltr"><div><div>Thanks for this detailed Email with clear instructions Niels.<br><br></div>Can we have this as github issues as well? I don't want this information to be lost as part of email thread in ML archive. I would like to see if I can get some interns to work on these leaks per component as part of their college project etc.<br><br></div>-Amar<br><div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 25, 2017 at 2:29 PM, Niels de Vos <span dir="ltr"><<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
with the use of gfapi it has become clear that Gluster was never really<br>
developed to be loaded in applications. There are many different memory<br>
leaks that get exposed through the usage of gfapi. Before, memory leaks<br>
were mostly cleaned up automatically because processes would exit. Now,<br>
there are applications that initialize a GlusterFS client to access a<br>
volume, and de-init that client once not needed anymore. Unfortunately<br>
upon the de-init, not all allocated memory is free'd again. These are<br>
often just a few bytes, but for long running processes this can become a<br>
real problem.<br>
<br>
Finding the memory leaks in xlators has been a tricky thing. Valgrind<br>
would often not know what function/source the allocation did, and fixing<br>
the leak would become a real hunt. There have been some patches merged<br>
that make Valgrind work more easily, and a few patches still need a<br>
little more review before everything is available. The document below<br>
describes how to use a new "sink" xlator to detect memory leaks. This<br>
xlator can only be merged when a change for the graph initialization is<br>
included too:<br>
<br>
- <a href="https://review.gluster.org/16796" rel="noreferrer" target="_blank">https://review.gluster.org/<wbr>16796</a> - glusterfs_graph_prepare<br>
- <a href="https://review.gluster.org/16806" rel="noreferrer" target="_blank">https://review.gluster.org/<wbr>16806</a> - sink xlator and developer doc<br>
<br>
It would be most welcome if other developers can reviwe the linked<br>
changes, so that everyone can easily debug memory leaks from thair<br>
favorite xlators.<br>
<br>
There is a "run-xlator.sh" script in my (for now) personal<br>
"gluster-debug" repository that can be used to load an arbitrary xlator<br>
along with "sink". See<br>
<a href="https://github.com/nixpanic/gluster-debug/tree/master/gfapi-load-volfile" rel="noreferrer" target="_blank">https://github.com/nixpanic/<wbr>gluster-debug/tree/master/<wbr>gfapi-load-volfile</a><br>
for more details.<br>
<br>
Thanks,<br>
Niels<br>
<br>
<br>
>From doc/developer-guide/<a href="http://identifying-resource-leaks.md" rel="noreferrer" target="_blank">identifyin<wbr>g-resource-leaks.md</a>:<br>
<br>
# Identifying Resource Leaks<br>
<br>
Like most other pieces of software, GlusterFS is not perfect in how it manages<br>
its resources like memory, threads and the like. Gluster developers try hard to<br>
prevent leaking resources but releasing and unallocating the used structures.<br>
Unfortunately every now and then some resource leaks are unintentionally added.<br>
<br>
This document tries to explain a few helpful tricks to identify resource leaks<br>
so that they can be addressed.<br>
<br>
<br>
## Debug Builds<br>
<br>
There are certain techniques used in GlusterFS that make it difficult to use<br>
tools like Valgrind for memory leak detection. There are some build options<br>
that make it more practical to use Valgrind and other tools. When running<br>
Valgrind, it is important to have GlusterFS builds that contain the<br>
debuginfo/symbols. Some distributions (try to) strip the debuginfo to get<br>
smaller executables. Fedora and RHEL based distributions have sub-packages<br>
called ...-debuginfo that need to be installed for symbol resolving.<br>
<br>
<br>
### Memory Pools<br>
<br>
By using memory pools, there are no allocation/freeing of single structures<br>
needed. This improves performance, but also makes it impossible to track the<br>
allocation and freeing of srtuctures.<br>
<br>
It is possible to disable the use of memory pools, and use standard `malloc()`<br>
and `free()` functions provided by the C library. Valgrind is then able to<br>
track the allocated areas and verify if they have been free'd. In order to<br>
disable memory pools, the Gluster sources needs to be configured with the<br>
`--enable-debug` option:<br>
<br>
```shell<br>
./configure --enable-debug<br>
```<br>
<br>
When building RPMs, the `.spec` handles the `--with=debug` option too:<br>
<br>
```shell<br>
make dist<br>
rpmbuild -ta --with=debug glusterfs-....tar.gz<br>
```<br>
<br>
### Dynamically Loaded xlators<br>
<br>
Valgrind tracks the call chain of functions that do memory allocations. The<br>
addresses of the functions are stored and before Valgrind exits the addresses<br>
are resolved into human readable function names and offsets (line numbers in<br>
source files). Because Gluster loads xlators dynamically, and unloads then<br>
before exiting, Valgrind is not able to resolve the function addresses into<br>
symbols anymore. Whenever this happend, Valgrind shows `???` in the output,<br>
like<br>
<br>
```<br>
==25170== 344 bytes in 1 blocks are definitely lost in loss record 233 of 324<br>
==25170== at 0x4C29975: calloc (vg_replace_malloc.c:711)<br>
==25170== by 0x52C7C0B: __gf_calloc (mem-pool.c:117)<br>
==25170== by 0x12B0638A: ???<br>
==25170== by 0x528FCE6: __xlator_init (xlator.c:472)<br>
==25170== by 0x528FE16: xlator_init (xlator.c:498)<br>
...<br>
```<br>
<br>
These `???` can be prevented by not calling `dlclose()` for unloading the<br>
xlator. This will cause a small leak of the handle that was returned with<br>
`dlopen()`, but for improved debugging this can be acceptible. For this and<br>
other Valgrind features, a `--enable-valgrind` option is available to<br>
`./configure`. When GlusterFS is built with this option, Valgrind will be able<br>
to resolve the symbol names of the functions that do memory allocations inside<br>
xlators.<br>
<br>
```shell<br>
./configure --enable-valgrind<br>
```<br>
<br>
When building RPMs, the `.spec` handles the `--with=valgrind` option too:<br>
<br>
```shell<br>
make dist<br>
rpmbuild -ta --with=valgrind glusterfs-....tar.gz<br>
```<br>
<br>
## Running Valgrind against a single xlator<br>
<br>
Debugging a single xlator is not trivial. But there are some tools to make it<br>
easier. The `sink` xlator does not do any memory allocations itself, but<br>
contains just enough functionality to mount a volume with only the `sink`<br>
xlator. There is a little gfapi application under `tests/basic/gfapi/` in the<br>
GlusterFS sources that can be used to run only gfapi and the core GlusterFS<br>
infrastructure with the `sink` xlator. By extending the `.vol` file to load<br>
more xlators, each xlator can be debugged pretty much separately (as long as<br>
the xlators have no dependencies on each other). A basic Valgrind run with the<br>
suitable configure options looks like this:<br>
<br>
```shell<br>
./autogen.sh<br>
./configure --enable-debug --enable-valgrind<br>
make && make install<br>
cd tests/basic/gfapi/<br>
make gfapi-load-volfile<br>
valgrind ./gfapi-load-volfile sink.vol<br>
```<br>
<br>
Combined with other very useful options to Valgrind, the following execution<br>
shows many more useful details:<br>
<br>
```shell<br>
valgrind \<br>
--fullpath-after= --leak-check=full --show-leak-kinds=all \<br>
./gfapi-load-volfile sink.vol<br>
```<br>
<br>
Note that the `--fullpath-after=` option is left empty, this makes Valgrind<br>
print the full path and filename that contains the functions:<br>
<br>
```<br>
==2450== 80 bytes in 1 blocks are definitely lost in loss record 8 of 60<br>
==2450== at 0x4C29975: calloc (/builddir/build/BUILD/<wbr>valgrind-3.11.0/coregrind/m_<wbr>replacemalloc/vg_replace_<wbr>malloc.c:711)<br>
==2450== by 0x52C6F73: __gf_calloc (/usr/src/debug/glusterfs-3.<wbr>11dev/libglusterfs/src/mem-<wbr>pool.c:117)<br>
==2450== by 0x12F10CDA: init (/usr/src/debug/glusterfs-3.<wbr>11dev/xlators/meta/src/meta.c:<wbr>231)<br>
==2450== by 0x528EFD5: __xlator_init (/usr/src/debug/glusterfs-3.<wbr>11dev/libglusterfs/src/xlator.<wbr>c:472)<br>
==2450== by 0x528F105: xlator_init (/usr/src/debug/glusterfs-3.<wbr>11dev/libglusterfs/src/xlator.<wbr>c:498)<br>
==2450== by 0x52D9D8B: glusterfs_graph_init (/usr/src/debug/glusterfs-3.<wbr>11dev/libglusterfs/src/graph.<wbr>c:321)<br>
...<br>
```<br>
<br>
In the above example, the `init` function in `xlators/meta/src/meta.c` does a<br>
memory allocation on line 231. This memory is never free'd again, and hence<br>
Valgrind logs this call stack. When looking in the code, it seems that the<br>
allocation of `priv` is assigned to the `this->private` member of the<br>
`xlator_t` structure. Because the allocation is done in `init()`, free'ing is<br>
expected to happen in `fini()`. Both functions are shown below, with the<br>
inclusion of the empty `fini()`:<br>
<br>
<br>
```<br>
226 int<br>
227 init (xlator_t *this)<br>
228 {<br>
229 meta_priv_t *priv = NULL;<br>
230<br>
231 priv = GF_CALLOC (sizeof(*priv), 1, gf_meta_mt_priv_t);<br>
232 if (!priv)<br>
233 return -1;<br>
234<br>
235 GF_OPTION_INIT ("meta-dir-name", priv->meta_dir_name, str, out);<br>
236<br>
237 this->private = priv;<br>
238 out:<br>
239 return 0;<br>
240 }<br>
241<br>
242<br>
243 int<br>
244 fini (xlator_t *this)<br>
245 {<br>
246 return 0;<br>
247 }<br>
```<br>
<br>
In this case, the resource leak can be addressed by adding a single line to the<br>
`fini()` function:<br>
<br>
```<br>
243 int<br>
244 fini (xlator_t *this)<br>
245 {<br>
246 GF_FREE (this->private);<br>
247 return 0;<br>
248 }<br>
```<br>
<br>
Running the same Valgrind command and comparing the output will show that the<br>
memory leak in `xlators/meta/src/meta.c:init` is not reported anymore.<br>
<br>______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>
</div></div></div></div></div>