[Gluster-devel] Gluster statedumps and mallinfo

Raghavendra Gowdappa rgowdapp at redhat.com
Mon Jul 3 09:55:44 UTC 2017 

Hi,

Recently I observed one of the mallinfo fields had a negative value.

DUMP-START-TIME: 2017-06-09 10:59:43.747440

[mallinfo]
mallinfo_arena=-1517670400
mallinfo_ordblks=8008214
mallinfo_smblks=0
mallinfo_hblks=1009
mallinfo_hblkhd=863453184
mallinfo_usmblks=0
mallinfo_fsmblks=0
mallinfo_uordblks=1473090528
mallinfo_fordblks=1304206368
mallinfo_keepcost=2232208

As seen above mallinfo_arena is negative.

On probing further I came across posts that said mallinfo is not the ideal interface to get metadata about memory allocated by malloc [1]. Instead there were two alternatives - malloc_stats and malloc_info - suggested.

With all the three information taken statedump looks as below:
DUMP-START-TIME: 2017-07-03 06:56:35.991499

[malloc_stats]
Arena 0:
system bytes     =     622592
in use bytes     =     498320
Arena 1:
system bytes     =     135168
in use bytes     =       2224
Arena 2:
system bytes     =     135168
in use bytes     =       2400
Arena 3:
system bytes     =     135168
in use bytes     =       2256
Arena 4:
system bytes     =     135168
in use bytes     =       2256
Arena 5:
system bytes     =     135168
in use bytes     =       2256
Arena 6:
system bytes     =     135168
in use bytes     =       2256
Arena 7:
system bytes     =     135168
in use bytes     =       2256
Arena 8:
system bytes     =     135168
in use bytes     =       2256
Arena 9:
system bytes     =     704512
in use bytes     =     668864
Total (incl. mmap):
system bytes     =   17334272
in use bytes     =   16111168
max mmap regions =          7
max mmap bytes   =   14925824

[malloc_info]
<malloc version="1">
<heap nr="0">
<sizes>
<size from="81" to="81" total="81" count="1"/>
<unsorted from="0" to="273" total="418" count="2"/>
</sizes>
<total type="fast" count="0" size="0"/>
<total type="rest" count="3" size="499"/>
<system type="current" size="622592"/>
<system type="max" size="622592"/>
<aspace type="total" size="622592"/>
<aspace type="mprotect" size="622592"/>
</heap>
<heap nr="1">
<sizes>
<unsorted from="0" to="273" total="418" count="2"/>
</sizes>
<total type="fast" count="0" size="0"/>
<total type="rest" count="2" size="418"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="2">
<sizes>
<unsorted from="0" to="273" total="418" count="2"/>
</sizes>
<total type="fast" count="0" size="0"/>
<total type="rest" count="2" size="418"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="3">
<sizes>
<unsorted from="0" to="273" total="418" count="2"/>
</sizes>
<total type="fast" count="0" size="0"/>
<total type="rest" count="2" size="418"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="4">
<sizes>
<unsorted from="0" to="273" total="418" count="2"/>
</sizes>
<total type="fast" count="0" size="0"/>
<total type="rest" count="2" size="418"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="5">
<sizes>
<unsorted from="0" to="273" total="418" count="2"/>
</sizes>
<total type="fast" count="0" size="0"/>
<total type="rest" count="2" size="418"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="6">
<sizes>
<unsorted from="0" to="273" total="418" count="2"/>
</sizes>
<total type="fast" count="0" size="0"/>
<total type="rest" count="2" size="418"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="7">
<sizes>
<unsorted from="0" to="273" total="418" count="2"/>
</sizes>
<total type="fast" count="0" size="0"/>
<total type="rest" count="2" size="418"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="8">
<sizes>
<unsorted from="0" to="273" total="418" count="2"/>
</sizes>
<total type="fast" count="0" size="0"/>
<total type="rest" count="2" size="418"/>
<system type="current" size="135168"/>
<system type="max" size="135168"/>
<aspace type="total" size="135168"/>
<aspace type="mprotect" size="135168"/>
</heap>
<heap nr="9">
<sizes>
<size from="33" to="33" total="33" count="1"/>
<size from="49" to="49" total="637" count="13"/>
<size from="65" to="65" total="65" count="1"/>
<size from="97" to="97" total="97" count="1"/>
<size from="113" to="113" total="226" count="2"/>
<size from="177" to="177" total="177" count="1"/>
<size from="241" to="241" total="241" count="1"/>
<size from="497" to="497" total="497" count="1"/>
<unsorted from="0" to="2993" total="3556" count="4"/>
</sizes>
<total type="fast" count="0" size="0"/>
<total type="rest" count="25" size="5529"/>
<system type="current" size="704512"/>
<system type="max" size="704512"/>
<aspace type="total" size="704512"/>
<aspace type="mprotect" size="704512"/>
</heap>
<total type="fast" count="0" size="0"/>
<total type="rest" count="44" size="9372"/>
<system type="current" size="2408448"/>
<system type="max" size="2408448"/>
<aspace type="total" size="2408448"/>
<aspace type="mprotect" size="2408448"/>
</malloc>

[mallinfo]
mallinfo_arena=622592
mallinfo_ordblks=4
mallinfo_smblks=0
mallinfo_hblks=7
mallinfo_hblkhd=14925824
mallinfo_usmblks=0
mallinfo_fsmblks=0
mallinfo_uordblks=498320
mallinfo_fordblks=124272
mallinfo_keepcost=123776

The questions now are

* what among the above gives accurate and simple explanation about memory consumption of glusterfs?
* Should we deprecate mallinfo and just retain malloc_stats and malloc_info outputs? IOW, which among these need to be retained in statedump?

Since I've limited understanding of glibc memory allocator, I am reaching out to the wider community for feedback.

[1] http://udrepper.livejournal.com/20948.html

regards,
Raghavendra

More information about the Gluster-devel mailing list