[Gluster-devel] Need some advices regarding glusterd memory leak upto 120GB

[Jaden Liang]

Hi all,

I am running gluster-3.4.5 on 2 servers. Each of them has 7 2TB HDDs to
build a 7 * 2 distributed + replicated volume.
I just notice that the glusterd consume about 120GB memory and get a
coredump today. I read the mempool code try to identify which mempool eat
the memory. Unfortunetly, the glusterd did not run with
--mem-accounting. Now I just have a coredump file to debug... Anyway, I
read some codes about mem_pool try to identify which mem_pool consumes such
large memory. Here is the result:

I wrote a gdb script to print out the glusterfsd_ctx->mempool_list:

# script of gdb to print out all none-zero mem_pool
set $head = &glusterfsd_ctx->mempool_list
set $offset = (unsigned long)(&((struct mem_pool*)0)->global_list)
set $pos = (struct mem_pool*)((unsigned long)($head->next) - $offset)
set $memsum = 0
while ( &$pos->global_list != $head)
if ($pos->hot_count + $pos->curr_stdalloc)
p *$pos
set $thismempoolsize = ($pos->hot_count + $pos->curr_stdalloc) *
$pos->padded_sizeof_type
# This is the single mem_pool memory consume
p $pos->name
p $thismempoolsize
set $memsum += $thismempoolsize
end
set $pos = (struct mem_pool*)((unsigned long)($pos->global_list.next) -
$offset)
end
echo "Total mem used\n"
p $memsum

Then I got this output:

(gdb) source gdb_show_mempool_list.gdb
$459 = {list = {next = 0x1625a50, prev = 0x1625a50}, hot_count = 64,
cold_count = 0, lock = 1, padded_sizeof_type = 6116, pool = 0x7ff2c9f94010,
pool_end = 0x7ff2c9ff3910, real_sizeof_type = 6088,
  alloc_count = 16919588, pool_misses = 16919096, max_alloc = 64,
curr_stdalloc = 16824653, max_stdalloc = 16824655, name = 0x1625ad0
"management:rpcsvc_request_t", global_list = {next = 0x16211f8,
    prev = 0x1639368}}
$460 = 0x1625ad0 "management:rpcsvc_request_t"
$461 = 102899969172
$462 = {list = {next = 0x7ff2cc0bf374, prev = 0x7ff2cc0bc2b4}, hot_count =
16352, cold_count = 32, lock = 1, padded_sizeof_type = 52, pool =
0x7ff2cc0bc010, pool_end = 0x7ff2cc18c010,
  real_sizeof_type = 24, alloc_count = 169845909, pool_misses = 168448980,
max_alloc = 16384, curr_stdalloc = 168231365, max_stdalloc = 168231560,
name = 0x1621210 "glusterfs:data_t", global_list = {
    next = 0x1621158, prev = 0x1625ab8}}
$463 = 0x1621210 "glusterfs:data_t"
$464 = 8748881284
$465 = {list = {next = 0x7ff2cc18e770, prev = 0x7ff2cc18d2fc}, hot_count =
16350, cold_count = 34, lock = 1, padded_sizeof_type = 68, pool =
0x7ff2cc18d010, pool_end = 0x7ff2cc29d010,
  real_sizeof_type = 40, alloc_count = 152853817, pool_misses = 151477891,
max_alloc = 16384, curr_stdalloc = 151406417, max_stdalloc = 151406601,
name = 0x1621170 "glusterfs:data_pair_t",
  global_list = {next = 0x16210b8, prev = 0x16211f8}}
$466 = 0x1621170 "glusterfs:data_pair_t"
$467 = 10296748156
$468 = {list = {next = 0x1621050, prev = 0x1621050}, hot_count = 4096,
cold_count = 0, lock = 1, padded_sizeof_type = 140, pool = 0x7ff2cc29e010,
pool_end = 0x7ff2cc32a010, real_sizeof_type = 112,
  alloc_count = 16995288, pool_misses = 16986651, max_alloc = 4096,
curr_stdalloc = 16820855, max_stdalloc = 16820882, name = 0x16210d0
"glusterfs:dict_t", global_list = {next = 0x1621018,
    prev = 0x1621158}}
$469 = 0x16210d0 "glusterfs:dict_t"
$470 = 2355493140
"Total mem used
"$471 = 124301091752

--------------------------------------------------------------------------------------
"management:rpcsvc_request_t" used 100G
"glusterfs:data_t" used 8.7GB
"glusterfs:data_pair_t" used 10GB
"glusterfs:dict_t" use 2.3G
Total: 124GB memory

---------------------------------------------------------------------------------------
I assume this might happen in a lot of rpc request and not free.
This happened several days ago, I am still trying to figure out what happen
several days ago on my servers.
Hope someone here might encountered this issue before, or any advices will
be grateful!!!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20141112/7a704d40/attachment-0001.html>