[Bugs] [Bug 1457936] possible memory leak in glusterfsd with multiplexing

Thu Jun 15 10:24:20 UTC 2017

https://bugzilla.redhat.com/show_bug.cgi?id=1457936

--- Comment #5 from Raghavendra G <rgowdapp at redhat.com> ---
I did a small test by exiting all threads other than epoll-thread. I could add
upto 3000 bricks to a single process and memory consumption was not as high

# gluster volume info testvol
Volume Name: testvol
Type: Distribute
Volume ID: e62a1bae-ac50-4121-a77b-c034c0dec548
Status: Started
Snapshot Count: 0
Number of Bricks: 3001
Transport-type: tcp

top output:
21522 root      20   0 72.9g 2.7g  32m S  0.0 35.3   0:15.11 glusterfsd         

Note that Resident memory consumption is 50% (2.7G as compared to 5.5g earlier)
and virtual memory is less (72.9g compared to 228g earlier)

However, when I tried removing 2000 bricks (as mount was not responsive),
glusterd crashed:

 (gdb) bt
#0  0x0000003866a35935 in __GI_raise (sig=sig at entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x0000003866a370e8 in __GI_abort () at abort.c:91
#2  0x0000003866a74e8b in __libc_message (do_abort=do_abort at entry=2,
fmt=fmt at entry=0x3866b78928 "*** glibc detected *** %s: %s: 0x%s ***\n")
    at ../sysdeps/unix/sysv/linux/libc_fatal.c:198
#3  0x0000003866a7ae16 in malloc_printerr (action=3, str=0x3866b78a60
"munmap_chunk(): invalid pointer", ptr=<optimized out>) at malloc.c:5027
#4  0x0000003866b15c98 in __check_pf (seen_ipv4=seen_ipv4 at entry=0x7f85e0402760,
seen_ipv6=seen_ipv6 at entry=0x7f85e0402770, in6ai=in6ai at entry=0x7f85e04027b0, 
    in6ailen=in6ailen at entry=0x7f85e04027c0) at
../sysdeps/unix/sysv/linux/check_pf.c:367
#5  0x0000003866adcf19 in __GI_getaddrinfo (name=name at entry=0x7f85a8a392f0
"booradley", service=<optimized out>, service at entry=0x0, hints=0x3866b73500, 
    hints at entry=0x0, pai=pai at entry=0x7f85e0402968) at
../sysdeps/posix/getaddrinfo.c:2305
#6  0x00007f85eaeceab9 in glusterd_peerinfo_find_by_hostname
(hoststr=hoststr at entry=0x7f85a8a392f0 "booradley") at glusterd-peer-utils.c:115
#7  0x00007f85eaecedd7 in glusterd_hostname_to_uuid
(hostname=hostname at entry=0x7f85a8a392f0 "booradley",
uuid=uuid at entry=0x7f85e0402a20 "")
    at glusterd-peer-utils.c:155
#8  0x00007f85eae1367f in glusterd_volume_brickinfo_get (uuid=uuid at entry=0x0,
hostname=0x7f85a8a392f0 "booradley", path=0x7f85a8a394d0
"/home/export/testvol/1820", 
    volinfo=volinfo at entry=0x16a51f0, brickinfo=brickinfo at entry=0x7f85e0402a78)
at glusterd-utils.c:1448
#9  0x00007f85eae1384c in glusterd_is_brick_decommissioned
(volinfo=volinfo at entry=0x16a51f0, hostname=<optimized out>, path=<optimized
out>) at glusterd-utils.c:1511
#10 0x00007f85eae59bec in _xl_is_client_decommissioned (xl=0x7f85a8a38490,
volinfo=0x16a51f0) at glusterd-volgen.c:3248
#11 0x00007f85eae59e49 in _graph_get_decommissioned_children
(volinfo=volinfo at entry=0x16a51f0, children=children at entry=0x7f85e0402b58, 
    dht=<error reading variable: Unhandled dwarf expression opcode 0xfa>) at
glusterd-volgen.c:3297
#12 0x00007f85eae5a0c0 in volgen_graph_build_dht_cluster (is_quotad=_gf_false,
child_count=3001, volinfo=0x16a51f0, graph=<optimized out>) at
glusterd-volgen.c:3390
#13 volume_volgen_graph_build_clusters (graph=graph at entry=0x7f85e0402e30,
volinfo=volinfo at entry=0x16a51f0, is_quotad=is_quotad at entry=_gf_false)
    at glusterd-volgen.c:3602
#14 0x00007f85eae5a873 in client_graph_builder (graph=0x7f85e0402e30,
volinfo=0x16a51f0, set_dict=0x7f85b7e7bc80, param=<optimized out>) at
glusterd-volgen.c:3949
#15 0x00007f85eae51e6a in build_graph_generic
(graph=graph at entry=0x7f85e0402e30, volinfo=0x16a51f0, mod_dict=0x7f85b7e80bb0,
param=param at entry=0x0, 
    builder=builder at entry=0x7f85eae5a7d0 <client_graph_builder>) at
glusterd-volgen.c:1064
#16 0x00007f85eae53bfe in build_client_graph (mod_dict=<optimized out>,
volinfo=<optimized out>, graph=0x7f85e0402e30) at glusterd-volgen.c:4167
#17 generate_single_transport_client_volfile (volinfo=volinfo at entry=0x16a51f0, 
    filepath=filepath at entry=0x7f85e0403070
"/var/lib/glusterd/vols/testvol/trusted-testvol.tcp-fuse.vol",
dict=dict at entry=0x7f85b7e80bb0) at glusterd-volgen.c:5362
#18 0x00007f85eae5cf5f in generate_client_volfiles
(volinfo=volinfo at entry=0x16a51f0,
client_type=client_type at entry=GF_CLIENT_TRUSTED) at glusterd-volgen.c:5534
#19 0x00007f85eae5e4ca in glusterd_create_volfiles
(volinfo=volinfo at entry=0x16a51f0) at glusterd-volgen.c:6052
#20 0x00007f85eae5e5d1 in glusterd_create_volfiles_and_notify_services
(volinfo=0x16a51f0) at glusterd-volgen.c:6078
#21 0x00007f85eae9575a in glusterd_op_remove_brick
(dict=dict at entry=0x7f85d13747d0, op_errstr=op_errstr at entry=0x7f85e04061c0) at
glusterd-brick-ops.c:2878
#22 0x00007f85eae04f30 in glusterd_op_commit_perform
(op=op at entry=GD_OP_REMOVE_BRICK, dict=dict at entry=0x7f85d13747d0,
op_errstr=op_errstr at entry=0x7f85e04061c0, 
    rsp_dict=rsp_dict at entry=0x7f85b0706080) at glusterd-op-sm.c:6085
#23 0x00007f85eae9ba36 in gd_commit_op_phase (op=GD_OP_REMOVE_BRICK,
op_ctx=op_ctx at entry=0x7f85d13747d0, req_dict=0x7f85d13747d0, 
    op_errstr=op_errstr at entry=0x7f85e04061c0,
txn_opinfo=txn_opinfo at entry=0x7f85e04061e0) at glusterd-syncop.c:1406
#24 0x00007f85eae9d035 in gd_sync_task_begin
(op_ctx=op_ctx at entry=0x7f85d13747d0, req=req at entry=0x7f85e00d9640) at
glusterd-syncop.c:1926
#25 0x00007f85eae9d31b in glusterd_op_begin_synctask
(req=req at entry=0x7f85e00d9640, op=op at entry=GD_OP_REMOVE_BRICK,
dict=0x7f85d13747d0) at glusterd-syncop.c:1991
#26 0x00007f85eae901d4 in __glusterd_handle_remove_brick
(req=req at entry=0x7f85e00d9640) at glusterd-brick-ops.c:1160
#27 0x00007f85eadee99d in glusterd_big_locked_handler (req=0x7f85e00d9640,
actor_fn=0x7f85eae8f2f0 <__glusterd_handle_remove_brick>) at
glusterd-handler.c:81
#28 0x00007f85f4e8cb12 in synctask_wrap (old_task=<optimized out>) at
syncop.c:375
#29 0x0000003866a46370 in ?? () from /lib64/libc.so.6
#30 0x0000000000000000 in ?? ()

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=tEWerRPygN&a=cc_unsubscribe