[Bugs] [Bug 1457936] possible memory leak in glusterfsd with multiplexing

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1457936



--- Comment #4 from Raghavendra G <rgowdapp at redhat.com> ---
I tried to add 3k bricks to a server process and Resident memory consumption of
the process went upto 5.5g (note that Virt memory consumption is 228g)


10205 root      20   0  228g 5.5g  804 t  0.0 71.9   3:35.32 glusterfsd 

However the command itself failed timing out

On checking the threads associated with server process I found it has 30k+
threads
[root at unused shberry]# ls -lr /proc/10205/task/  | wc -l
30993

I've given different threads and their number below:

[root at unused shberry]# grep start_thread -B1 gdb.txt | grep -v start_thread |
grep -v "\-\-" | grep -v posix_janitor_thread_proc | cut -d" " -f 5 | sort |
uniq
br_stub_signth
br_stub_worker
changelog_ev_connector
changelog_ev_dispatch
event_dispatch_epoll_worker
gf_timer_proc
glusterfs_sigwaiter
index_worker
iot_worker
pool_sweeper
posix_fsyncer
posix_health_check_thread_proc
syncenv_processor

[root at unused shberry]# grep gf_timer_proc ./gdb.txt | wc -l
1
[root at unused shberry]# grep glusterfs_sigwaiter ./gdb.txt | wc -l
1
[root at unused shberry]# grep pool_sweeper ./gdb.txt | wc -l
1
[root at unused shberry]# grep event_dispatch_epoll_worker ./gdb.txt | wc -l
1024
[root at unused shberry]# grep index_worker ./gdb.txt | wc -l
2725
[root at unused shberry]# grep iot_worker ./gdb.txt | wc -l
2723
[root at unused shberry]# grep br_stub_signth ./gdb.txt | wc -l
2723
[root at unused shberry]# grep changelog_ev_connector ./gdb.txt | wc -l
2722
[root at unused shberry]# grep changelog_ev_dispatch ./gdb.txt | wc -l
8166
[root at unused shberry]# grep posix_fsyncer ./gdb.txt | wc -l
5444
[root at unused shberry]# grep posix_janitor_thread_proc ./gdb.txt | wc -l
2722
[root at unused shberry]# grep posix_health_check_thread_proc gdb.txt | wc -l
2722
[root at unused shberry]# grep syncenv_processor gdb.txt | wc -l
16
[root at unused shberry]# grep br_stub_worker gdb.txt | wc -l
2723

Though not sure how much memory these threads accounted for, number of threads
is definitely a factor that affects how many bricks we can multiplex

I could see OOM and thread creation failure messages in server logs.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=JP3pZh1Q9Y&a=cc_unsubscribe