[Gluster-users] Out of memory: Kill process 1447 (glusterfs) score 453 or sacrifice child
De Blende Tom
tom at cloudar.be
Mon Feb 29 10:39:12 UTC 2016
Hi all,
We have a two node replicated gluster cluster. There are two clients that each mount one share using Fuse client. On one of these clients, the gluster process was killed with this nice log:
Feb 27 14:03:46 client kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Feb 27 14:03:46 client kernel: java cpuset=c02e902ddefa0c92d05791530938caac1867444e608d9f8531af96f206a20bb0 mems_allowed=0
Feb 27 14:03:46 client kernel: Pid: 32427, comm: java Not tainted 2.6.32-573.18.1.el6.x86_64 #1
Feb 27 14:03:46 client kernel: Call Trace:
Feb 27 14:03:46 client kernel: [<ffffffff810d6d71>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Feb 27 14:03:46 client kernel: [<ffffffff8112a570>] ? dump_header+0x90/0x1b0
Feb 27 14:03:46 client kernel: [<ffffffff8123320c>] ? security_real_capable_noaudit+0x3c/0x70
Feb 27 14:03:46 client kernel: [<ffffffff8112a9f2>] ? oom_kill_process+0x82/0x2a0
Feb 27 14:03:46 client kernel: [<ffffffff8112a931>] ? select_bad_process+0xe1/0x120
Feb 27 14:03:46 client kernel: [<ffffffff8112ae30>] ? out_of_memory+0x220/0x3c0
Feb 27 14:03:46 client kernel: [<ffffffff8113780c>] ? __alloc_pages_nodemask+0x93c/0x950
Feb 27 14:03:46 client kernel: [<ffffffffa008f560>] ? ext4_get_block+0x0/0x120 [ext4]
Feb 27 14:03:46 client kernel: [<ffffffff8117058a>] ? alloc_pages_current+0xaa/0x110
Feb 27 14:03:46 client kernel: [<ffffffff81127967>] ? __page_cache_alloc+0x87/0x90
Feb 27 14:03:46 client kernel: [<ffffffff8112734e>] ? find_get_page+0x1e/0xa0
Feb 27 14:03:46 client kernel: [<ffffffff81128907>] ? filemap_fault+0x1a7/0x500
Feb 27 14:03:46 client kernel: [<ffffffff81151ee4>] ? __do_fault+0x54/0x530
Feb 27 14:03:46 client kernel: [<ffffffff81458ee9>] ? sock_common_recvmsg+0x39/0x50
Feb 27 14:03:46 client kernel: [<ffffffff811524b7>] ? handle_pte_fault+0xf7/0xb20
Feb 27 14:03:46 client kernel: [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40
Feb 27 14:03:46 client kernel: [<ffffffff81153179>] ? handle_mm_fault+0x299/0x3d0
Feb 27 14:03:46 client kernel: [<ffffffff8104f156>] ? __do_page_fault+0x146/0x500
Feb 27 14:03:46 client kernel: [<ffffffff814586db>] ? sys_recvfrom+0x16b/0x180
Feb 27 14:03:46 client kernel: [<ffffffff81007ca9>] ? xen_clocksource_get_cycles+0x9/0x10
Feb 27 14:03:46 client kernel: [<ffffffff8153f48e>] ? do_page_fault+0x3e/0xa0
Feb 27 14:03:46 client kernel: [<ffffffff8153c835>] ? page_fault+0x25/0x30
Feb 27 14:03:46 client kernel: Mem-Info:
Feb 27 14:03:46 client kernel: Node 0 DMA per-cpu:
Feb 27 14:03:46 client kernel: CPU 0: hi: 0, btch: 1 usd: 0
Feb 27 14:03:46 client kernel: CPU 1: hi: 0, btch: 1 usd: 0
Feb 27 14:03:46 client kernel: CPU 2: hi: 0, btch: 1 usd: 0
Feb 27 14:03:46 client kernel: CPU 3: hi: 0, btch: 1 usd: 0
Feb 27 14:03:46 client kernel: CPU 4: hi: 0, btch: 1 usd: 0
Feb 27 14:03:46 client kernel: CPU 5: hi: 0, btch: 1 usd: 0
Feb 27 14:03:46 client kernel: CPU 6: hi: 0, btch: 1 usd: 0
Feb 27 14:03:46 client kernel: CPU 7: hi: 0, btch: 1 usd: 0
Feb 27 14:03:46 client kernel: Node 0 DMA32 per-cpu:
Feb 27 14:03:46 client kernel: CPU 0: hi: 186, btch: 31 usd: 0
Feb 27 14:03:46 client kernel: CPU 1: hi: 186, btch: 31 usd: 0
Feb 27 14:03:46 client kernel: CPU 2: hi: 186, btch: 31 usd: 0
Feb 27 14:03:46 client kernel: CPU 3: hi: 186, btch: 31 usd: 0
Feb 27 14:03:46 client kernel: CPU 4: hi: 186, btch: 31 usd: 0
Feb 27 14:03:46 client kernel: CPU 5: hi: 186, btch: 31 usd: 0
Feb 27 14:03:46 client kernel: CPU 6: hi: 186, btch: 31 usd: 0
Feb 27 14:03:46 client kernel: CPU 7: hi: 186, btch: 31 usd: 0
Feb 27 14:03:46 client kernel: Node 0 Normal per-cpu:
Feb 27 14:03:46 client kernel: CPU 0: hi: 186, btch: 31 usd: 24
Feb 27 14:03:46 client kernel: CPU 1: hi: 186, btch: 31 usd: 0
Feb 27 14:03:46 client kernel: CPU 2: hi: 186, btch: 31 usd: 21
Feb 27 14:03:46 client kernel: CPU 3: hi: 186, btch: 31 usd: 0
Feb 27 14:03:46 client kernel: CPU 4: hi: 186, btch: 31 usd: 48
Feb 27 14:03:46 client kernel: CPU 5: hi: 186, btch: 31 usd: 164
Feb 27 14:03:46 client kernel: CPU 6: hi: 186, btch: 31 usd: 18
Feb 27 14:03:46 client kernel: CPU 7: hi: 186, btch: 31 usd: 10
Feb 27 14:03:46 client kernel: active_anon:8061669 inactive_anon:3 isolated_anon:0
Feb 27 14:03:46 client kernel: active_file:125 inactive_file:7110 isolated_file:0
Feb 27 14:03:46 client kernel: unevictable:0 dirty:17 writeback:16 unstable:0
Feb 27 14:03:46 client kernel: free:49585 slab_reclaimable:3251 slab_unreclaimable:23727
Feb 27 14:03:46 client kernel: mapped:593 shmem:38 pagetables:17542 bounce:0
Feb 27 14:03:46 client kernel: Node 0 DMA free:15628kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15232kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Feb 27 14:03:46 client kernel: lowmem_reserve[]: 0 3768 32300 32300
Feb 27 14:03:46 client kernel: Node 0 DMA32 free:121860kB min:7880kB low:9848kB high:11820kB active_anon:3101704kB inactive_anon:0kB active_file:184kB inactive_file:388kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3858656kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:236kB slab_unreclaimable:9036kB kernel_stack:496kB pagetables:5164kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Feb 27 14:03:46 client kernel: lowmem_reserve[]: 0 0 28532 28532
Feb 27 14:03:47 client kernel: Node 0 Normal free:60852kB min:59672kB low:74588kB high:89508kB active_anon:29144972kB inactive_anon:12kB active_file:316kB inactive_file:28752kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:29217280kB mlocked:0kB dirty:68kB writeback:64kB mapped:2512kB shmem:152kB slab_reclaimable:12768kB slab_unreclaimable:85872kB kernel_stack:7280kB pagetables:65004kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no
Feb 27 14:03:47 client kernel: lowmem_reserve[]: 0 0 0 0
Feb 27 14:03:47 client kernel: Node 0 DMA: 1*4kB 1*8kB 2*16kB 1*32kB 1*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15628kB
Feb 27 14:03:47 client kernel: Node 0 DMA32: 380*4kB 368*8kB 104*16kB 226*32kB 257*64kB 196*128kB 95*256kB 54*512kB 15*1024kB 0*2048kB 0*4096kB = 122224kB
Feb 27 14:03:47 client kernel: Node 0 Normal: 15933*4kB 74*8kB 7*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 64436kB
Feb 27 14:03:47 client kernel: 5988 total pagecache pages
Feb 27 14:03:47 client kernel: 0 pages in swap cache
Feb 27 14:03:47 client kernel: Swap cache stats: add 0, delete 0, find 0/0
Feb 27 14:03:47 client kernel: Free swap = 0kB
Feb 27 14:03:47 client kernel: Total swap = 0kB
Feb 27 14:03:47 client kernel: 8388607 pages RAM
Feb 27 14:03:47 client kernel: 169218 pages reserved
Feb 27 14:03:47 client kernel: 1204 pages shared
Feb 27 14:03:47 client kernel: 8158288 pages non-shared
Feb 27 14:03:47 client kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Feb 27 14:03:47 client kernel: [ 522] 0 522 2812 253 0 -17 -1000 udevd
Feb 27 14:03:47 client kernel: [ 1157] 0 1157 2280 124 3 0 0 dhclient
Feb 27 14:03:47 client kernel: [ 1204] 0 1204 23283 67 1 -17 -1000 auditd
Feb 27 14:03:47 client kernel: [ 1238] 0 1238 62338 616 3 0 0 rsyslogd
Feb 27 14:03:47 client kernel: [ 1261] 0 1261 4596 76 3 0 0 irqbalance
Feb 27 14:03:47 client kernel: [ 1279] 32 1279 4744 65 3 0 0 rpcbind
Feb 27 14:03:47 client kernel: [ 1301] 29 1301 5837 113 0 0 0 rpc.statd
Feb 27 14:03:47 client kernel: [ 1335] 81 1335 24335 89 4 0 0 dbus-daemon
Feb 27 14:03:47 client kernel: [ 1357] 0 1357 47235 221 0 0 0 cupsd
Feb 27 14:03:47 client kernel: [ 1447] 0 1447 4105054 3969537 0 0 0 glusterfs
Feb 27 14:03:47 client kernel: [ 1471] 0 1471 1020 28 4 0 0 acpid
Feb 27 14:03:47 client kernel: [ 1483] 68 1483 9458 171 3 0 0 hald
Feb 27 14:03:47 client kernel: [ 1484] 0 1484 5100 51 1 0 0 hald-runner
Feb 27 14:03:47 client kernel: [ 1513] 0 1513 5630 46 4 0 0 hald-addon-inpu
Feb 27 14:03:47 client kernel: [ 1521] 68 1521 4502 42 5 0 0 hald-addon-acpi
Feb 27 14:03:47 client kernel: [ 1545] 0 1545 96535 637 0 0 0 automount
Feb 27 14:03:47 client kernel: [ 1669] 0 1669 16556 177 0 -17 -1000 sshd
Feb 27 14:03:47 client kernel: [ 1748] 0 1748 20217 225 3 0 0 master
Feb 27 14:03:47 client kernel: [ 1759] 89 1759 20280 220 3 0 0 qmgr
Feb 27 14:03:47 client kernel: [ 1777] 0 1777 45233 245 3 0 0 abrtd
Feb 27 14:03:47 client kernel: [ 1789] 0 1789 29216 156 3 0 0 crond
Feb 27 14:03:47 client kernel: [ 1805] 0 1805 5276 46 3 0 0 atd
Feb 27 14:03:47 client kernel: [ 1837] 0 1837 319752 4831 1 0 0 docker
Feb 27 14:03:47 client kernel: [ 2107] 0 2107 27085 39 3 0 0 rhsmcertd
Feb 27 14:03:47 client kernel: [ 2201] 0 2201 16081 171 3 0 0 certmonger
Feb 27 14:03:47 client kernel: [ 2257] 0 2257 1016 21 7 0 0 mingetty
Feb 27 14:03:47 client kernel: [ 2259] 0 2259 1016 22 3 0 0 mingetty
Feb 27 14:03:47 client kernel: [ 2261] 0 2261 1016 22 3 0 0 mingetty
Feb 27 14:03:47 client kernel: [ 2263] 0 2263 1016 21 3 0 0 mingetty
Feb 27 14:03:47 client kernel: [ 2265] 0 2265 1016 21 3 0 0 mingetty
Feb 27 14:03:47 client kernel: [ 2267] 0 2267 1016 20 3 0 0 mingetty
Feb 27 14:03:47 client kernel: [ 4985] 0 4985 26827 37 6 0 0 rpc.rquotad
Feb 27 14:03:47 client kernel: [ 4990] 0 4990 5443 159 3 0 0 rpc.mountd
Feb 27 14:03:47 client kernel: [ 5037] 0 5037 5774 60 0 0 0 rpc.idmapd
Feb 27 14:03:47 client kernel: [ 5619] 0 5619 29660 144 3 0 0 screen
Feb 27 14:03:47 client kernel: [ 5620] 0 5620 27085 101 0 0 0 bash
Feb 27 14:03:47 client kernel: [16661] 500 16661 29659 157 1 0 0 screen
Feb 27 14:03:47 client kernel: [16662] 500 16662 27118 116 0 0 0 bash
Feb 27 14:03:47 client kernel: [ 6355] 0 6355 2826 251 0 -17 -1000 udevd
Feb 27 14:03:47 client kernel: [27671] 0 27671 35786 411 7 0 0 docker
Feb 27 14:03:47 client kernel: [27683] 0 27683 2811 252 0 -17 -1000 udevd
Feb 27 14:03:47 client kernel: [27688] 0 27688 33225 347 7 0 0 docker
Feb 27 14:03:47 client kernel: [27695] 0 27695 33497 366 3 0 0 docker
Feb 27 14:03:47 client kernel: [27744] 0 27744 11766 104 6 0 0 sudo
Feb 27 14:03:47 client kernel: [27748] 1000 27748 4489 76 6 0 0 init.sh
Feb 27 14:03:47 client kernel: [27818] 1000 27818 2325695 1157905 7 0 0 java
Feb 27 14:03:47 client kernel: [27877] 1000 27877 1082 39 6 0 0 tail
Feb 27 14:03:47 client kernel: [32337] 0 32337 11766 103 0 0 0 sudo
Feb 27 14:03:47 client kernel: [32342] 1000 32342 4485 72 0 0 0 init.sh
Feb 27 14:03:47 client kernel: [32381] 1000 32381 3824856 2912518 4 0 0 java
Feb 27 14:03:47 client kernel: [32411] 1000 32411 1082 38 2 0 0 tail
Feb 27 14:03:47 client kernel: [ 571] 1000 571 211038 5419 6 0 0 soffice.bin
Feb 27 14:03:47 client kernel: [ 626] 1000 626 210839 5347 7 0 0 soffice.bin
Feb 27 14:03:47 client kernel: [ 2124] 89 2124 20237 220 3 0 0 pickup
Feb 27 14:03:47 client kernel: Out of memory: Kill process 1447 (glusterfs) score 453 or sacrifice child
Feb 27 14:03:47 client kernel: Killed process 1447, UID 0, (glusterfs) total-vm:16420216kB, anon-rss:15877020kB, file-rss:1192kB
Any ideas what caused this or how we can prevent this from happening?
Thanks,
Tom
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160229/18c7d4fb/attachment.html>
More information about the Gluster-users
mailing list