I have an issue with a K8S POD getting OOM killed, but with some weird conditions and observations.
The pod is a golang 1.15.6 based REST service, running on X86 64 bit architecture. When the pod runs on VM based clusters, everything is fine, the service behaves normally. When the service runs on nodes provisioned directly on hardware, it appears to experience a memory leak and ends up getting OOMed.
Observations are that when running on the problematic configuration, "kubectl top pod" will report continually increasing memory utilization until the defined limit (64MiB) is reached, at which time OOM killer is invoked.
Observations from inside the pod using "top" suggest that memory usage of the various processes inside the pod are stable, using around 40MiB RSS. The values for VIRT,RES,SHR as reported by top remain stable over time, with only minor fluctuations.
I've analyzed the golang code extensively, including obtaining memory profiles over time (pprof). No sign of a leak in the actual golang code, which tallies with correct operation in VM based environment and observations from top.
The OOM message below also suggests that the total RSS used by the pod was only 38.75MiB (sum/RSS = 9919 pages *4k = 38.75MiB).
kernel: [651076.945552] xxxxxxxxxxxx invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=999
kernel: [651076.945556] CPU: 35 PID: 158127 Comm: xxxxxxxxxxxx Not tainted 5.4.0-73-generic #82~18.04.1
kernel: [651076.945558] Call Trace:
kernel: [651076.945567] dump_stack+0x6d/0x8b
kernel: [651076.945573] dump_header+0x4f/0x200
kernel: [651076.945575] oom_kill_process+0xe6/0x120
kernel: [651076.945577] out_of_memory+0x109/0x510
kernel: [651076.945582] mem_cgroup_out_of_memory+0xbb/0xd0
kernel: [651076.945584] try_charge+0x79a/0x7d0
kernel: [651076.945585] mem_cgroup_try_charge+0x75/0x190
kernel: [651076.945587] __add_to_page_cache_locked+0x1e1/0x340
kernel: [651076.945592] ? scan_shadow_nodes+0x30/0x30
kernel: [651076.945594] add_to_page_cache_lru+0x4f/0xd0
kernel: [651076.945595] pagecache_get_page+0xea/0x2c0
kernel: [651076.945596] filemap_fault+0x685/0xb80
kernel: [651076.945600] ? __switch_to_asm+0x40/0x70
kernel: [651076.945601] ? __switch_to_asm+0x34/0x70
kernel: [651076.945602] ? __switch_to_asm+0x40/0x70
kernel: [651076.945603] ? __switch_to_asm+0x34/0x70
kernel: [651076.945604] ? __switch_to_asm+0x40/0x70
kernel: [651076.945605] ? __switch_to_asm+0x34/0x70
kernel: [651076.945606] ? __switch_to_asm+0x40/0x70
kernel: [651076.945608] ? filemap_map_pages+0x181/0x3b0
kernel: [651076.945611] ext4_filemap_fault+0x31/0x50
kernel: [651076.945614] __do_fault+0x57/0x110
kernel: [651076.945615] __handle_mm_fault+0xdde/0x1270
kernel: [651076.945617] handle_mm_fault+0xcb/0x210
kernel: [651076.945621] __do_page_fault+0x2a1/0x4d0
kernel: [651076.945625] ? __audit_syscall_exit+0x1e8/0x2a0
kernel: [651076.945627] do_page_fault+0x2c/0xe0
kernel: [651076.945628] page_fault+0x34/0x40
kernel: [651076.945630] RIP: 0033:0x5606e773349b
kernel: [651076.945634] Code: Bad RIP value.
kernel: [651076.945635] RSP: 002b:00007fbdf9088df0 EFLAGS: 00010206
kernel: [651076.945637] RAX: 0000000000000000 RBX: 0000000000004e20 RCX: 00005606e775ce7d
kernel: [651076.945637] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00007fbdf9088dd0
kernel: [651076.945638] RBP: 00007fbdf9088e48 R08: 0000000000006c50 R09: 00007fbdf9088dc0
kernel: [651076.945638] R10: 0000000000000000 R11: 0000000000000202 R12: 00007fbdf9088dd0
kernel: [651076.945639] R13: 0000000000000000 R14: 00005606e7c6140c R15: 0000000000000000
kernel: [651076.945640] memory: usage 65536kB, limit 65536kB, failcnt 26279526
kernel: [651076.945641] memory+swap: usage 65536kB, limit 9007199254740988kB, failcnt 0
kernel: [651076.945642] kmem: usage 37468kB, limit 9007199254740988kB, failcnt 0
kernel: [651076.945642] Memory cgroup stats for /kubepods/burstable/pod34ffde14-8e80-4b3a-99ac-910137a04dfe:
kernel: [651076.945652] anon 25112576
kernel: [651076.945652] file 0
kernel: [651076.945652] kernel_stack 221184
kernel: [651076.945652] slab 41406464
kernel: [651076.945652] sock 0
kernel: [651076.945652] shmem 0
kernel: [651076.945652] file_mapped 2838528
kernel: [651076.945652] file_dirty 0
kernel: [651076.945652] file_writeback 0
kernel: [651076.945652] anon_thp 0
kernel: [651076.945652] inactive_anon 0
kernel: [651076.945652] active_anon 25411584
kernel: [651076.945652] inactive_file 0
kernel: [651076.945652] active_file 536576
kernel: [651076.945652] unevictable 0
kernel: [651076.945652] slab_reclaimable 16769024
kernel: [651076.945652] slab_unreclaimable 24637440
kernel: [651076.945652] pgfault 7211542
kernel: [651076.945652] pgmajfault 2895749
kernel: [651076.945652] workingset_refault 71200645
kernel: [651076.945652] workingset_activate 5871824
kernel: [651076.945652] workingset_nodereclaim 330
kernel: [651076.945652] pgrefill 39987763
kernel: [651076.945652] pgscan 144468270
kernel: [651076.945652] pgsteal 71255273
kernel: [651076.945652] pgactivate 27649178
kernel: [651076.945652] pgdeactivate 33525031
kernel: [651076.945653] Tasks state (memory values in pages):
kernel: [651076.945653] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
kernel: [651076.945656] [ 151091] 0 151091 255 1 36864 0 -998 pause
kernel: [651076.945675] [ 157986] 0 157986 58 4 32768 0 999 dumb-init
kernel: [651076.945676] [ 158060] 0 158060 13792 869 151552 0 999 su
kernel: [651076.945678] [ 158061] 1234 158061 18476 6452 192512 0 999 yyyyyy
kernel: [651076.945679] [ 158124] 1234 158124 1161 224 53248 0 999 sh
kernel: [651076.945681] [ 158125] 1234 158125 348755 2369 233472 0 999 xxxxxxxxxxxx
kernel: [651076.945682] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=a0027a4fe415aa7a6ad54aa3fbf553b9af27c61043d08101931e985efeee0ed7,mems_allowed=0-3,oom_memcg=/kubepods/burstable/pod34ffde14-8e80-4b3a-99ac-910137a04dfe,task_memcg=/kubepods/burstable/pod34ffde14-8e80-4b3a-99ac-910137a04dfe/a0027a4fe415aa7a6ad54aa3fbf553b9af27c61043d08101931e985efeee0ed7,task=yyyyyy,pid=158061,uid=1234
kernel: [651076.945695] Memory cgroup out of memory: Killed process 158061 (yyyyyy) total-vm:73904kB, anon-rss:17008kB, file-rss:8800kB, shmem-rss:0kB, UID:1234 pgtables:188kB oom_score_adj:999
kernel: [651076.947429] oom_reaper: reaped process 158061 (yyyyyy), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
The OOM message clearly states that usage = 65536kB, limit = 65536kB, but I don't immediately where the approximately 25MiB of memory not accounted for under RSS has gone.
I see slab_unreclaimable = 24637440, (24MiB), which is approximately the amount of memory that appears to be unaccounted for, not sure if there is any significant in this though.
Looking for any suggestions as to where the memory is being used. Any input would be most welcome.