I have VM with 8GB of memory (Terraformed), on which there are 2 Docker containers:
- a minimal metrics exporter, of 32MB
- a Bitnami Postgres12 container with my database.
The data has been fully loaded, however during the import I got stuck on a CREATE INDEX statement that adds a simple hash(ID) index to a table that has about 800 million records.
The problem during index creation is that the postgres process grows in size to the point where the host kernel does an OOM kill on it.
The Docker container is configured to use about 6GB (but I've tried lower values as well), using the (Nomad) config parameters:
config {
memory_hard_limit = 6144
...
resources {
memory = 6144
Furthermore, vm.overcommit_memory = 1 (this was the default 0 before)
and vm.overcommit_ratio = 100 (this was the default 50 before).
The Docker CLI correctly shows the memory limit:
# more /sys/fs/cgroup/memory/memory.limit_in_bytes
6442450944
However when Postgres does the CREATE INDEX, it eventually gets killed for OOM:
Oct 1 15:17:46 prod kernel: [11515.764300] postgres invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=0
Oct 1 15:17:46 prod kernel: [11515.764302] postgres cpuset=d562f548ef35884fb77a41e53466a18205d63cce3d9ffa8a808d2e95b5609e3e mems_allowed=0
Oct 1 15:17:46 prod kernel: [11515.764308] CPU: 1 PID: 18485 Comm: postgres Not tainted 4.15.0-118-generic #119-Ubuntu
Oct 1 15:17:46 prod kernel: [11515.764309] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.10.2-1ubuntu1 04/01/2014
Oct 1 15:17:46 prod kernel: [11515.764309] Call Trace:
Oct 1 15:17:46 prod kernel: [11515.764331] dump_stack+0x6d/0x8e
Oct 1 15:17:46 prod kernel: [11515.764335] dump_header+0x71/0x285
Oct 1 15:17:46 prod kernel: [11515.764337] oom_kill_process+0x21f/0x420
Oct 1 15:17:46 prod kernel: [11515.764339] out_of_memory+0x116/0x4e0
Oct 1 15:17:46 prod kernel: [11515.764343] mem_cgroup_out_of_memory+0xbb/0xd0
Oct 1 15:17:46 prod kernel: [11515.764345] mem_cgroup_oom_synchronize+0x2e8/0x320
Oct 1 15:17:46 prod kernel: [11515.764347] ? mem_cgroup_css_reset+0xe0/0xe0
Oct 1 15:17:46 prod kernel: [11515.764349] pagefault_out_of_memory+0x36/0x7b
Oct 1 15:17:46 prod kernel: [11515.764354] mm_fault_error+0x90/0x180
Oct 1 15:17:46 prod kernel: [11515.764355] __do_page_fault+0x46b/0x4b0
Oct 1 15:17:46 prod kernel: [11515.764357] do_page_fault+0x2e/0xe0
Oct 1 15:17:46 prod kernel: [11515.764363] ? async_page_fault+0x2f/0x50
Oct 1 15:17:46 prod kernel: [11515.764366] do_async_page_fault+0x51/0x80
Oct 1 15:17:46 prod kernel: [11515.764367] async_page_fault+0x45/0x50
Oct 1 15:17:46 prod kernel: [11515.764369] RIP: 0033:0x562aff7df26d
Oct 1 15:17:46 prod kernel: [11515.764370] RSP: 002b:00007ffef148b790 EFLAGS: 00010206
[...]
Oct 1 15:17:46 prod kernel: [11515.764374] R13: 0000000000000195 R14: 0000000000000000 R15: 0000000000000000
Oct 1 15:17:46 prod kernel: [11515.764376] Task in /docker/d562f548ef35884fb77a41e53466a18205d63cce3d9ffa8a808d2e95b5609e3e killed as a result of limit of /docker/d562f548ef35884fb77a41e53466a18205d63cce3d9ffa8a808d2e95b5609e3e
Oct 1 15:17:46 prod kernel: [11515.764382] memory: usage 6291456kB, limit 6291456kB, failcnt 6210203
Oct 1 15:17:46 prod kernel: [11515.764383] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
Oct 1 15:17:46 prod kernel: [11515.764384] kmem: usage 29264kB, limit 9007199254740988kB, failcnt 0
Oct 1 15:17:46 prod kernel: [11515.764384] Memory cgroup stats for /docker/d562f548ef35884fb77a41e53466a18205d63cce3d9ffa8a808d2e95b5609e3e: cache:1628316KB rss:4633876KB rss_huge:0KB shmem:1628260KB mapped_file:1628272KB dirty:0KB writeback:0KB inactive_anon:1617900KB active_anon:4644212KB inactive_file:0KB active_file:0KB unevictable:0KB
Oct 1 15:17:46 prod kernel: [11515.764391] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Oct 1 15:17:46 prod kernel: [11515.764512] [18146] 1001 18146 424925 16612 262144 0 0 postgres
Oct 1 15:17:46 prod kernel: [11515.764525] [18341] 1001 18341 425526 24841 954368 0 0 postgres
Oct 1 15:17:46 prod kernel: [11515.764527] [18342] 1001 18342 424958 388655 3293184 0 0 postgres
Oct 1 15:17:46 prod kernel: [11515.764529] [18343] 1001 18343 424958 5540 180224 0 0 postgres
Oct 1 15:17:46 prod kernel: [11515.764530] [18344] 1001 18344 425071 2106 180224 0 0 postgres
Oct 1 15:17:46 prod kernel: [11515.764531] [18345] 1001 18345 16582 1156 135168 0 0 postgres
Oct 1 15:17:46 prod kernel: [11515.764533] [18346] 1001 18346 425062 1743 159744 0 0 postgres
Oct 1 15:17:46 prod kernel: [11515.764537] [18418] 1001 18418 425631 5605 196608 0 0 postgres
Oct 1 15:17:46 prod kernel: [11515.764540] [18483] 1001 18483 428502 7794 225280 0 0 postgres
Oct 1 15:17:46 prod kernel: [11515.764541] [18485] 1001 18485 1578423 1561820 12652544 0 0 postgres
Oct 1 15:17:46 prod kernel: [11515.764545] Memory cgroup out of memory: Kill process 18485 (postgres) score 994 or sacrifice child
Oct 1 15:17:46 prod kernel: [11515.767788] Killed process 18485 (postgres) total-vm:6313692kB, anon-rss:4614856kB, file-rss:11372kB, shmem-rss:1621052kB
Oct 1 15:17:47 prod kernel: [11516.011553] oom_reaper: reaped process 18485 (postgres), now anon-rss:0kB, file-rss:0kB, shmem-rss:1621052kB
I've tried various combinations of memory sizing, both on the host, Docker config, and Postgres memory settings, however every time memory usage just keeps going up until the oom-killer off's it.
Why does Postgres not seem to play nice and stay within - what I believe - to be it's allowed memory allocation?