I have a node app built inside docker. When I try to compile it inside the container, it runs out of memory. I run docker-compose run api npm run build and I get:
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
server |
server | Writing Node.js report to file: report.20200915.155423.48.0.001.json
server | <--- Last few GCs --->
server |
server | [48:0x557b183d4d20] 7413 ms: Scavenge 252.0 (254.4) -> 251.7 (254.9) MB, 1.9 / 0.0 ms (average mu = 0.336, current mu = 0.325) allocation failure
server | [48:0x557b183d4d20] 7416 ms: Scavenge 252.4 (254.9) -> 251.9 (255.4) MB, 1.3 / 0.1 ms (average mu = 0.336, current mu = 0.325) allocation failure
server | [48:0x557b183d4d20] 7425 ms: Scavenge 252.7 (255.4) -> 252.1 (256.4) MB, 7.5 / 0.0 ms (average mu = 0.336, current mu = 0.325) allocation failure
server |
server |
server | <--- JS stacktrace --->
server |
server | ==== JS stack trace =========================================
server |
server | 0: ExitFrame [pc: 0x557b16d8e2e2]
server | 1: StubFrame [pc: 0x557b16d1dfa7]
server | Security context: 0x348fb7f9a299 <JSObject>
server | 2: createNodeArray(aka createNodeArray) [0xed8f52fff49] [/app/node_modules/typescript/lib/tsc.js:~15644] [pc=0x1e5b159af9a1](this=0x0138ef5004d1 <undefined>,0x2eff39aac181 <JSArray[1]>,82415,0x0138ef5004d1 <undefined>)
server | 3: parseStatement(aka parseStatement) [0x34818d601ef9] [/app/node_modules/typescript/lib/tsc.js...
Memory allocated to docker is 12GB, I can confirm this with docker stats:
2aeda754d6f8 graphql-services_api_run_30766f61adf8 0.00% 2.734MiB / 11.7GiB 0.02% 726B / 0B 0B / 0B 1
The process blows up when it approaches just 500mb, which is typical memory usage for typescript compilation:
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 0.08% 444KiB / 11.7GiB 0.00% 746B / 0B 0B / 0B 1
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 52.33% 28.81MiB / 11.7GiB 0.24% 746B / 0B 0B / 0B 19
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 52.33% 28.81MiB / 11.7GiB 0.24% 746B / 0B 0B / 0B 19
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 142.23% 118.7MiB / 11.7GiB 0.99% 746B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 142.23% 118.7MiB / 11.7GiB 0.99% 746B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 158.09% 228.6MiB / 11.7GiB 1.91% 746B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 158.09% 228.6MiB / 11.7GiB 1.91% 746B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 234.13% 337.4MiB / 11.7GiB 2.82% 746B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 234.13% 337.4MiB / 11.7GiB 2.82% 746B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 209.31% 348.4MiB / 11.7GiB 2.91% 746B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 209.31% 348.4MiB / 11.7GiB 2.91% 746B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 24.58% 3.992MiB / 11.7GiB 0.03% 746B / 0B 0B / 0B 1
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 24.58% 3.992MiB / 11.7GiB 0.03% 746B / 0B 0B / 0B 1
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 0.00% 3.828MiB / 11.7GiB 0.03% 746B / 0B 0B / 0B 1
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 0.00% 3.828MiB / 11.7GiB 0.03% 746B / 0B 0B / 0B 1
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 0.00% 3.527MiB / 11.7GiB 0.03% 746B / 0B 0B / 0B 1
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 0.00% 3.527MiB / 11.7GiB 0.03% 746B / 0B 0B / 0B 1
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 0.00% 3.504MiB / 11.7GiB 0.03% 746B / 0B 0B / 0B 1
I've noticed that CPU usage blows up, but not memory?
I've done a tone of looking around, but in all the cases I've found the memory available to docker was limited and the suggestion was to increase it or specify NODE_OPTIONS="--max-old-space-size like here FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory in ionic 3.
Specifying NODE_OPTIONS="--max-old-space-size=512"does the trick, but I don't understand why I need to do it. I'm running node 12.13 which has dynamic heap allocation. For memory ~ 12GB the heap size should be well over the required 512.. If I build it this way, it never goes over 5% ~ 600MB of available memory:
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 0.04% 2.344MiB / 11.7GiB 0.02% 956B / 0B 0B / 0B 1
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 0.04% 2.344MiB / 11.7GiB 0.02% 956B / 0B 0B / 0B 1
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 18.79% 19.18MiB / 11.7GiB 0.16% 956B / 0B 0B / 0B 12
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 18.79% 19.18MiB / 11.7GiB 0.16% 956B / 0B 0B / 0B 12
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 116.90% 84.32MiB / 11.7GiB 0.70% 956B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 116.90% 84.32MiB / 11.7GiB 0.70% 956B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 176.67% 188.2MiB / 11.7GiB 1.57% 956B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 176.67% 188.2MiB / 11.7GiB 1.57% 956B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 150.59% 329.9MiB / 11.7GiB 2.75% 956B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 150.59% 329.9MiB / 11.7GiB 2.75% 956B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 146.39% 417.2MiB / 11.7GiB 3.48% 956B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 146.39% 417.2MiB / 11.7GiB 3.48% 956B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 207.50% 507.9MiB / 11.7GiB 4.24% 956B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 207.50% 507.9MiB / 11.7GiB 4.24% 956B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 193.23% 567.1MiB / 11.7GiB 4.73% 956B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 193.23% 567.1MiB / 11.7GiB 4.73% 956B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 220.27% 567.6MiB / 11.7GiB 4.74% 956B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 220.27% 567.6MiB / 11.7GiB 4.74% 956B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 234.85% 598.1MiB / 11.7GiB 4.99% 956B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 234.85% 598.1MiB / 11.7GiB 4.99% 956B / 0B 0B / 0B 30
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 218.03% 4.875MiB / 11.7GiB 0.04% 956B / 0B 0B / 0B 1
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 218.03% 4.875MiB / 11.7GiB 0.04% 956B / 0B 0B / 0B 1
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 0.00% 4.77MiB / 11.7GiB 0.04% 956B / 0B 0B / 0B 1
cab79aa591a3 graphql-services_api_run_97f2b8f5c605 0.00% 4.77MiB / 11.7GiB 0.04% 956B / 0B 0B / 0B 1
More over, if I just do docker-compose build, it seems to work just find even though my docker file also does npm run build. This leads me to believe that the issue with docker-compose run? Any pointers would be really helpful. My dockerfiles are below.
Dockerfile
# Base is just light node config with common tools to building
# for building npm packages
FROM mhart/alpine-node:12.18.3 AS base
RUN apk add --no-cache gcc python perl-utils g++ make perl-dev tzdata openssh git curl perl-dbd-pg postgresql-client
RUN cpan App::cpanminus
RUN cpanm App::Sqitch --no-wget --notest --quiet
RUN rm -rf /root/.cpan
# Build is the base with the installed npm packages
FROM base AS build
ENV TZ UTC
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
# test creates the build for testing and can be used later for prod
FROM build AS test
WORKDIR /app
COPY . .
RUN chmod +x scripts/wait-for
RUN chmod +x scripts/restore_databases
# THIS WORKS JUST FINE
# When I ssh into the container after it gets built, I see the compiled code and can run it
RUN npm run build
# dev adds an additional dependencies needed for container management
# in our dev environment
FROM test AS dev
RUN apk add netcat-openbsd
WORKDIR /app
# release builds from a lighter node image and copies stuff over
# from tested build
FROM mhart/alpine-node:slim-12.18.3 AS release
ENV TZ UTC
WORKDIR /app
COPY --from=test /app/node_modules /app/build /app/db_changes ./
docker-compose.yml
version: '3.7'
volumes:
pgdata:
external: false
databases:
external: false
services:
api:
container_name: server
build:
context: .
target: dev
env_file:
- .env
volumes:
- ./src:/app/src
- ./db_changes:/app/db_changes
- ./db_install/logs:/app/logs
- ./scripts:/app/scripts
- ./package.json:/app/package.json
- ./tsconfig.json:/app/tsconfig.json
- ./jest.config.js:/app/jest.config.js
ports:
- 5000:5000
- 5001:5001
- 5002:5002
- 5003:5003
command: sh -c "chmod +x ./scripts/wait-for && ./scripts/wait-for db:5432 -- ${COMMAND:-npm run migrate && npm run dev}"
Running: docker-compose run api sh to get into the shell and then npm run build produces OOM. Running NODE_OPTIONS="--max-old-space-size=512" npm run build compiles successfully.
Docker version 19.03.12, build 48a66213fe
docker-compose version 1.27.2, build 18f557f9
node v12.18.3
typescript Version 3.8.3
Thank you!