A very strange "JavaScript heap out of memory" issue

user_2483271 0
Viewed 102

I'm attempting to build a Next.js project on CircleCI using an ARM-based machine (specifically, arm.xlarge) which has 32GB of RAM. However, it seems to very quickly exit with an error about lack of memory. I've tried to increase the memory of the Node process all the way up to that of the machine, but with no luck - I receive the same error.

What's particularly strange is that the GCs have a tiny amount of memory utilization (3.5MB), and that it exits in sub-second time!

This build does work, if I swap out the arm resource type for an x86_64 resource type, and build a Docker image for the x86_64 platform instead. Obviously this doesn't cover my needs, but verifies the build works.

What might cause this very strange behaviour? What else can I do to further debug this issue?

What I've tried

  • Changing the Node version, ranging from 14 to 18.8.0
  • Changing the machine resource size
  • Changing the options of Docker, including adding a --memory flag to the extra-build-args field of the build job
  • Removing the yarn.lock
  • Purging the Node cache
  • Deleting the node_modules folder

Relevant resources:

Here's the trace:

#16 [11/11] RUN NODE_OPTIONS="--max_old_space_size=8192" yarn build
#16 sha256:da0777711cb35943386ef4812f2df1ff4932fb5bee717be330a217104e5240c7
#16 0.178 
#16 0.178 <--- Last few GCs --->
#16 0.178 
#16 0.178 [1:0x570e320]      100 ms: Mark-sweep 1.2 (3.5) -> 1.2 (3.5) MB, 2.2 / 0.0 ms  (average mu = 0.818, current mu = 0.007) allocation failure scavenge might not succeed
#16 0.178 [1:0x570e320]      102 ms: Mark-sweep (reduce) 1.2 (3.5) -> 1.2 (3.5) MB, 2.3 / 0.0 ms  (average mu = 0.704, current mu = 0.006) last resort GC in old space requested
#16 0.178 [1:0x570e320]      104 ms: Mark-sweep (reduce) 1.2 (2.5) -> 1.1 (3.5) MB, 2.3 / 0.0 ms  (average mu = 0.549, current mu = 0.010) last resort GC in old space requested
#16 0.178 
#16 0.178 
#16 0.178 <--- JS stacktrace --->
#16 0.178 
#16 0.178 FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
#16 ERROR: process "/bin/sh -c NODE_OPTIONS=\"--max_old_space_size=8192\" yarn build" did not complete successfully: exit code: 132
------
 > [11/11] RUN NODE_OPTIONS="--max_old_space_size=8192" yarn build:
------
Dockerfile:32
--------------------
  31 |     
  32 | >>> RUN NODE_OPTIONS="--max_old_space_size=8192" yarn build
  34 |     
--------------------
error: failed to solve: rpc error: code = Unknown desc = process "/bin/sh -c NODE_OPTIONS=\"--max_old_space_size=8192\"   CI_BUILD=${CI_BUILD} DEPLOYMENT_VERSION=${DEPLOYMENT_VERSION} yarn build" did not complete successfully: exit code: 132

Here's the relevant parts of the Dockerfile:

FROM node:14.18.0-alpine
WORKDIR /app

RUN apk update
RUN apk add python3
RUN apk add git gnupg jq g++ make py3-pip

COPY package.json yarn.lock ./
RUN yarn
RUN yarn add pm2@5.2.0 --global

COPY . .

RUN NODE_OPTIONS="--max_old_space_size=8192" yarn build

EXPOSE 3001

CMD [ "npx", ... ]

and here's the job from CircleCI config.yml:

  build_and_push_image:
    machine:
      image: ubuntu-2004:202101-01
    resource_class: arm.xlarge
    steps:
      - checkout
      - docker/pull:
          images: 'node:16.14.2'
      - aws-ecr/build-and-push-image:
          aws-access-key-id: AWS_ACCESS_KEY_ID
          aws-secret-access-key: AWS_SECRET_ACCESS_KEY
          aws-cli-version: latest
          create-repo: true
          skip-when-tags-exist: true  
          dockerfile: Dockerfile
          extra-build-args: >
            --build-arg GPG_ENCRYPT_PASSPHRASE
            --build-arg BUILD_STAGE
            --build-arg CI_BUILD=true
          platform: linux/arm64/v7
          push-image: true
          registry-id: AWS_ACCOUNT_ID
          region: $AWS_REGION
          repo: $PROJECT_NAME
          tag: $version_tag
circleci yarnpkg docker node.js
2 Answers

After a good few hours of debugging with trial and error, I fixed with the following changes to config.yml:

  build_and_push_image:
    machine:
      image: ubuntu-2004:202101-01
    resource_class: arm.xlarge
    steps:
      - checkout
      - docker/pull:
          images: 'node:16.14.2'
      - aws-ecr/build-and-push-image:
          aws-access-key-id: AWS_ACCESS_KEY_ID
          aws-secret-access-key: AWS_SECRET_ACCESS_KEY
          aws-cli-version: latest
          create-repo: true
          skip-when-tags-exist: true  
          dockerfile: Dockerfile
          extra-build-args: >
            --build-arg GPG_ENCRYPT_PASSPHRASE
            --build-arg BUILD_STAGE
            --build-arg CI_BUILD=true

-         platform: linux/arm64/v7
+         platform: linux/arm64

          push-image: true
          registry-id: AWS_ACCOUNT_ID
          region: $AWS_REGION
          repo: $PROJECT_NAME
          tag: $version_tag

I'm not sure why this works, but here's my best thoughts for others who are blocked by this in the future:

  • Some miconfiguration of the docker buildx command from the CircleCI orb. I'm also noticing that a lot of people are having problems with Docker image caching using the orb, and maybe there's some issue with the command itself (see https://github.com/CircleCI-Public/aws-ecr-orb/issues/202). It looks something like this:
docker context create builder

# This here is interesting - it is using qemu to emulate for builds 
# See https://github.com/tonistiigi/binfmt

docker run --privileged --rm tonistiigi/binfmt --install all
docker --context builder buildx create --use
docker --context builder buildx build \
  -f "${PARAM_PATH}"/"${PARAM_DOCKERFILE}" \
  ${docker_tag_args} \
  --platform "${PARAM_PLATFORM}" \
  --progress plain \
  "$@" \
  "${PARAM_PATH}"
user_24832710

Have you checked the versions of OS and Node? The Versions should match. (x64=x64)

user_160694850
Related