Is there a reason why yarn build takes long inside Docker?

I'm trying to build a nextjs app (yarn build) inside docker but it's taking extremely long 1hr+ to generate static pages. Outside docker, it doesn't take long. I've also noticed it doesn't take this long inside github actions so I'm suspecting it might be something related to my Docker? I'm using Docker version 20.10.14, build a224086 on Mac-OS

Here is My Dockerfile just incase

# Install dependencies only when needed
FROM node:16 AS deps
WORKDIR /app
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile

# Rebuild the source code only when needed
FROM node:16 AS builder

ARG API_KEY \
    API_BASE \
    # Next.js collects completely anonymous telemetry data about general usage.
    # Learn more here: https://nextjs.org/telemetry
    NEXT_TELEMETRY_DISABLED=1

ENV API_KEY=${API_KEY} \
    API_BASE=${API_BASE} \
    NEXT_TELEMETRY_DISABLED=${NEXT_TELEMETRY_DISABLED}

WORKDIR /app

COPY . .
COPY --from=deps /app/node_modules ./node_modules
RUN yarn build

# Production image, copy all the files and run next
FROM node:16 AS runner

ENV NODE_ENV production \
    NEXT_TELEMETRY_DISABLED=1

WORKDIR /app

RUN addgroup --system --gid 1001 nodejs
RUN adduser --system --uid 1001 nextjs

COPY --from=builder /app/next.config.js ./
COPY --from=builder /app/public ./public
COPY --from=builder --chown=nextjs:nodejs /app/.next ./.next
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/package.json ./package.json
COPY --from=builder /app/scripts ./scripts
COPY --from=builder --chown=nextjs:nodejs /app/src/data ./src/data

USER nextjs

EXPOSE 3000

CMD ["yarn", "start"]