What is the best way to use a different API URL in production for a create-react-app using docker?

Okay so I have been banging my head around for the entire day, but still cannot figure out an elegant solution. I have created a React App which makes some API calls to a Django Server in the backend. I have created a settings.js file, which changes the API urls based on whether the app is running in development or production. The settings.js file looks like below:

let API_SERVER_VAL = '';
let MEDIA_SERVER_VAL = '';
let FRONTEND_SERVER_VAL = ''; 

switch (process.env.NODE_ENV) {
    case 'development':
        API_SERVER_VAL = 'http://localhost:8000';
        MEDIA_SERVER_VAL = 'http://localhost:8000';
        FRONTEND_SERVER_VAL = 'http://localhost:3000';
        break;
    case 'production':
        API_SERVER_VAL = process.env.API_SERVER;
        MEDIA_SERVER_VAL = process.env.API_SERVER;
        FRONTEND_SERVER_VAL = process.env.API_SERVER;
        break;
    default:
        API_SERVER_VAL = 'http://localhost:8000';
        MEDIA_SERVER_VAL = 'http://localhost:8000';
        FRONTEND_SERVER_VAL = 'http://localhost:3000';
        break;
}

export const API_SERVER = API_SERVER_VAL;
export const MEDIA_SERVER = MEDIA_SERVER_VAL;
export const FRONTEND_SERVER = FRONTEND_SERVER_VAL;

export const SESSION_DURATION = 5*3600*1000;

A sample API based on the above is shown below:

import * as settings from '../../settings';
....
axios.post(`${settings.API_SERVER}/api/auth/logout/`, ...)

The Dockerfile looks like below. I am performing a two stage build of the image.

###########
# BUILDER #
###########

# pull official base image
FROM node:12.18.3-alpine3.9 as builder

# set work directory
WORKDIR /usr/src/app

# install dependencies and avoid `node-gyp rebuild` errors
COPY package.json .
RUN apk add --no-cache --virtual .gyp \
        python \
        make \
        g++ \
    && npm install \
    && apk del .gyp

# copy our react project
COPY . .

# perform npm build
ARG API_SERVER
ENV API_SERVER=${API_SERVER}
RUN API_SERVER=${API_SERVER} \ 
  npm run build

#########
# FINAL #
#########

# pull official base image
FROM node:12.18.3-alpine3.9 

# set work directory
WORKDIR /usr/src/app

# install serve - deployment static server suggested by official create-react-app
RUN npm install -g serve

# copy our build files from our builder stage
COPY --from=builder /usr/src/app/build ./build

The docker-compose file looks like below (the react service). Basically the docker-compose react service passes an argument called API_SERVER=127.0.0.1 to the builder stage of the Dockerfile.

version: "3.7"

services:
  django:
    build:
      context: ./backend/app
      dockerfile: Dockerfile.prod
    command: gunicorn mainapp.wsgi:application --bind 0.0.0.0:8000
    volumes:
      - django_static_volume:/usr/src/app/files_static
      - app_media_volume:/usr/src/app/files_media
    expose:
      - 8000
    # ports:
    #   - 8000:8000
    env_file:
      - ./backend/.env.prod
    depends_on:
      - db
  db:
    image: postgres:12.0-alpine
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    env_file:
      - ./backend/.env.prod.db
  react:
    build:
      context: ./frontend/app
      dockerfile: Dockerfile.prod
      args: 
          - API_SERVER=127.0.0.1 
    volumes:
      - react_static_volume:/usr/src/app/build/static
    expose:
      - 3000
    command: serve -s build -l 3000
    depends_on:
      - django
  nginx:
    restart: always
    build: ./nginx
    volumes:
      - django_static_volume:/usr/src/app/django_files/static
      - app_media_volume:/usr/src/app/media
      - react_static_volume:/usr/src/app/react_files/static
    ports:
      - 80:80
      # - 8000:80
    depends_on:
      - react

volumes:
  postgres_data:
  django_static_volume:
  app_media_volume:
  react_static_volume:

I am also checking whether this settings.API_SERVER has been correctly passed from docker by printing it in the console on the home page. So within Home.js I have the below:

console.log(`The environment is ${process.env.NODE_ENV}`);
console.log(`The API server is ${process.env.API_SERVER}`);
console.log(`The settings API server is ${settings.API_SERVER}`);
console.log(`The PUBLIC URL is ${process.env.PUBLIC_URL}`);

However, I am seeing this in the console.

So only the process.env.NODE_ENV is correctly being shown!! I think this is because this is showing the environment variable from the second stage of the image rather than the builder stage.

What am I doing wrong? What is the most elegant way around this with no additional npm-packages required?