How to specify environment variables with docker image in docker compose

I've got a workflow where I build a specific image and then (after pushing to an ECR repo and then pulling it onto an AWS server) essentially run it with a docker-compose file. My docker compose file looks as follows:

version: "3.8"
services:
  web:
    image: <my-aws-server>/my-repo:latest
    command: gunicorn vms.wsgi:application --bind 0.0.0.0:8000
    expose:
      - 8000
  nginx:
    build: ../nginx
    ports:
      - 1337:80
    depends_on: 
      - web

and my dockerfile is something like this:

FROM python:3
ENV PYTHONUNBUFFERED=1
WORKDIR /code
COPY requirements.txt /code/
RUN pip install -r requirements.txt
COPY . /code/
EXPOSE 8000
CMD [ "python", "manage.py", "runserver", "0.0.0.0:8000"]

I'd like to be able to do something like this in my docker-compose:

version: "3.8"
services:
  web:
    image: <my-aws-server>/my-repo:latest
    env: SECRET_PASSWORD #note change here
    command: gunicorn vms.wsgi:application --bind 0.0.0.0:8000
    expose:
      - 8000
  nginx:
    build: ../nginx
    ports:
      - 1337:80
    depends_on: 
      - web

where I specify the environment variables, which are stored in a file on the server. Is there any way I can do this? Perhaps it's impossible if the image file is just a binary.

Or do I have to actually pass in the environment variables from the get-go, when I build the image in my GitHub action, here:

    steps:
      - uses: actions/checkout@v2
        name: Check out code

      - uses: mr-smithers-excellent/docker-build-push@v5
        name: Build & Push Docker image
        with:
          image: my-image
          registry: ${{ secrets.AWS_ECR_REGISTRY }}
          tags: latest
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

Edit: in my GitHub actions, I tried something like this:

      - name: Start new container
        run: ssh staging 'cd my_dir; sudo docker-compose --env-file ~/code/secrets/.env -f docker-compose.prod.yml up -d'

but that didn't seem to work. Is there something I'm doing wrong there? Or should that have worked as expected where whatever environment variables are in that file will be used in the pre-built image? (I'm not building it again, just starting the image, as is evident from the docker compose file).