Pass variables from .env file to dockerfile through docker-compose

My solution was annoying which is why it took me so long to figure it out. My dockerfile was using powershell on a windows server, so I had to do this for every argument:

ARG ARG1
RUN echo $env:ARG1

This seems pretty niche especially since using windows containers on a windows server is not my first choice, so check out @ErikMD 's answer if your having issues with env files and whatnot.

I'm posting a new answer to highlight the various assumptions related to the OP's question, in particular, the fact that there's a subtle difference between the ".env" unique filename and *.env files (arguments for env_file:).

But apart from this subtlety, the process to pass arguments from docker-compose.yml to docker build -f Dockerfile . and/or docker run -e … is easy, as shown by the comprehensive example below.

Minimal working example

Let's consider the following files in a given directory, say ./docker.

File docker-compose.yml:

services:
  demo-1:
    image: demo-${ENV_NUM}
    build:
      context: .
      args:
        ARG1: "demo-1/${ARG1}"
        ARG3: "demo-1/${ARG3}"
  demo-2:
    image: demo-2${ENV_FILE_NUM}
    build:
      context: .
      args:
        ARG1: "demo-2/${ARG1}"
        ARG3: "demo-2/${ARG3}"
    env_file:
      - var.env

Remark: even if we use a build: field, it appears to be a good idea to also add an image: field to automatically tag the built image; but note that these image names must be pairwise different.

File .env:

KEY="some value"
ENV_NUM=1
ARG1=.env/ARG1
ARG2=.env/ARG2
ARG3=.env/ARG3

File var.env:

ENV_FILE_NUM="some number"
ARG1=var.env/ARG1
ARG2=var.env/ARG2
ARG3=var.env/ARG3
ARG4=var.env/ARG4

File Dockerfile:

FROM debian:10

# Read build arguments (default value if omitted at CLI)
ARG ARG1="default 1"
ARG ARG2="default 2"
ARG ARG3="default 3"

# the build args are exported at build time
RUN echo "ARG1=${ARG1}" | tee /root/arg1.txt
RUN echo "ARG2=${ARG2}" | tee /root/arg2.txt
RUN echo "ARG3=${ARG3}" | tee /root/arg3.txt

# Export part of these args at runtime also
ENV ARG1="${ARG1}"
ENV ARG2="${ARG2}"

# exec-form is mandatory for ENTRYPOINT/CMD
CMD ["/bin/bash", "-c", "echo ARG1=\"${ARG1}\" ARG2=\"${ARG2}\" ARG3=\"${ARG3}\"; echo while at build time:; cat /root/arg{1,2,3}.txt"]

Experiment session 1

First, as suggested by @SergioSantiago in the comments, a very handy command to preview the effective docker-compose.yml file after interpolation is docker-compose config:

$ docker-compose config

WARN[0000] The "ENV_FILE_NUM" variable is not set. Defaulting to a blank string. 
name: docker
services:
  demo-1:
    build:
      context: /home/debian/docker
      dockerfile: Dockerfile
      args:
        ARG1: demo-1/.env/ARG1
        ARG3: demo-1/.env/ARG3
    image: demo-1
    networks:
      default: null
  demo-2:
    build:
      context: /home/debian/docker
      dockerfile: Dockerfile
      args:
        ARG1: demo-2/.env/ARG1
        ARG3: demo-2/.env/ARG3
    environment:
      ARG1: var.env/ARG1
      ARG2: var.env/ARG2
      ARG3: var.env/ARG3
      ARG4: var.env/ARG4
      ENV_FILE_NUM: some number
    image: demo-2
    networks:
      default: null
networks:
  default:
    name: docker_default

Here, as indicated by the warning, we see there's an issue for interpolating ENV_FILE_NUM despite the fact this variable is mentioned by var.env. The reason is that env_files lines just add new environment variables for the underlying docker run -e … command, but don't interpolate anything in the docker-compose.yml.

Contrarily, one can notice that the value ARG1=.env/ARG1 taken from ".env" is interpolated within the args: field of docker-compose.yml, cf. the output line:

args:
  ARG1: demo-1/.env/ARG1
  …

This very distinct semantics of ".env" vs. env_files is described in this page of the official documentation.

Experiment session 2

Next, let us run:

$ docker-compose up --build
                                                                                         
WARN[0000] The "ENV_FILE_NUM" variable is not set. Defaulting to a blank string.                                                                     
[+] Building 10.4s (13/13) FINISHED
 => [demo-1 internal] load build definition from Dockerfile
 => => transferring dockerfile: 609B
 => [demo-2 internal] load build definition from Dockerfile
 => => transferring dockerfile: 609B
 => [demo-1 internal] load .dockerignore
 => => transferring context: 2B
 => [demo-2 internal] load .dockerignore
 => => transferring context: 2B
 => [demo-2 internal] load metadata for docker.io/library/debian:10
 => [demo-2 1/4] FROM docker.io/library/debian:10@sha256:ebe4b9831fb22dfa778de4ffcb8ea0ad69b5d782d4e86cab14cc1fded5d8e761
 => => resolve docker.io/library/debian:10@sha256:ebe4b9831fb22dfa778de4ffcb8ea0ad69b5d782d4e86cab14cc1fded5d8e761
 => => sha256:85bed84afb9a834cf090b55d2e584abd55b4792d93b750db896f486680638344 50.44MB / 50.44MB
 => => sha256:ebe4b9831fb22dfa778de4ffcb8ea0ad69b5d782d4e86cab14cc1fded5d8e761 1.85kB / 1.85kB
 => => sha256:40dd1c1b1c36eac161ab63b6ce3a57d56ad79a667a37717a31721bac3f30aaf9 529B / 529B
 => => sha256:26a2b081e03207d26a105340161109ba0f00e857cbb0ff85aaeeeadd46b709c5 1.46kB / 1.46kB
 => => extracting sha256:85bed84afb9a834cf090b55d2e584abd55b4792d93b750db896f486680638344
 => [demo-2 2/4] RUN echo "ARG1=demo-2/.env/ARG1" | tee /root/arg1.txt
 => [demo-1 2/4] RUN echo "ARG1=demo-1/.env/ARG1" | tee /root/arg1.txt
 => [demo-1 3/4] RUN echo "ARG2=default 2" | tee /root/arg2.txt
 => [demo-2 3/4] RUN echo "ARG2=default 2" | tee /root/arg2.txt
 => [demo-2 4/4] RUN echo "ARG3=demo-2/.env/ARG3" | tee /root/arg3.txt
 => [demo-1 4/4] RUN echo "ARG3=demo-1/.env/ARG3" | tee /root/arg3.txt
 => [demo-2] exporting to image
 => => exporting layers
 => => writing image sha256:553f294a410ceeb3c0ac9d252d443710c804d3f7437ad7fffa586967517f5e7a
 => => naming to docker.io/library/demo-1
 => => writing image sha256:84bb2bd0ffae67ffed0e74efbf9253b6d634a6f37c6f99bc4eedea81846a9352
 => => naming to docker.io/library/demo-2
                                     
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[+] Running 3/3              
 ⠿ Network docker_default     Created
 ⠿ Container docker-demo-1-1  Created
 ⠿ Container docker-demo-2-1  Created

Attaching to docker-demo-1-1, docker-demo-2-1

docker-demo-1-1  | ARG1=demo-1/.env/ARG1 ARG2=default 2 ARG3=
docker-demo-1-1  | while at build time:
docker-demo-1-1  | ARG1=demo-1/.env/ARG1
docker-demo-1-1  | ARG2=default 2
docker-demo-1-1  | ARG3=demo-1/.env/ARG3

docker-demo-2-1  | ARG1=var.env/ARG1 ARG2=var.env/ARG2 ARG3=var.env/ARG3
docker-demo-2-1  | while at build time:
docker-demo-2-1  | ARG1=demo-2/.env/ARG1
docker-demo-2-1  | ARG2=default 2
docker-demo-2-1  | ARG3=demo-2/.env/ARG3

docker-demo-1-1 exited with code 0
docker-demo-2-1 exited with code 0

Here, we can see again that the ".env" values and those of file_env: [ filename.env ] play different roles that don't overlap.

Furthermore:

• Given the absence of a Dockerfile command line ENV ARG3="${ARG3}", the value of build-arg ARG3 is not propagated at runtime (see the ARG3= line in the output above).
• But the value can be exported at runtime anyway if it is defined/overidden in the environment: or env_file: sections in the docker-compose.yml file (see the ARG3=var.env/ARG3 line in the output above).

For more details, see the documentation of the ARG directive.

Remarks on the docker-compose --env-file option use-case

As mentioned by the OP, docker-compose also enjoys a useful CLI option --enf-file (which is precisely named the same way as the very different env-file: field, which is unfortunate, but nevermind).

This option allows for the following use-case (excerpt from OP's code):

File docker-compose.yml:

services:
  home:
    image: home-${ENV_NUM}
    build:
      args:
        ARG1: "${ARG1}"
      ...
    labels:
      - traefik.http.routers.home.rule=Host(`${DOMAIN}`)
      ...
    env_file:
      - ${ENV}

File prod.env:

DOMAIN = 'actualdomain.com'
ENV = 'prod.env'
ENV_NUM = 1
ARG1 = 'value 1'

File dev.env:

DOMAIN = 'localhost'
ENV = 'dev.env'
ENV_NUM = 0
ARG1 = 'value 1'

Then run:

• docker-compose --env-file prod.env build,
• or docker-compose --env-file dev.env build

As an aside, even if most of this answer up to now, amounted to illustrating that the ".env" filename and env_file: files enjoy a very different semantics… it is true that they can also be combined "nicely" this way, as suggested by the OP, to achieve this use case.

Note in passing that the docker-compose config is also applicable to "debug" the Compose specification:

• docker-compose --env-file prod.env config,
• or docker-compose --env-file dev.env config.

Now regarding the last question:

Getting the values from the prod.env or dev.env files to docker-compose is not the issue. The issue is getting those values to the Dockerfile.

it can first be noted that there are two different cases:

1. Either the two different deployment environments (prod.env and dev.env) can share the same image, so that the difference only lies in the runtime environment variables (not the docker build args).
2. Or, depending on the file passed for --env-file, the images should be different (and then a docker-compose --env-file … build is indeed necessary).

It appears that most of the time, case 1. can be achieved (and it is also the case in the question's configuration, because the ARG1 values are the same in prod.env and dev.env) and can be viewed as more-interesting for the sake of reproducibility (because we are sure that the "prod" image will be the same as the "dev" image).

Yet, sometimes it's impossible to do so and we are "in case 2.", e.g. if the Dockerfile has a specific step, maybe related to tests or so, that has to be enabled (resp. disabled) in production mode.

So now, let us assume we're in case 2. How can we pass "everything" from the --env-file to the Dockerfile? There is only one solution, namely, extending the args: map of the docker-compose.yml and include each variable you are interested in, for example:

services:
  home:
    image: home-${ENV_NUM}
    build: 
      context: .\home
      args:
        DOMAIN: "${DOMAIN}"
        ENV_NUM: "${ENV_NUM}"
        ARG1: "${ARG1}"
    networks:
      - demo-net
    env_file:
      - ${ENV}
    labels:
      - traefik.enable=true
      - traefik.http.routers.home.rule=Host(`${DOMAIN}`)
      - traefik.http.routers.home.entrypoints=web
    volumes:
      - g:\:c:\sharedrive

Even if there is no other solution to pass arguments at build time (from docker-compose to the underlying docker build -f Dockerfile …), this has the advantage of being "declarative" (only the variables mentioned in args: will be actually passed to the Dockerfile).

Drawback?

The only drawback I see is that you may have unneeded extra environment variables at runtime (from docker-compose to the underlying docker run -e …), such as ENV=prod.env.

If this is an issue, you might want to split your ".env" files like this:

File prod.env:

DOMAIN = 'actualdomain.com'
ENV = 'prod-run.env'
ENV_NUM = 1
ARG1 = 'value 1'

File prod-run.env:

DOMAIN = 'actualdomain.com'
ENV_NUM = 1

(assuming you only want to export these two environment variables at runtime).

Or alternatively, to better follow the usual Do-not-Repeat-Yourself rule, remove prod-run.env, then pass these values as docker-compose build arguments as mentioned previously:

args:
  DOMAIN: "${DOMAIN}"
  ENV_NUM: "${ENV_NUM}"

and write in the Dockerfile:

ARG DOMAIN
ARG ENV_NUM

# ... and in the end:

ENV DOMAIN="${DOMAIN}"
ENV ENV_NUM="${ENV_NUM}"

I already gave an example of these Dockerfile directives in section "Experiment session 2".

(Sorry for the significant length of this answer BTW :)

You can use the .env file from docker compose so you use the same time that you defined in your service definition:

services:
    app:
         build:
            context: .\app
            args:
               ARG1: ${ARG3}
               ARG2: ${ARG4}
         env_file:
            - .env

In this way both can share the same env file, but you still have the drawback of redefining the variables as a placeholder.

This is a suggestion, choose the one that fits better to you