Docker Compose doesn't seem to read env variables defined in env file

Question

I have the following project directory structure:

myapp/
    docker/
        docker-compose.yml
    .env
    src/
        <my source code here>
    config.properties

Here is my .env file:

ENV=local
SERVICE_DB_HOST=0.0.0.0
SERVICE_DB_PORT=3306
SERVICE_DB_ROOT_PASSWORD=12345
SERVICE_DB_APP_USER=my-service-user
SERVICE_DB_APP_PASSWORD=23456

Here is my docker/docker-compose.yml:

version: "3.7"
services:
  myapp-main-db:
    env_file:
      - ../.env
    image: mysql:8
    container_name: myapp-main-db
    command: --default-authentication-plugin=mysql_native_password
    restart: always
    ports:
      - $SERVICE_DB_PORT:$SERVICE_DB_PORT
    environment:
      MYSQL_ROOT_PASSWORD: $SERVICE_DB_ROOT_PASSWORD
      MYSQL_DATABASE: myapp_service_db_$ENV
      MYSQL_USER: $SERVICE_DB_APP_USER
      MYSQL_PASSWORD: $SERVICE_DB_APP_PASSWORD
    volumes:
      - myapp-service-db-data:/var/lib/mysql
volumes:
  myapp-service-db-data:

When I run docker-compose -f docker/docker-compose.yml up -d from the project root directory, I see the MySQL container spin up without issue, and verify its running via docker ps.

However I am unable to connect to it using the expected connection string and credentials, which I am sure are correct. This leads me to believe that Docker Compose is not loading the ../.env file as I was expecting it to, and so none of the variables defined in ../.env are being injected as env vars.

Can anybody spot where I'm going awry?

Update

When I run docker-compose config I get:

$ docker-compose config
WARNING: The SERVICE_DB_PORT variable is not set. Defaulting to a blank string.
WARNING: The SERVICE_DB_ROOT_PASSWORD variable is not set. Defaulting to a blank string.
WARNING: The ENV variable is not set. Defaulting to a blank string.
WARNING: The SERVICE_DB_APP_USER variable is not set. Defaulting to a blank string.
WARNING: The SERVICE_DB_APP_PASSWORD variable is not set. Defaulting to a blank string.
ERROR: The Compose file './docker-compose.yml' is invalid because:
services.myapp-main-db.ports contains an invalid type, it should be a number, or an object

Answer 1

Compose has two separate layers of environment file handling:

1. When the docker-compose.yml file is read in, it reads a .env file from the same directory as the docker-compose.yml, and uses that to expand $VARIABLE references.
2. When an individual container is launched, it reads the env_file: and uses that to set environment variables in the container.

Since .env isn't in the same directory as the docker-compose.yml file, Compose won't read it before expanding variables in the file. That leads to the container getting empty strings for the database credentials, and the error because ports: is malformed.

You can either move .env into the same directory as docker-compose.yml

mv .env docker
docker-compose -f docker/docker-compose.yml up

or pass a docker-compose --env-file option

docker-compose -f docker-docker-compose.yml --env-file .env up

You don't need env_file: in the docker-compose.yml file for this particular use and can remove it.

---

To debug this further, you can docker-compose run a container with an alternate command:. You could for example docker-compose run myapp-main-db env to print out its environment instead of launching a database. The flow I'd expect you to see here is:

1. Compose reads docker/.env (from the same directory as the docker-compose.yml file), which is absent
2. Compose replaces variables in the file; I'd expect to see MYSQL_USER to be an empty string
3. Compose reads the env_file:; the container will have the additional environment variable SERVICE_DB_APP_USER=my-service-user from that file

Answer 2

According to default behavior of docker, Dockerfile and docker-compose never read any file or value from parent directory. It always read from current directory or from Child directory.

so for your code I suggest please use docker-compose file in source/root directory.

Example:

myapp/
    docker-compose.yml
    .env
    src/
        <my source code here>
    config.properties

Answer 3

Quote from the official Docker docs: Using the “--env-file” option.

You can use an .env file from anywhere, using the --env-file option:

By passing the file as an argument, you can store it anywhere and name it appropriately, for example, .env.ci, .env.dev, .env.prod. Passing the file path is done using the --env-file option:

docker-compose --env-file ./config/.env.dev up

Obviously, you cannot go to a parent directory of your current directory to get the .env file when you are inside the docker-compose, thus, not like this:

version: "3.7"
services:
  myapp-main-db:
    env_file:
      - ../.env

One way to use an .env file from somewhere else like here at the parent folder is to use the --project-directory parameter of the docker-compose command. According to the official Docker docs:

The .env file path is as follows:

Starting with +v1.28, .env file is placed at the base of the project directory

For previous versions, it is placed in the current working directory where the Docker Compose command is executed unless a --project-directory is defined which overrides the path for the .env file. This inconsistency is addressed in +v1.28 by limiting the filepath to the project directory."

Which would mean in your case:

docker-compose --project-directory .. up

Or: you can reach any .env file on your system if you pass it as the --env-file parameter of the docker-compose command.

docker-compose --env-file ../.env up

If you use neither --project-directory nor --env-file you need to put the .env file where the docker-compose file is.