Configuring Services with Environment Variables and .env Files

1. The Problem: Why Hardcoding is a Bad Idea

Any piece of information that changes between deployment environments (e.g., database credentials, API keys, domain names) or is sensitive in nature is considered configuration. Hardcoding this directly into your docker-compose.yml is risky and inflexible.

To understand why this is such a critical issue in DevOps, let's watch the beginning of the following video. It clearly explains the security implications.

The key takeaway is that your docker-compose.yml file defines the structure of your application, and it should be safe to commit to version control. The specific, sensitive values that configure it should be kept separate.

2. The Solution: Using .env Files

The standard solution in the Docker ecosystem is to use an environment file, conventionally named .env. This is a simple text file where you define configuration variables as key-value pairs.

By default, when you run docker compose up, Docker Compose automatically looks for a file named .env in the same directory as your docker-compose.yml file and loads the variables defined within it.

Here is the flow:

Step 1: Create the .env File

You create a file named .env in your project root with the following format:

# .env file

# This is a comment
# No spaces around the equals sign
POSTGRES_USER=myuser
POSTGRES_PASSWORD=mysecretpassword
POSTGRES_DB=mydatabase

Step 2: Reference Variables in docker-compose.yml

Next, you modify your docker-compose.yml to use these variables. The syntax for this is ${VARIABLE_NAME}.

# docker-compose.yml

services:
  db:
    image: postgres:14-alpine
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
    volumes:
      - db-data:/var/lib/postgresql/data

volumes:
  db-data:

Now, when Docker Compose reads this file, it will substitute ${POSTGRES_PASSWORD} with the value mysecretpassword from the .env file.

Step 3: The Most Important Step - .gitignore

The whole point of this is to keep secrets out of version control. Therefore, you must add the .env file to your .gitignore file.

# .gitignore

.env

This tells Git to ignore this file, ensuring your secrets are never accidentally committed.

Let's watch a practical demonstration of these steps.

3. Different Ways to Use Environment Variables

Environment variables in Docker Compose can be used in two main ways, and it's important to understand the difference.

A) For Variable Substitution in the Compose File

Docker Compose can use variables from your .env file to substitute values within the docker-compose.yml itself. This is useful for non-secret configuration that you might want to change easily, like image tags or container names.

The following video provides an excellent, concise example of this.

In this case, the variables are used by Docker Compose during the parsing of the file but are not necessarily passed into the container's runtime environment.

B) For the Container's Runtime Environment

This is what we did with our database credentials. The environment key under a service definition passes the variables into the container so that the application running inside (in our case, PostgreSQL) can use them.

You have two primary directives for this:

1. environment: Passes variables one by one. You can set a value directly or pass through a value from the host or .env file.

   services:
     app:
       environment:
         - DEBUG=true  # Set a value directly
         - LOG_LEVEL=${LOG_LEVEL} # Pass through from .env
   

2. env_file: Passes all variables from a specified file directly into the container.

   services:
     app:
       env_file:
         - ./app.env # Pass all variables from app.env
   

The following article provides a great summary of the different approaches and best practices.

4. Advanced Techniques and Best Practices

As you move from a simple project to a real-world DevOps workflow, you'll need more advanced techniques for managing configurations.

Managing Multiple Environments

A typical project has multiple environments: development, staging, production, etc. Each requires a different configuration. A common pattern is to have multiple .env files:

• .env.development
• .env.production

You can then tell Docker Compose which one to use with the --env-file flag:
docker compose --env-file .env.production up -d

Default and Required Variables

Your docker-compose.yml can be made more robust by setting default values for development or requiring certain variables to be set in production.

• Set a default value: ${VAR:-default}. If VAR is not set or is empty, default will be used.

  # Use 'latest' tag if VERSION is not set
  image: myapp:${VERSION:-latest}
  

• Require a variable: ${VAR:?error message}. If VAR is not set or is empty, Compose will exit with an error.

  # Fail if API_KEY is not provided
  API_KEY: ${API_KEY:?API_KEY must be set in the environment!}
  

Documenting with .env.template

Since .env is not committed to Git, how does a new developer on your team know which variables to set? The best practice is to create a template file, often called .env.template or .env.example, that is committed to the repository.

# .env.template
# This file documents required environment variables.
# Copy this to .env and fill in the values.

POSTGRES_USER=
POSTGRES_PASSWORD=
POSTGRES_DB=

This serves as clear documentation for your project's configuration.

services:
  backend:
    image: my-backend-app:1.2
    container_name: dev_backend
    environment:
      API_SECRET_KEY: "super-secret-key"
      REDIS_HOST: "redis"

  cache:
    image: redis:6-alpine
    container_name: dev_cache

Conclusion

Congratulations! You've taken a massive step towards building professional, secure, and maintainable applications with Docker Compose. By externalizing configuration and keeping secrets out of version control, you are adopting a core principle of modern software development and DevOps.

Key Takeaways:

• Never hardcode secrets or environment-specific configuration in docker-compose.yml.
• Use a .env file to store these values and always add it to .gitignore.
• Reference variables in docker-compose.yml using the ${VARIABLE_NAME} syntax.
• Use variables for both Compose-level substitution (e.g., image tags) and for passing into the container's runtime environment (e.g., database credentials).
• Manage different configurations (dev, prod) using multiple .env files and the --env-file flag.
• Use a .env.template file to document the required variables for your project.
• Make your Compose files more robust with default values (:-) and required variables (:?).

Next Up

Our application services are now networked, persistent, and securely configured. However, there's a potential race condition. What if our backend service starts before the database is ready to accept connections? The application will crash. In our next lesson, we will solve this by learning how to control service startup order and dependencies using depends_on.