Monitoring Docker with Prometheus and Grafana

The Full Monitoring Stack Architecture

While the Docker daemon provides great metrics about the engine, a comprehensive monitoring setup also needs to see what's happening inside each container. For this, we'll introduce a few key components that work together:

• Prometheus: Our central time-series database. It actively "scrapes" (pulls) metrics from configured endpoints and stores them.
• cAdvisor (Container Advisor): An exporter from Google that specializes in collecting detailed resource usage and performance data from running containers (CPU, memory, network, etc.).
• Node Exporter: A Prometheus exporter that provides detailed metrics about the host machine (the node), such as CPU, memory, and disk I/O of the underlying server.
• Grafana: The visualization layer. It queries Prometheus and renders the data in powerful, customizable dashboards.

The diagram below illustrates how these components fit together.

Today, we will build this entire stack, and we will configure Prometheus to scrape both cAdvisor and the Docker daemon metrics endpoint you enabled in the last lesson. This will give you a complete view of your containerized environment.

Step 1: Defining the Stack with Docker Compose

Given your experience with DevOps and Docker Compose, we'll define our entire monitoring stack in a single docker-compose.yml file. This approach makes the stack portable, version-controllable, and easy to manage.

Create a new directory for this project. Inside it, create a file named docker-compose.yml.

my-monitoring-stack/
└── docker-compose.yml

Copy the following content into your docker-compose.yml file. This configuration is based on a common professional setup.

version: '3.8'

volumes:
  prometheus_data: {}
  grafana_data: {}

networks:
  monitoring:
    driver: bridge

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus/:/etc/prometheus/
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
    ports:
      - "9090:9090"
    networks:
      - monitoring
    # This entry makes the host's IP address available to the container
    # under the name 'host.docker.internal'. It's crucial for scraping the Docker daemon.
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    ports:
      - "8080:8080"
    networks:
      - monitoring
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
    ports:
      - "3000:3000"
    networks:
      - monitoring
    restart: unless-stopped

A quick breakdown of this file:

• services: We define three services: prometheus, cadvisor, and grafana.
• volumes: We use named volumes (prometheus_data, grafana_data) to persist data even if the containers are removed. We also use bind mounts to inject configuration files (e.g., for Prometheus).
• networks: All services are attached to a custom bridge network called monitoring. This allows them to communicate with each other using their service names as hostnames (e.g., http://prometheus:9090).
• extra_hosts: The host.docker.internal:host-gateway entry in the Prometheus service is a modern, reliable way to allow a container to reach services running on the host machine. We'll use this to scrape the Docker daemon's metrics endpoint.

Step 2: Configuring Prometheus Scrape Targets

Prometheus needs a configuration file to know which endpoints to scrape. Create a prometheus directory and a prometheus.yml file inside it.

my-monitoring-stack/
├── docker-compose.yml
└── prometheus/
    └── prometheus.yml

Paste the following configuration into prometheus/prometheus.yml.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']
  
  - job_name: 'docker-daemon'
    static_configs:
      - targets: ['host.docker.internal:9323']

Here, we've defined three scrape_configs jobs:

1. prometheus: To monitor Prometheus itself.
2. cadvisor: To scrape container metrics from cAdvisor. It uses the service name cadvisor since they are on the same Docker network.
3. docker-daemon: To scrape the Docker daemon metrics you enabled in the last lesson. It uses host.docker.internal to reach the host machine's port 9323.

Step 3: Launch and Verify the Stack

Now, you're ready to bring everything online.

1. Open a terminal in the my-monitoring-stack directory.
2. Run docker-compose up -d. This will pull the images and start the three containers in the background.
3. Verify that all containers are running with docker ps. You should see prometheus, cadvisor, and grafana.

Next, let's verify that Prometheus is successfully scraping all its targets.

• Open your browser and navigate to the Prometheus UI: http://localhost:9090.
• Go to the Status > Targets page.

The following video shows what you should be looking for on the Targets page.

If any target is DOWN, double-check your configuration files and ensure the Docker daemon metrics endpoint is still active.

Step 4: Visualizing Metrics in Grafana

With data flowing into Prometheus, it's time to visualize it.

1. Open Grafana in your browser: http://localhost:3000.
2. Log in with the default credentials: username admin and password admin. Grafana will prompt you to set a new password.

Connecting Grafana to Prometheus

First, you need to tell Grafana where to find your data. You'll add Prometheus as a data source. The process is straightforward, as shown in this next video clip.

Importing a dashboard from the Grafana community is the fastest way to get a rich visualization. After you import the dashboard, you should see a detailed overview of your running containers.

A Note on Provisioning (The DevOps Way)

While adding data sources and dashboards manually via the UI is fine for getting started, in a professional CI/CD environment, you'd want this to be automated. Grafana supports "provisioning," where you can define data sources and dashboards in YAML/JSON files. When Grafana starts, it automatically configures itself.

The Prometheus with Docker Compose article we've referenced provides an excellent example of this. You could extend your docker-compose.yml to mount these provisioning files into the Grafana container, fully automating your setup. This is a powerful pattern to keep in mind for production environments.

Conclusion

Congratulations! You have successfully deployed a production-grade monitoring stack for Docker using Prometheus and Grafana. You now have visibility into both the Docker engine itself and the individual containers it manages.

Key Takeaways:

• A complete monitoring stack typically consists of Prometheus, Grafana, and one or more exporters like cAdvisor.
• Docker Compose is an ideal tool for defining, deploying, and managing this multi-container application.
• Prometheus is configured with a prometheus.yml file to define scrape jobs for different metric sources.
• Special networking configurations like extra_hosts or host.docker.internal are needed for containers to scrape metrics from the host.
• Grafana visualizes Prometheus data, and importing community dashboards is an efficient way to get started.

In our next lesson, we will dive deeper into troubleshooting and inspection. You will learn how to inspect the state of Docker objects and monitor daemon events for in-depth troubleshooting, using commands like docker inspect and docker events.