Optimizing Docker Builds in CI with Caching

The Need for Caching in CI

In a typical CI/CD environment, each job often runs in a fresh, ephemeral environment. This means that after a build completes, the environment is discarded, along with the local Docker build cache. When the next build starts, Docker has to rebuild everything from scratch. This is slow and inefficient.

The solution is to use a remote cache that persists between builds.

Part 1: The Foundation - Optimizing Your Dockerfile for Caching

Before we even get to CI-specific tools, the most significant performance gains come from a well-structured Dockerfile. Docker builds images in layers, and each instruction in a Dockerfile creates a new layer. If an instruction and its inputs haven't changed, Docker can reuse the layer from its cache instead of re-executing the instruction.

The key principle is: Order your Dockerfile instructions from least to most frequently changing.

Let's watch a segment that explains this core concept and also revisits multi-stage builds, a technique you were interested in mastering.

As you saw, the typical pattern is:

1. COPY dependency manifest files (e.g., package.json, pom.xml, requirements.txt).
2. RUN the dependency installation command (e.g., npm install, mvn dependency:go-offline, pip install).
3. COPY the rest of your application source code.

This way, dependency installation is only re-run when the manifest files change, not every time you change a line of source code.

FROM python:3.9-slim
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD ["python", "app.py"]

FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "app.py"]

Part 2: Automating the Build with GitHub Actions

Now that our Dockerfile is optimized, let's automate the build process. We'll use GitHub Actions, a popular CI/CD tool integrated directly into GitHub. Since you work with Kubernetes and CI/CD systems, the core concepts of workflows, jobs, and steps will be familiar.

The following video walks through setting up a complete CI pipeline that builds a Docker image and pushes it to a private Docker Hub repository.

The key component in the video is the docker/build-push-action. It's a powerful wrapper around docker buildx that simplifies building and pushing images within a GitHub Actions workflow.

Here is a basic workflow file. It checks out the code, sets up Buildx, logs into a registry, and then builds and pushes the image. Notice that this initial version has no caching configured yet.

# .github/workflows/docker-publish.yml
name: Build and Push Docker Image
on:
  push:
    branches: ['main']

jobs:
  build-and-push:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: your-dockerhub-username/my-app:latest
          # Add multi-arch support from our last lesson!
          platforms: linux/amd64,linux/arm64

Part 3: Enabling Effective Caching in the Pipeline

This is where we connect everything. The workflow above will work, but it will be slow because it rebuilds from scratch every time. To fix this, we'll use the cache-from and cache-to inputs of the docker/build-push-action.

There are several types of cache backends you can use. The following resource provides an excellent comparison.

As you've read, while type=gha is simple, it's limited. The most robust and scalable solution is using a registry-backed cache. This strategy stores the cache metadata as a separate image manifest in your container registry. It overcomes the size and scope limitations of the GHA cache and, crucially, supports mode=max, which caches layers from all stages of a multi-stage build.

Here is our final, optimized GitHub Actions workflow using a registry cache:

# .github/workflows/docker-publish.yml
name: Build and Push Docker Image with Cache
on:
  push:
    branches: ['main']

jobs:
  build-and-push:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Build and push with cache
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: your-dockerhub-username/my-app:latest
          platforms: linux/amd64,linux/arm64
          # Enable registry-based caching
          cache-from: type=registry,ref=your-dockerhub-username/my-app:buildcache
          cache-to: type=registry,ref=your-dockerhub-username/my-app:buildcache,mode=max

With this configuration, the first run on the main branch will be slow as it builds and pushes the cache. Every subsequent run will be significantly faster, as it will pull the remote cache first and only rebuild the layers that have actually changed.

Conclusion

You have now learned how to create fast, automated, and portable Docker build pipelines. This is a cornerstone of modern DevOps practices.

Key Takeaways:

• Optimize Locally First: A well-ordered Dockerfile using multi-stage builds is the foundation of any effective caching strategy.
• Automate with Actions: GitHub Actions and the docker/build-push-action provide a powerful and declarative way to integrate Docker builds into your CI workflow.
• Use Remote Caching for CI: Since CI runners are ephemeral, a remote cache is essential.
• Prefer Registry Cache: Using type=registry with mode=max is the most robust strategy. It stores all intermediate layers from multi-stage builds in your container registry, maximizing cache hits and build speed across your team and pipelines.

In our next lesson, we will enhance our CI pipeline even further. We'll learn how to run unit and integration tests within a multi-stage Docker build, ensuring that we not only build our images quickly but also validate their correctness before they are ever pushed to a registry.