Immutable Containers for Enhanced Security

1. The Principle of Immutability

In a traditional, mutable server environment, administrators apply patches and configuration changes directly to a running system. Immutable infrastructure is a different paradigm: instead of changing a running instance, you replace it with a new one built from an updated image.

Running a container with a read-only filesystem is a core tenet of this approach. It guarantees that the running container is an exact replica of the image it was created from. Any attempt to modify the filesystem, whether malicious or accidental, will fail.

Let's begin by reading a brief overview of the security benefits.

By making the filesystem read-only, you dramatically reduce the container's attack surface. An attacker who exploits a vulnerability in your application will find it much harder to escalate their attack if they cannot write files.

2. Hands-On: Launching a Read-Only Container

Let's put this into practice. The simplest way to enforce this is with the --read-only flag in docker run or the read_only: true key in Docker Compose.

The following guide from dev.to provides a perfect, self-contained lab to see this in action.

After completing the exercise, you will have seen firsthand that even the root user inside the container is blocked from making changes. The command docker inspect <container_id> | grep '"ReadonlyRootfs"' gives you a programmatic way to confirm that your security policy is being enforced.

3. The Challenge: Real-World Applications Need to Write

As you probably suspect, simply adding --read-only to a real-world application like Nginx, a Node.js API, or a database will likely cause it to crash. Most applications need to write to the filesystem for legitimate reasons:

• Creating PID files (e.g., in /var/run)
• Writing to caches (e.g., /var/cache/nginx)
• Storing temporary files (e.g., in /tmp)
• Writing application logs

The key is not to abandon read-only filesystems, but to selectively enable writes only where they are absolutely necessary. We can do this using tmpfs mounts for temporary data and volumes for persistent data.

This video from HackerSploit provides a clear demonstration of this exact problem and its solution.

As the video shows, Docker provides two main mechanisms to handle this:

1. tmpfs mounts (--tmpfs or tmpfs in Compose): This creates a temporary filesystem in memory. It's perfect for non-persistent data like PID files, sockets, or caches that don't need to survive a container restart. Data in a tmpfs mount is erased when the container stops.

2. Volumes (-v or volumes in Compose): These are used for data that must be persisted, such as database files, user uploads, or critical log files. You should already be familiar with volumes from your previous work with Docker.

The image below, though set in a Kubernetes context, perfectly illustrates this separation. Runtime data (/var/run) is mounted into memory (tmpfs), while cache data that might be useful to persist across restarts (/var/cache/nginx) is mounted to a disk (a volume).

4. Identifying Writable Directories and Configuring Your Container

As a DevOps engineer, your main task will be to identify which directories an application needs to write to and configure the appropriate mounts.

The most straightforward method is often iterative:

1. Run the container with read_only: true and no mounts.
2. Check the logs (docker logs <container_name>) for "Read-only file system" errors.
3. Add a tmpfs mount for the directory mentioned in the error.
4. Repeat until the container starts successfully.

For stateful data, like a PostgreSQL database's data directory (/var/lib/postgresql/data), you would use a named volume instead of tmpfs.

version: '3.8'

services:
  python-app:
    image: my-python-app:1.0
    read_only: true
    tmpfs:
      - /tmp  # For temporary, non-persistent files
    volumes:
      - app-logs:/var/log/app/ # For persistent log data

volumes:
  app-logs:

Conclusion

You've now added another critical tool to your security hardening arsenal. By enforcing an immutable filesystem, you make your containers more predictable, reliable, and resilient to attack.

Key Takeaways:

• Running a container with a read-only root filesystem (--read-only or read_only: true) prevents runtime modifications, enhancing security and stability.
• Most applications will not work out-of-the-box in a read-only environment because they need to write temporary or persistent data.
• Use tmpfs mounts to provide writable, ephemeral storage in memory for things like PID files, sockets, and temporary caches.
• Use named volumes to provide writable, persistent storage for data that must survive container restarts, like database files or application logs.
• The most common way to identify required writable paths is to run the container as read-only and inspect the logs for "Read-only file system" errors.

In our next lesson, we will shift our focus from the filesystem to another critical area of security: managing sensitive information. We will explore the topic: Differentiate between Docker secrets and environment variables for managing sensitive data.