Hello! Welcome to the final lesson in our module on Advanced Networking and Data Persistence.
In our previous lesson, we learned how to methodically troubleshoot container connectivity issues. The process of creating, testing, and destroying networks and containers for debugging is essential, but it often leaves behind a trail of unused Docker objects. Over time, this digital clutter can consume significant disk space and make your development environment unwieldy.
Today, we'll address this by learning how to clean up unused volumes and networks to manage host resources. As a DevOps engineer, keeping systems clean and efficient is a core responsibility. Mastering these cleanup operations will help you manage costs, maintain performance, and ensure your Docker hosts are tidy and predictable.
We'll explore the specific prune commands for each resource type and the powerful docker system prune command for a comprehensive cleanup.
The Problem of "Docker Debris"
Every time you build an image, run a container, or use docker-compose up, Docker creates objects: images, containers, volumes, and networks. When you stop and remove containers, these associated objects aren't always removed automatically, leading to a gradual buildup of unused resources.
Let's quickly review the main types of "debris":
- Stopped Containers: Containers that you've stopped but not removed.
- Dangling Images: Image layers that have no relationship to any tagged image. They often result from rebuilding images.
- Unused Networks: Custom networks that remain even after all containers connected to them have been removed.
- Unused Volumes: Volumes that are not attached to any container. Docker is especially careful with these to prevent accidental data loss, so they are never removed automatically.
To see why this is a significant issue, especially for developers and CI/CD systems, let's start with a short video that frames the problem.
Docker Quietly Eats Drives | Docker Prune & Cleanup Guide
The video 'Docker Quietly Eats Drives' by TroubleChute Linux effectively introduces the problem of Docker consuming disk space and provides a brief but clear overview of the different types of resources that need to be managed.
Watch the first 3 minutes and 20 seconds of the video. Focus on understanding how day-to-day Docker operations can lead to a buildup of containers, images, volumes, and networks that consume large amounts of disk space.
A Systematic Approach to Pruning
The prune commands are powerful, but they are also destructive. There is no "undo" button. Therefore, the cardinal rule of pruning is to inspect before you prune.
Before removing anything, you should always list the resources to understand what will be affected.
- Networks:
docker network ls - Volumes:
docker volume ls - Containers:
docker ps -a(ordocker container ls -a) - Images:
docker image ls
Now, let's look at the specific commands for cleaning up volumes and networks, which are the focus of today's outcome. The official Docker documentation is the best source for this.
Let's review the official documentation for pruning volumes and networks. This will give you the authoritative syntax and behavior for these commands.
Read the sections titled 'Prune volumes' and 'Prune networks'. Pay close attention to the warnings and the description of what each command removes.
Pruning Networks
As you saw in the documentation, cleaning up networks is straightforward. The docker network prune command removes all custom networks that are not being used by at least one container.
docker network prune
This command is generally safe, as it won't affect running applications. It's useful for clearing out networks left over from old Docker Compose projects.
Pruning Volumes
Pruning volumes requires more caution. The docker volume prune command removes all local volumes that are not in use by at least one container.
docker volume prune
Warning: This command removes both anonymous and named volumes that are not currently attached to a container. This could lead to permanent data loss if you have a named volume that you intended to reuse later (e.g., a database volume that you temporarily detached). This is why Docker prompts for confirmation.
To bypass the confirmation prompt for any prune command, you can use the -f or --force flag, but do so with extreme care.
The All-in-One Cleanup: docker system prune
Instead of running individual prune commands, you can use docker system prune to perform a comprehensive cleanup in one go.
By default, docker system prune removes:
- All stopped containers
- All networks not used by at least one container
- All dangling images
- All dangling build cache
Notice what's missing? Volumes.
This is a critical safety feature. To include unused volumes in a system-wide prune, you must explicitly add the --volumes flag.
# This will also remove all unused volumes
docker system prune --volumes
Another useful flag is -a or --all. When used with docker image prune or docker system prune, it removes all unused images, not just dangling ones. An "unused" image is one that is not associated with any container.
The following video demonstrates these commands and their variations.
Clean Up Unused Docker Containers, Images, Network, Volumes and Build Cache
Let's watch a demonstration of the system prune command, including its powerful and potentially dangerous flags like -a and --volumes.
Watch the section from 06:14 to 08:47. The presenter clearly explains the difference between a standard docker system prune and what happens when you add the --volumes and --all flags.
Best Practices and Safe Pruning
Given your role as a DevOps Engineer, it's crucial to use these tools safely, especially in production or shared environments.
The following article offers a superb guide on best practices.
Docker Prune: A Complete Guide with Hands-On Examples
The article 'Docker Prune: A Complete Guide with Hands-On Examples' provides excellent advice on using these commands safely and effectively.
Read the sections 'Using Docker Prune Safely' and 'Best Practices for Docker Prune'. The key takeaways are the importance of inspection, the lack of a dry-run option, and how to automate cleanup cautiously.
Here's a summary of the key best practices:
- Inspect Before Pruning: Always use
docker ... lscommands to see what will be removed. Since Docker has no--dry-runoption for pruning, this is your only way to preview the impact. - Use Filters for Precision: Prune commands support the
--filterflag. A very useful filter isuntil, which allows you to remove objects created before a certain time. This is perfect for automated cleanup jobs, as it won't interfere with recent work.# Remove all unused images created more than 24 hours ago docker image prune -a --filter "until=24h" - Automate with Caution: On Linux systems, you can schedule cleanup using a cron job. A weekly or nightly job can keep a system tidy.
# At 3 AM every Sunday, remove all unused Docker objects created over a week ago 0 3 * * 0 /usr/bin/docker system prune -a --volumes --filter "until=168h" -f
Test your understanding!
A colleague runs docker system prune on a shared development server to free up disk space. Later, another developer complains that the data in their PostgreSQL database container, which was stopped for the weekend, is gone. The first colleague is confused because they thought system prune was safe. What is the most likely mistake they made?
Show answer
The most likely mistake is that they ran docker system prune --volumes. The default docker system prune does not remove volumes. By adding the --volumes flag, the command also removed all volumes not currently attached to a container. Since the PostgreSQL container was stopped, its associated named volume was considered "unused" and was deleted, resulting in data loss.
This highlights the critical importance of understanding the --volumes flag and inspecting the system (docker volume ls) before running destructive commands.
Conclusion
You've now reached the end of this module and have a solid grasp of advanced Docker networking, data persistence, and maintenance.
Key Takeaways:
- Docker objects like containers, images, volumes, and networks accumulate over time and consume host resources.
- The
docker prunefamily of commands is used to reclaim disk space and keep your environment clean. docker network pruneremoves unused custom networks, whiledocker volume pruneremoves unused volumes. Be extra cautious with volume pruning to avoid data loss.docker system pruneis the all-in-one command, but it does not prune volumes by default. You must use the--volumesflag explicitly.- The best practice is to always inspect before pruning (
docker ... ls) and to use filters likeuntilfor safe, automated cleanup.
With your knowledge of creating robust network architectures and now keeping them clean, you are well-prepared to build and manage complex applications.
In our next module, we will dive deeper into Multi-Service Applications with Docker Compose. We'll start by learning how to implement health checks and service dependencies, ensuring that your applications not only start but do so in the correct order and only when they are truly ready.