Hello! Welcome to the first lesson of your course on distributed systems design.
Over your career as a software engineer, you've gained extensive hands-on experience with tools like Kafka, Kubernetes, and various distributed databases. This course is designed to build on that practical knowledge by exploring the fundamental principles that govern these systems. We'll move beyond the "how" of using these tools to the "why" of their design.
This first lesson addresses the most fundamental learning outcome of the course: explaining the core challenges that make distributed systems fundamentally different and more complex than single-machine applications. We will explore three interconnected problems:
- Partial Failures: What happens when one part of your system breaks, but the rest keeps going?
- Network Unreliability: Why you can never trust the network that connects your services.
- Unbounded Latency: The fact that operations can, and do, take an unpredictably long time to complete.
Mastering these concepts is the first step toward designing robust, scalable, and resilient systems. Let's begin.
Why Distributed Systems?
Before diving into the problems, it's worth briefly recapping why we build distributed systems in the first place. You've engineered backend services that serve a large user base, so you've seen the limits of a single, powerful machine. Modern applications demand high availability, low latency for a global audience, and the ability to handle ever-increasing amounts of data. Distributing the workload across multiple machines—or "scaling horizontally"—is the standard approach to meeting these demands.
To get a feel for the motivations and the architectural paradigm that emerged, please watch the following segment from a talk on this topic.
Why Distributed Systems Are Hard
This video, 'Why Distributed Systems Are Hard', provides excellent context on the shift from single-server architectures to modern distributed systems.
Watch the section from 06:25 to 12:13. Focus on the three reasons for horizontal scaling: scalability, availability, and latency. Also, pay attention to the description of 'shared-nothing architecture', as this is the foundational model for most systems we'll discuss.
As the video highlights, we've moved to a shared-nothing architecture, where independent computers (nodes) communicate exclusively by sending messages over a network. This simple fact is the source of all the complexity we're about to explore. We can no longer assume that different parts of our application can reliably and instantaneously communicate, as they would if they were just different threads in the same process.
Challenge 1: Partial Failures
On a single computer, failures are often total. If the operating system has a kernel panic or the power supply fails, the entire machine stops. The application is either working, or it's not.
Distributed systems introduce a new, more insidious type of failure: partial failure.
A partial failure is a condition where one or more components in a distributed system fail or become unresponsive, while other components continue to operate normally.
This is the defining characteristic of distributed systems. A node might crash, a network link might break, but the rest of the system is expected to carry on. This creates a state of ambiguity and uncertainty that developers must explicitly handle.
For a clear definition and a comparison of different system-building philosophies, please read the following sections from a summary of Martin Kleppmann's Designing Data-Intensive Applications.
Designing Data-Intensive Applications - Dhiman Seal
This article, based on the highly-regarded book, effectively breaks down the troubles with distributed systems. These first sections introduce partial failures and contrast the fault-tolerance models of different computing paradigms.
Read the sections 'Faults and Partial Failures in Distributed Computing' and 'The Philosophies of Building Large-Scale Computing Systems'. Note the contrast between the total failure model of High-Performance Computing (HPC) and the partial failure model of Cloud Computing.
The key takeaway is that in the world of cloud services and microservices, we build reliable systems from unreliable, commodity components. We assume things are always in some state of partial failure and design for it.
To reason about these failures, it's helpful to have a formal model. The video below explains the common abstractions for node failures.
Distributed Systems 2.3: System models
In this segment from his lecture series, Martin Kleppmann categorizes the different ways a node can fail.
Watch the segment from 08:30 to 11:26. Focus on the distinction between the 'crash-stop', 'crash-recovery', and 'Byzantine' fault models.
Most systems you've worked on likely operate under a crash-recovery model. For example, when a pod in a Kubernetes cluster crashes due to a bug, Kubernetes doesn't shut down the whole application. It simply restarts the pod on a healthy node, assuming its state can be recovered or was stateless to begin with. The rest of the application must handle the temporary absence of that pod.
Test your understanding!
Imagine you are running a critical service with three replicas. One of the replicas experiences a hardware failure and enters a crash-stop state. Another replica's process crashes but is immediately restarted by the orchestration system (crash-recovery). How do these two types of partial failures present different challenges to your system's load balancer and data replication logic?
Show answer
The crash-stop failure is simpler in some ways. The node is gone and will never return. The load balancer must permanently remove it from the rotation, and the data replication mechanism must create a new replica elsewhere to restore the desired redundancy level.
The crash-recovery failure is more subtle. The node disappears and then reappears.
- Load Balancer: It must detect the failure (e.g., via failed health checks), stop sending traffic, and then detect the node's recovery to add it back to the pool.
- Data Replication: If the node was a leader, a failover process would have been triggered. When the old leader recovers, it must not believe it is still the leader (this is known as a "split-brain" problem). If it was a follower, it might have missed updates while it was down and needs a way to catch up before it can start serving requests again. This highlights the need for mechanisms to handle stale state.
Challenge 2: Network Unreliability
In a shared-nothing architecture, the network is the only way for nodes to communicate. Unfortunately, the network is fundamentally unreliable. This is one of the most famous "Fallacies of Distributed Computing": the assumption that the network is reliable.
When you make an API call from one service to another, you might assume it just works. But what if you don't get a response?
- Did your request get lost on the way?
- Did the remote node crash before processing it?
- Did the remote node process the request successfully, but the response got lost?
From your perspective as the sender, all these scenarios are identical: you sent a message and received silence. This ambiguity is a central problem in distributed systems.
The following article dives deep into why we can't trust the network, with concrete data and examples.
The Problems of Distributed Systems: Part 2 — Unreliable ...
This article, 'The Problems of Distributed Systems: Part 2 — Unreliable Networks', focuses entirely on this challenge, explaining the ambiguity of network failures and why they are inevitable.
Read the first four sections: the introduction, 'What Does It Mean for a Network to Be Unreliable?', 'Network Failures in Practice...', and 'The Wild and Unpredictable Nature of Real-World Failures'. Focus on the inherent ambiguity when a sender doesn't receive a reply.
The real-world examples—from sharks biting cables to simple human error during a router configuration—underscore a critical point: you must design your system assuming the network will fail. A system that can't handle a timeout or a dropped packet is not a production-ready distributed system.
Challenge 3: Unbounded Latency
The final core challenge is closely related to the previous two. Not only can messages get lost, but even when they are delivered, they can take an arbitrarily long time to arrive. This is known as unbounded latency (or unbounded delay).
The problem isn't just network latency. A process on a remote node might be paused at any moment for an unpredictable duration.
For a detailed explanation of why we cannot put an upper bound on time in distributed systems, please watch this last video segment.
Distributed Systems 2.3: System models
Martin Kleppmann returns to explain the different timing models and, crucially, the real-world sources of unbounded delays.
Watch the segment from 11:26 to 19:07. Pay close attention to the reasons why assuming a 'synchronous' model (with bounded delays) is dangerous. The examples of garbage collection pauses, context switches, and network queuing are particularly relevant to your work as a software engineer.
The concept of a "stop-the-world" garbage collection pause is a perfect, concrete example. You've worked with Python and Go, both of which have automatic memory management. A GC pause on a remote service can halt its execution for hundreds of milliseconds, or even seconds. To any client communicating with it, the service will appear to be down or experiencing extreme network lag.
This leads to the Timeout Paradox:
- Set a short timeout: You can react quickly to actual failures, but you risk incorrectly flagging a node as dead when it's just slow (e.g., due to a GC pause). This can cause cascading failures if healthy but slow nodes are shed from the system, overloading the remaining ones.
- Set a long timeout: You reduce false positives, but your system becomes sluggish. Users may be left waiting for a long time before the system reacts to a genuine failure.
There is no "correct" timeout value. It is always a trade-off that depends on the specific requirements of your service.
Test your understanding!
You are designing a payment processing service. The service has a strict 1-second timeout for calls to a third-party fraud detection API. One day, you observe that about 1% of payments are failing due to timeouts, even though the fraud API provider insists their service is healthy. What are some possible root causes related to unbounded latency that could explain this behavior, even if the network path is clear and the remote service is running?
Show answer
Several factors related to unbounded latency could be at play:
- Garbage Collection: The fraud detection service might be experiencing periodic "stop-the-world" GC pauses that last longer than 1 second, causing it to be unresponsive during those times.
- Queueing: Your request might get stuck in a queue on the remote server. If the fraud service suddenly receives a burst of traffic from another client, your request has to wait its turn, and the total processing time (wait time + execution time) might exceed your 1-second timeout.
- Context Switching / "Noisy Neighbors": The fraud service might be running in a multi-tenant environment (like a shared VM or container host). A "noisy neighbor" process could be monopolizing CPU resources, causing the fraud service's process to be paused by the OS scheduler for a significant period.
This illustrates that even with a healthy network and a "running" remote process, there are no guarantees about response time.
Conclusion
In this lesson, we established the three fundamental challenges that define distributed computing and separate it from traditional, single-machine programming:
- Partial Failures: Components fail independently, and the system must be designed to function in a partially broken state.
- Network Unreliability: The communication links between components cannot be trusted. Messages are lost, delayed, and reordered, creating fundamental ambiguity.
- Unbounded Latency: There are no upper bounds on the time it takes for a message to be delivered or for a process to execute a step, thanks to network queues, process pauses, and other sources of unpredictable delay.
These three problems are not independent; they are deeply intertwined. A network partition is a form of partial failure. A timeout caused by a GC pause is an instance of unbounded latency manifesting as a network-like failure. Together, they mean that a node in a distributed system can never be sure about the state of any other node. It can only make educated guesses based on messages—or the lack thereof.
This fundamental uncertainty is why we need the specialized algorithms and patterns we will study in the rest of this course.
Preview of the next lesson: The lack of reliable, instantaneous communication leads to another major problem: there is no single source of truth for time. In the next lesson, we will explore the implications of having no global clock and introduce logical clocks, our first tool for imposing order on the chaos of distributed events.