Distributed System Challenges: Latency & Partial Failure

A New World of Problems

In the monolithic Django applications you've built, operations are relatively simple. When one part of your code calls another—for instance, a view calling a service function—that call happens in-memory. It's incredibly fast, and it's predictable: it either works, or it throws an exception and fails completely. If the server itself crashes, the entire application goes down. This is the world of total failure, and in many ways, it's a world of certainty.

Distributed systems shatter this simple reality.

These "fallacies" are a perfect lens through which to view our new challenges. They were first documented by engineers at Sun Microsystems who saw developers repeatedly making the same incorrect assumptions about networks. Let's get a quick overview of all eight, as they will be a recurring theme.

As the video highlights, the moment you separate your monolith into services that communicate over a network, you must abandon the assumptions of reliability and instant communication. Let's explore exactly what that means.

Challenge 1: Partial Failure

In a monolithic system, components share a single fault domain. If the server's power supply fails, the web server, the business logic, and the database connection pool all fail together.

In a microservices architecture, each service is its own fault domain. This introduces a concept that is entirely new and deeply challenging: partial failure.

Partial Failure: A state in which one component of a distributed system has failed, while other components continue to operate normally.

To understand why this is so difficult, let's start with a foundational reading.

The Ambiguity of Timeouts and the UNKNOWN State

The "Response Timeout Problem" you just read about is the most critical consequence of partial failure. When your Order Service calls the Payment Service and the request times out, what do you know? The uncomfortable answer is: almost nothing.

The AWS Builders' Library has an excellent article that breaks down a single network call into eight distinct steps, each of which can fail independently. This provides a very concrete model for understanding partial failure.

As the AWS article explains, a timeout forces the client into an UNKNOWN state. From the client's perspective, any of these scenarios could be true:

1. The request never reached the server.
2. The request reached the server, was processed successfully, but the response was lost.
3. The request reached the server, but the server crashed before processing it.
4. The request reached the server, it was processed, but the server crashed after processing it but before sending a response.

In a monolithic Django app, a call to account.withdraw(amount) returns True, False, or throws an exception. You always know the outcome. In a distributed system, account_service.withdraw(account, amount) can result in SUCCESS, INSUFFICIENT_FUNDS, or UNKNOWN.

How you handle UNKNOWN is one of the hardest problems in system design. If you retry, do you risk charging the customer twice? If you don't retry, does the customer's order fail unnecessarily? We will spend much of this course learning patterns (like idempotency, which we'll cover soon) to manage this uncertainty.

Challenge 2: Network Latency

The second fallacy, "latency is zero," is equally important. In your monolithic experience, a function call takes nanoseconds. A network call takes milliseconds, which is millions of times slower. But the absolute slowness is not the main problem; it's the unpredictability and unboundedness of that slowness.

Network Latency: The delay in data communication over a network. In distributed systems, this delay is not only significant but also variable and has no theoretical upper bound.

Let's explore why this is the case.

The Physics of Latency

Latency isn't just a software problem; it's a physics problem. Even in a perfect network, the speed of light imposes a minimum delay.

The combination of physical limits (speed of light), network infrastructure behavior (congestion, routing), and software behavior (garbage collection, CPU contention) means you can never guarantee a maximum response time.

This has two critical implications for system design:

1. Timeouts are essential: You cannot wait forever for a response that may never come. You must set aggressive timeouts.
2. Latency Amplification: As the reading highlighted, when a single API call fans out to multiple downstream microservices, the probability of encountering one slow response increases dramatically. If a single service call has a 1% chance of being slow (p99 latency), and your user request involves 50 such calls, you have a nearly 40% chance of that user request being slow. $$P(\text{slow request})=1-P(\text{all fast}{)}^n=1-0.99^{50}\approx 0.395$$

At scale, the "unlikely" long-tail latency events become certainties that happen every second.

Conclusion

In this lesson, we've taken the first and most important step into the world of distributed systems. We've replaced the comforting certainties of monolithic development with the harsh, unavoidable realities of network communication.

Key Takeaways:

• Mindset Shift: Moving to microservices requires a fundamental shift from assuming reliability (monolith) to designing defensively for failure (distributed).
• Partial Failure is Inevitable: Unlike a monolith that fails completely, distributed systems can fail in parts, leaving the overall system in an ambiguous state.
• The UNKNOWN State is the Core Problem: Because of partial failure and timeouts, you often cannot know if a remote operation succeeded or failed. Designing for this uncertainty is a primary task of a distributed systems engineer.
• Latency is Unbounded and Unpredictable: Network calls are not just slow; their slowness is variable and has no upper limit. This "long tail" of latency dramatically affects user experience at scale.

These two challenges—partial failure and network latency—are the source of almost all other problems in distributed systems. They are why we will need to learn about consistency models, retries, idempotency, circuit breakers, and all the other patterns we'll cover in this course.

Preview of the Next Lesson:

Now that we understand these foundational challenges, how do we begin to reason about them formally? In our next lesson, we will introduce the CAP Theorem. This famous theorem provides a powerful framework for understanding the fundamental trade-offs you must make between Consistency, Availability, and Partition Tolerance when designing a system that lives with partial failure and network latency.