Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 1

Choosing Consistency: User Experience, Performance, and Complexity Trade-offs

Recommended resources

Introduction

Welcome to the first lesson in your course on designing scalable and performant distributed systems. Given your extensive background in engineering leadership within finance, you've undoubtedly navigated the critical demands of system integrity and performance. This module builds on that foundation by formalizing the core principles that govern data in large-scale systems.

Today, we'll focus on a fundamental architectural choice: the consistency model. Our goal is to equip you with a framework to select an appropriate model—whether strong, eventual, or causal—for any given feature. We will move beyond purely theoretical concepts to justify these choices by analyzing the real-world trade-offs between user experience, system performance, and operational complexity. This decision is one of the most consequential in system design, as it dictates how data behaves across a distributed environment and profoundly impacts scalability.

From Theory to Practice: The CAP and PACELC Frameworks

In distributed systems, a core challenge is ensuring that data remains correct and available across multiple nodes, especially in the face of network failures. The classic starting point for this discussion is the CAP theorem, which posits that a system can only provide two of three guarantees: Consistency, Availability, and Partition Tolerance.

However, since network partitions are a fact of life in any distributed architecture, the choice realistically boils down to favoring either consistency or availability during a partition. To get a foundational understanding of this trade-off, let's start with a clear definition of the two most common models: strong and eventual consistency.

Why Eventual Consistency is Preferred in Distributed Systems

This article by Arpit Bhayani provides a concise overview of the core concepts. Focus on understanding the fundamental differences between strong and eventual consistency and how the CAP theorem frames the decision-making process.

Please read the sections defining Strong and Eventual Consistency and then the section on the CAP theorem and its implications.

As you've just read, strong consistency ensures all reads see the most recent write, which is critical in many financial contexts. Eventual consistency prioritizes availability, allowing for temporary data divergence that eventually resolves. The diagram below illustrates this difference in data flow.

This diagram contrasts the data flow in eventual and strong consistency models. In eventual consistency, a write is acknowledged quickly, and data propagates asynchronously, leading to temporary inconsistencies. In strong consistency, a write is confirmed only after it has been replicated to all necessary nodes, ensuring all subsequent reads see the new value.

While CAP is a useful lens, it only describes behavior during a network partition. As a system architect, you know that systems operate in a normal, non-partitioned state over 99.9% of the time. A more practical framework, proposed by Daniel Abadi, is the PACELC theorem.

PACELC states that if there is a Partition, a system must choose between Availability and Consistency (the CAP part). Else (during normal operation), the system must choose between Latency and Consistency. This "Else" clause is critical because it highlights that even without failures, achieving strong consistency has a direct performance cost—latency.

The following resource provides an excellent deep dive into PACELC, classifying real-world databases and explaining why this framework is what senior engineers use for strategic database decisions.

Beyond CAP: PACELC and Real-World Latency Trade-offs | Witty ...

This article frames the consistency discussion in terms of practical, everyday engineering decisions. It explains why consistency always costs latency and how different production systems navigate this trade-off.

Start by reading the introduction to understand the limitations of CAP and the motivation for PACELC. Then, review the deep conceptual breakdown of the theorem. Finally, study the table of production systems and the analyses of DynamoDB and Spanner to see how these trade-offs are realized in practice.

The key insight from PACELC is that consistency is not free. The coordination required for strong consistency (e.g., via consensus protocols like Paxos/Raft or synchronous replication) inevitably adds network round-trip time to your operations. For a high-throughput, user-facing service, adding 10-50ms of latency per write can be prohibitive. This forces a deliberate, feature-by-feature decision about where to place a system on the consistency spectrum.

This chart illustrates the spectrum of consistency models. As you move towards stronger consistency, you typically sacrifice lower latency and higher availability. Eventual consistency provides the highest availability, while strong consistency provides the strictest guarantees.

Applying Consistency Models: A Financial Technology Use Case

Let's ground this in a practical example from your domain. A stock trading platform has different features with vastly different requirements for data correctness and availability. Consider the design of two key components: the trade execution service and the real-time price ticker.

The following video discusses the non-functional requirements for a stock trading app, making an explicit distinction between the consistency needs of these two features.

Stock Trading App System Design Interview | Meta System Design

In this excerpt from a system design interview walkthrough, the presenter, Sanket Singh, outlines the critical consistency and availability requirements for different parts of a stock trading platform.

Watch the segment from the initial discussion where the presenter contrasts the requirements for buy/sell orders versus viewing stock prices. Pay close attention to the justification provided for each choice.

As the video highlights, this is a classic scenario for applying different consistency models within the same application:

  • Trade Orders (Buy/Sell): This is a core financial transaction. Data integrity is paramount. An error here—like a lost order or showing an incorrect balance—has significant financial consequences. Therefore, this subsystem must prioritize strong consistency (PC/EC in PACELC terms). A user would rather have a trade fail or be momentarily unavailable than have it execute incorrectly. The added latency of a consensus protocol to confirm the transaction is an acceptable trade-off for correctness.

  • Stock Price Ticker: This feature requires high availability and low latency to provide a good user experience. Users expect to see price movements in near real-time. If the price ticker is unavailable, the platform is perceived as "down." A slightly stale price (e.g., a few seconds old) is usually acceptable, especially compared to showing no price at all. This makes the price-display system a prime candidate for an eventually consistent model (PA/EL in PACELC terms) that prioritizes availability and low latency.

This bifurcation of requirements within a single product is common in complex systems and underscores why consistency is not a one-size-fits-all decision.

Causal Consistency and Asynchronous Workflows

Between the strict guarantees of strong consistency and the looser model of eventual consistency lies causal consistency. This model is a pragmatic middle ground: it doesn't guarantee a global ordering of all operations, but it does enforce the order of causally related ones. If operation A causes operation B, the system ensures that no process sees B without first having seen A.

A common example is a social media post and its comments. You should not see a comment on a post before you see the post itself. This causal link must be preserved.

In modern distributed architectures, achieving this often involves moving away from traditional, blocking protocols like two-phase commit (2PC), which are brittle and perform poorly at scale. Instead, architects use event-driven patterns to manage complex, multi-step business processes. The video below introduces the patterns that enable this, namely the Saga and Outbox/Inbox patterns.

Consistency and Agreements in Distributed Systems - Jimmy Bogard - NDC London 2025

This talk by Jimmy Bogard explains why traditional distributed transactions are problematic and presents a robust alternative for coordinating work across microservices using the Saga and Outbox patterns, which practically implement a form of causal consistency.

First, watch the segment on Life Beyond Distributed Transactions, where the Outbox and Inbox patterns are introduced as a way to reliably communicate between services. Next, jump to the explanation of the Saga pattern for managing long-running business transactions. Finally, see how these concepts are applied in an e-commerce order fulfillment example.

Let's break down how these patterns work together to enforce causal consistency:

  1. The Outbox Pattern: When a service needs to perform a state change and notify other services (a causal action), it does so within a single, local atomic transaction. It writes the business data to its primary table and writes the event message to an outbox table in the same database. This guarantees that an event is only created if the corresponding state change is successfully committed.

  2. Message Dispatch: A separate process (a message relay or dispatcher) polls the outbox table and publishes the events to a message broker (like Kafka or RabbitMQ). This decouples the business logic from the communication process.

  3. The Saga Pattern: A Saga is a long-running process that coordinates multiple services to execute a business transaction. It's essentially a state machine where each step is triggered by an event. If a step fails, the saga is responsible for executing compensating actions to "undo" the preceding steps, ensuring the system reaches a consistent state, even if it's not the initial one.

In the e-commerce example from the video, the OrderCreated event causes the StockReserved action. The Saga orchestrates this flow, ensuring the stock is not reserved until an order has verifiably been created. This preserves the causal chain of events, providing causal consistency across service boundaries without the overhead of distributed transactions.

Conclusion

In this lesson, we established a practical framework for selecting and justifying consistency models. You've seen that this is a nuanced decision that requires looking beyond theory and analyzing the specific needs of each feature.

Key Takeaways:

  • Consistency is a spectrum. The choice ranges from strong guarantees that prioritize data integrity to weaker models that prioritize availability and performance.
  • PACELC provides a superior framework to CAP for real-world system design by including the trade-off between latency and consistency during normal operation.
  • The right model is feature-dependent. Critical operations like financial transactions demand strong consistency, while less critical features like activity feeds can thrive with eventual consistency.
  • Causal consistency is a powerful middle ground, often implemented with event-driven patterns like Sagas and the Outbox pattern to reliably manage complex, asynchronous workflows across microservices.

In our next lesson, we will build directly on these concepts. Now that we understand the what and why of consistency, we'll explore the how. We will delve into data replication strategies, comparing leader-follower and multi-leader architectures, and address the crucial challenge of resolving write conflicts that arise in distributed environments.