Recommended resources
The freely available resources this lesson will direct you to.
Welcome back. In the previous lesson, you selected partitioning strategies by asking which requests must remain local, which keys can become hot, and where data must live. Partitioning decides which nodes own a record. Replication adds a second question: how many copies should own it, and how many of those copies must respond before a client can trust an operation?
This lesson develops a practical model for reasoning about that choice. You will distinguish replication factor from read and write quorums, use the quorum-intersection rule to predict whether a read can observe an acknowledged write, and evaluate the cost in latency and availability—especially when replicas span datacenters. These ideas apply broadly to Cassandra-style stores, Dynamo-inspired key-value systems, and many managed distributed databases.
Replication creates options—and disagreement
For a given partition key, a replicated data store keeps the same logical record on multiple nodes. The replication factor, usually written , is the target number of replicas.
Suppose the tenant_id = 4821 partition has:
Its data is placed on replicas A, B, and C. If node A fails, copies on B and C can still serve the data. This improves resilience, and can also improve read capacity because multiple nodes may serve reads.
But replicas are distinct machines connected by a network. A write can reach some before others, a node can be slow, or a network partition can make healthy nodes appear unavailable. For a short interval, replicas can legitimately disagree:
| Replica | Value for account_status after a partial write |
|---|---|
| A | active — new value |
| B | active — new value |
| C | pending — old value |
The system must therefore define what “write succeeded” and “read succeeded” mean. It does this through acknowledgment requirements.
- Write quorum : the number of replica acknowledgments required before reporting a write as successful.
- Read quorum : the number of replica responses required before returning a read.
- Replication factor : the total number of intended replicas.
These values concern the acknowledgment path, not necessarily the total number of replication messages sent. In Cassandra, for example, a coordinator normally sends writes to all relevant replicas; a low consistency level means it returns to the client once the required number has acknowledged. The remaining replicas may receive the update later or recover it through repair mechanisms.
The coordinator is a request router and aggregator. It is not necessarily the permanent owner of data and should not be confused with a single database leader.
The failure that quorums prevent
Consider a write at to a three-replica partition. The coordinator receives an acknowledgment from A and immediately tells the client the write succeeded. Before B and C receive the update, a network problem isolates A. A subsequent read directed only to C can return the old value.
This violates read-after-write consistency: after a client receives confirmation that it wrote a value, its later read may appear to make that value disappear.
Martin Kleppmann’s short explanation is worth watching before introducing the formal rule.
Distributed Systems 5.2: Quorums
In “Distributed Systems 5.2: Quorums,” Martin Kleppmann develops the problem from a stale read to the quorum-overlap rule. Watch it for the visual intuition behind why a majority is often enough.
Start with the stale read, where a write reaches one replica but a later read reaches a different, outdated one. Then watch two of three for the concrete quorum example. Continue through the overlap rule and majority quorums; focus on the fact that the read and write response sets must share at least one replica.
The governing condition is:
If this inequality holds, every successful read quorum overlaps every successful write quorum in at least one replica.
For , choosing and gives:
A completed write has been stored on some two replicas. Any later read that receives responses from two replicas must share at least one of those written replicas. That overlapping replica can supply the newer version. If another responder is stale, the coordinator compares version metadata and returns the latest valid version under the store’s conflict-resolution rules.
This is the core benefit of a quorum: it avoids requiring all replicas while still forcing the read and write sets to intersect.
What the inequality does and does not guarantee
The rule is powerful, but it is not a universal synonym for “perfectly consistent.”
It provides an overlap guarantee for an acknowledged write and a later quorum read, assuming the system can identify the newest version correctly. Real guarantees also depend on details such as:
- how writes are versioned and conflicts are resolved;
- whether concurrent writes can occur;
- whether the read is issued after the write has completed;
- whether a system uses strict replica membership or accepts acknowledgments from temporary substitutes;
- whether a stronger coordination protocol, such as consensus, is required for conditional updates.
In particular, quorum reads and writes do not automatically make two concurrent updates serializable or linearizable. A conditional operation such as “decrement inventory only if it is positive” usually needs a consensus or transaction mechanism, not merely ordinary replicated writes.
Choosing , , and
Replication factor is primarily a durability and failure-tolerance choice. Read and write quorum sizes determine the behavior of individual operations.
For a strict quorum over a fixed set of replicas:
- a write can succeed only if at least relevant replicas are reachable;
- a read can succeed only if at least relevant replicas are reachable;
- therefore, a write tolerates at most unavailable replicas;
- a read tolerates at most unavailable replicas.
With , the common configurations look like this:
| Configuration | Overlap? | Write behavior | Read behavior | Failure tolerance |
|---|---|---|---|---|
| No | Fastest acknowledgment; only one replica need respond | Fastest read; may see stale data | Each operation can succeed with two replicas unavailable, but an acknowledged update has a larger durability exposure | |
| Yes | Fast writes | Reads wait for all replicas and fail if one is unavailable | Writes tolerate two failures; reads tolerate none | |
| Yes | Majority acknowledgment | Majority read | Both reads and writes tolerate one unavailable replica | |
| Yes | Every replica must acknowledge | Fast reads after a completed write | Writes tolerate no unavailable replicas; reads tolerate two | |
| Yes | Maximum coordination | Maximum coordination | Neither operation tolerates even one unavailable replica |
The important operational insight is that availability is operation-specific. A system can be highly available for writes and unavailable for reads during the same node outage. Saying only that “the database is available” hides the decision that matters.
Why quorum is a common default
A majority quorum gives both read/write overlap and usable fault tolerance:
For , a majority is 2. For , it is 3.
When both reads and writes use a majority:
- a successful read and write overlap;
- one replica loss is tolerated for ;
- two replica losses are tolerated for ;
- the coordinator need not wait for every replica.
This does not mean that more replicas always make a service faster or safer. Every additional replica consumes storage, network bandwidth, repair work, and operational attention. A replication factor of five spread across five machines in one rack still has a major shared failure domain. Replicas should be distributed across independent hosts, racks or availability zones, and—when required—datacenters.
Latency is determined by the slowest required responder
A coordinator can return only after it has collected enough acknowledgments. That means a quorum operation is exposed to the slowest node among the responses it needs.
For a three-replica write:
- at , latency is roughly the fastest successful replica response;
- at , latency is roughly the second-fastest successful response;
- at , latency is roughly the slowest replica response.
The same logic applies to reads. As required acknowledgments increase, average latency often rises, but tail latency usually becomes the more serious concern. A single slow replica, congested network path, or garbage-collecting process can cause an ALL operation to time out.
A timeout also deserves careful interpretation. It means the coordinator did not receive enough acknowledgments before its deadline. It does not necessarily mean no replica applied the write. Retrying a non-idempotent operation without understanding this distinction can duplicate business effects.
The DataStax documentation expresses the same model and its practical consequence: acknowledgment thresholds trade latency against consistency.
How are consistent read and write operations handled? | DataStax Enterprise | DataStax Docs
Read these sections to connect the general N, R, and W model to a production system with tunable consistency. The documentation clearly separates “where writes are sent” from “when the coordinator may report success.”
In “Tunable consistency,” read the complete explanation. Focus especially on the definition of read and write consistency levels, and on the distinction between sending a write to replicas and waiting for acknowledgments. Then, in “Calculating consistency,” read the quorum condition and examples. Notice how lowering W can preserve read-after-write behavior only when R is raised enough to retain overlap.
Eventual consistency is a deliberate workload choice
If:
a successful read set and a successful write set can be disjoint. A stale read is therefore possible until replicas converge.
For example, with , , and , the write may be acknowledged by A and the subsequent read served from C. If replication has not caught up, C can return an older value.
That outcome is unacceptable for some facts:
- whether a payment was accepted;
- the current authorization decision for a user;
- whether inventory can be committed;
- a customer’s confirmation that a password or account setting changed.
Yet it can be perfectly appropriate for other data:
- observability logs;
- device telemetry;
- clickstream events;
- approximate dashboards;
- a feed where a short delay before a new item appears is tolerable.
The engineering question is not “is eventual consistency bad?” It is:
What incorrect or delayed observation can the business tolerate, for how long, and how will the application behave when it occurs?
A low quorum can improve availability and latency. It also increases the period during which an acknowledged update may be present on too few replicas, creating both a stale-read risk and a durability risk if that sole acknowledged replica fails before replication completes.
Replica convergence is supported by mechanisms such as:
- read repair, where a read detects a stale replica and updates it;
- hinted handoff, where another node temporarily retains a write intended for an unavailable replica;
- anti-entropy repair, periodic comparison and synchronization of replicas.
These mechanisms reduce divergence. They are not a substitute for choosing an acknowledgment policy that meets the immediate correctness requirement.
Availability during a partition: strict and sloppy quorums
So far, assume requests are answered by the partition’s intended replicas A, B, and C. This is a strict quorum model. If a network partition prevents enough of those exact replicas from responding, the operation fails rather than pretending that an arbitrary other node is an equivalent replica.
Dynamo introduced a different availability-oriented idea: the sloppy quorum. If an intended replica is down or unreachable, the system writes a temporary copy to another healthy node outside the usual replica set. That temporary holder records a hint so the data can later be delivered to its intended replica.
This behavior improves write availability during temporary failures. But it changes the clean intersection argument: a read and a write may each meet their nominal counts using different temporary nodes. A simple calculation based on no longer alone establishes that the read contacted an intended replica that holds the acknowledged write.
Dynamo accepts this complexity because some workloads, such as a shopping cart, value accepting user updates during infrastructure failures more than they value immediate single-copy behavior. It resolves conflicts later, potentially with application-specific merge logic.
The relevant sections of the Dynamo paper make the trade-off concrete.
Dynamo: Amazon's Highly Available Key-value Store
Read the Dynamo paper selectively for a production-oriented view of configurable quorums, temporary replica substitution, and the separation between consistency, durability, and availability.
In Section 4.5, “Request Routing,” begin at the quorum protocol; focus on the definitions of R, W, and coordinator behavior. Next, in Section 4.6, “Handling Failures: Hinted Handoff,” read sloppy quorum and hinted handoff. Finally, in Section 6, “Experiences & Lessons Learned,” read the tuning discussion, especially the difference between accepting a write and safely persisting it on several nodes.
Multi-datacenter quorums: local correctness versus global confirmation
Replication across datacenters protects against an entire site failure, but cross-region network latency can dominate request time. A design must state whether an operation needs acknowledgment from another datacenter before success is reported.
Suppose two datacenters each have local replication factor 3:
| Datacenter | Replicas for the partition |
|---|---|
| Alpha | 3 |
| Beta | 3 |
A global quorum can require responses across the combined replica set. This improves the assurance that an update is acknowledged beyond one site, but a normal request may wait for wide-area network communication.
A local quorum requires a majority only among replicas in the client’s local datacenter. With local replication factor 3:
Using local-quorum writes and local-quorum reads in Alpha ensures the Alpha read and write response sets overlap. It provides low-latency, read-after-write behavior within Alpha, assuming the requests stay in that locality. Replication to Beta may still occur, but it need not delay the client response.
This creates an intentional boundary:
- A client writing and then reading in Alpha can obtain a current value under local-quorum assumptions.
- A client in Beta may temporarily see an older value until the cross-datacenter replica converges.
- If Alpha fails immediately after a local-only acknowledgment, Beta’s ability to serve the newest write depends on whether that write reached Beta before the failure.
This is not an implementation defect. It is the consequence of choosing local latency and availability over synchronous global confirmation.
The DataStax consistency overview gives a compact walkthrough of this choice.
DS201.13 Consistency | Foundations of Apache Cassandra
In “DS201.13 Consistency,” DataStax Developers contrasts quorum with low-consistency ingestion and then explains why local quorum matters across datacenters. Watch these segments to relate the abstract equations to workload requirements.
Watch strong read after write for the practical comparison of ALL, ONE, and QUORUM. Then watch eventual consistency use cases to see why logs and telemetry may choose ONE. Finish with local quorum; focus on the distinction between writing to remote replicas and waiting for remote acknowledgments.
A decision method for a data engineer
Choose replication and quorum policies per dataset or operation class, not as a cluster-wide slogan.
1. Define the consequence of a stale read
Ask what happens if a reader observes the previous value for a few seconds. For a telemetry dashboard, perhaps nothing serious. For an entitlement check or transaction state, it may cause an incorrect action.
Also distinguish a stale value from an unavailable response. A system that refuses a request can often be retried; a system that returns a plausible but wrong old value can silently corrupt a workflow.
2. Define the required durability point
Does “write succeeded” mean:
- accepted by one machine and expected to propagate?
- durable on a majority within one availability zone or datacenter?
- durable in at least two datacenters?
- committed through a transaction or consensus protocol?
Do not claim cross-region durability just because remote replication is configured. Check whether the acknowledgment policy actually waits for remote durable copies.
3. Specify tolerated failures by failure domain
“Survive one failure” is incomplete. One process failure, one host failure, one rack failure, and one datacenter outage are different events.
A useful decision statement is:
For customer profile updates, acknowledge only after a quorum of replicas across independent availability zones has persisted the write. Reads for the profile use a quorum that overlaps those writes. Remote-datacenter replication is asynchronous; during a regional loss, a recently acknowledged update may not yet be visible in the recovery region.
That statement makes the residual risk explicit.
4. Measure tail latency, not just the median
A policy that performs well at the 50th percentile may be unusable at the 99th percentile if it waits for a remote replica or a frequently overloaded node. Evaluate:
- replica response-time distributions;
- timeout rates by consistency level;
- unavailable-operation rates;
- replication and repair backlog;
- the age of the newest data available in another datacenter.
5. Validate the application contract
If a client requires “I must see my own accepted update,” its writes and reads need compatible quorum choices and consistent routing assumptions. A low-quorum write followed by an arbitrary low-quorum read does not meet that contract.
Where a workflow needs a single global decision under concurrent updates, use the database’s suitable conditional-write, transaction, or consensus feature. Replication alone is not enough.
Key takeaways
- Replication factor is the number of intended copies; write quorum and read quorum define how many acknowledgments an operation needs.
- The condition
guarantees an intersection between a successful strict quorum read and write, supporting read-after-write behavior when versioning and conflict handling are correct.
- Higher improves the durability of an acknowledged write but makes writes slower and less available. Higher reduces stale-read risk but makes reads slower and less available.
- is a common balanced majority design: both reads and writes can survive one unavailable replica.
- prioritizes low latency and availability, while accepting eventual consistency and a larger durability exposure window.
- Acknowledgment requirements matter more than replication messages alone: sending to all replicas does not mean the client waited for all replicas.
- In multi-datacenter deployments, local quorum can provide low-latency local read-after-write behavior, but it does not imply that another datacenter has already confirmed the update.
- Sloppy quorum and hinted handoff can preserve write availability during failures, but require careful reasoning about convergence and conflict resolution.
Next, we will place these choices into a concrete network-failure scenario using CAP and PACELC. The focus will be not on memorizing labels, but on identifying exactly which latency, consistency, and availability outcomes a system chooses during normal operation and during partitions.