Tracing Writes and Failover Hazards in Single-Leader Replication

Good to see you again. In the previous lesson, you established the key acknowledgement trade-off in replication: synchronous replication places a follower’s confirmation on the client’s write path, while asynchronous replication lets the leader acknowledge after its own local commit. That choice determines both the window in which acknowledged writes might be lost and the extent to which a slow follower can delay writes.

Now we will make that model operational. You will trace what actually happens after a client sends a write to a single leader, see why followers can recover relatively simply, and then examine why replacing a failed leader is much more hazardous. The central question is not merely “can we promote a replica?” but: how can the system safely decide who is allowed to accept writes, and what happens to writes that replicas disagree about?

One writer, an ordered replication log

In single-leader replication, exactly one replica is designated as the leader. All writes must be directed to it. The remaining replicas, called followers, receive the leader’s changes and reproduce them locally. Reads may be served by either the leader or followers, subject to the freshness limitations introduced by replication lag.

The important structural constraint is that followers do not independently accept ordinary client writes. Instead, each follower receives the leader’s sequence of changes. The leader therefore establishes a single order for writes, even when several clients submit requests concurrently.

For example, suppose two users update the same profile:

1. Client A changes a user’s display name.
2. Client B changes that user’s profile picture.
3. The leader decides an order in which those writes commit.
4. It records that order in its replication log.
5. Each follower processes the same ordered records.

If every replica applies the same changes in the same order, they can converge on the same state. The replication log is thus more than a transport mechanism: it is the shared history from which followers reconstruct the leader’s state.

A follower’s position in the log gives a precise meaning to “how far behind it is.” If follower $F$ has applied through log position 850 and the leader has appended through 875, $F$ is missing positions 851 through 875. It must not apply position 860 first: applying records in order preserves the leader’s history.

This ordered-log model also makes follower failure comparatively routine. If a follower crashes or loses its network connection, the leader can continue accepting writes. When the follower returns, it reports its last safely processed position and fetches the missing suffix of the log. It then replays those records sequentially until caught up.

Follower recovery is easy because there remains an authoritative leader and a canonical log. Leader recovery is difficult precisely because that authority may be unavailable.

Tracing one write from client to replicas

Let us trace a concrete request:

User 1234 changes their profile picture from me-old.jpg to me-new.jpg.

The exact persistence details vary among databases, but the general single-leader write path has the following stages.

1. The client sends the write to the leader.
   A load balancer, driver, service-discovery layer, or database proxy must know which node currently holds the leader role. Sending the write to a follower is normally rejected or redirected.

2. The leader validates and commits the change locally.
   It executes the update under the database’s concurrency-control rules and records the outcome in durable local state, typically including a write-ahead log or an equivalent commit record. At this point, the leader’s database state includes the new picture URL.

3. The leader places the change in its replication stream.
   The precise representation may be a low-level storage log record, a row-level change, or another ordered replication format. Later in this chapter, you will compare those mechanisms. For this lesson, what matters is that the follower receives the committed changes in leader-defined order.

4. Followers receive and apply the replicated change.
   Each follower consumes the stream at its own pace. A follower may first receive the record, durably store it, and only later apply it to structures used for reads. Those are distinct milestones.

5. The leader acknowledges the client according to its replication policy.
   With asynchronous replication, it can reply after its local commit. With synchronous replication, it waits for confirmation from one or more required followers before replying.

The write path has one decisive asymmetry: a client’s request goes to the leader, but the replication stream goes from the leader to followers. Followers receive the leader’s committed history; they are not peers negotiating the outcome of each client write.

The next diagram illustrates a hybrid configuration: the leader waits for Follower 1 but not Follower 2. Thus, Follower 1 is synchronous for this write and Follower 2 is asynchronous.

Read the diagram from left to right:

• The leader receives and records the data change.
• It streams the change to both followers.
• Follower 1 confirms in time to satisfy the synchronous requirement.
• The leader returns success to the client.
• Follower 2 applies the same change later and then confirms.

At the instant the client receives success, the two followers are not necessarily equal. The acknowledgement contract tells you what is safely known: the leader and the required synchronous follower have met the configured condition. It does not imply that every follower has applied the write or that every subsequent read from every replica will observe it.

Leader failover: replacing the authority for writes

A leader failure removes the node that defines the current write history and accepts new writes. To restore write availability, the system must promote a follower to become the new leader. This process is failover.

Conceptually, a failover has three jobs:

1. Decide that the existing leader should no longer be treated as leader.
2. Choose a follower to become the new leader.
3. Reconfigure clients and remaining replicas around the new leader.

Choosing the most up-to-date follower reduces the amount of history that could be missing after promotion. If one follower has processed through position 875 and another only through 850, promoting the first follower preserves more of the leader’s known history.

After promotion, clients must send writes to the new leader. The remaining followers must stop consuming from the old leader and begin consuming the new leader’s replication stream. If the former leader eventually returns, it must not simply resume its old role. It needs to be brought back under the new leader’s history, often as a follower.

This sounds procedural, but every stage contains ambiguity. The main lesson of failover is that a distributed system often cannot immediately distinguish “the leader is dead” from “the leader is alive but temporarily unreachable.”

Hazard 1: failure detection is uncertain

Suppose follower $F_1$ stops receiving messages from the leader. Several realities are consistent with that observation:

• The leader process crashed.
• The leader’s host lost power.
• The network path between $F_1$ and the leader failed.
• The leader is overloaded and responding too slowly.
• $F_1$ itself is unhealthy or partitioned from the rest of the system.

A timeout can make the system take action: “if no heartbeat arrives within this interval, begin failover.” But a timeout does not prove that the old leader has crashed. It merely says the system has waited long enough that continuing to wait is unacceptable.

This creates a trade-off:

The choice is operational rather than purely mathematical. Networks can delay messages, and overloaded machines can pause unpredictably. A correct design must assume that a node can be temporarily unreachable without being permanently dead.

Hazard 2: acknowledged writes can be missing from the promoted follower

This is the central risk of asynchronous replication.

Assume the leader has committed log positions 851 through 875 and acknowledged all of them to clients. A follower, however, has only received through position 850. The leader then fails permanently, and the system promotes that follower.

The new leader’s history ends at 850. It cannot reproduce positions 851 through 875, because the only node that held them is gone. The previously acknowledged writes have been lost from the surviving system state.

This is not a failure of the new leader to “catch up.” There is nothing from which it can catch up. The authoritative history was lost with the old leader.

Synchronous replication narrows this risk but does not make failover generally simple. If the leader acknowledged a write only after a particular follower stored it, then promoting that qualifying follower can preserve the write. But the system must know which replica actually holds the needed history, and it must still prevent an unrecognized or recovering old leader from accepting conflicting writes.

There is another complication if the old leader later returns. It might contain writes that were not present on the promoted follower, while the new leader may now have accepted entirely new writes. The two histories diverge. A system may discard the old leader’s unreplicated writes, attempt conflict handling, or require manual intervention depending on its design. None of these choices can retroactively make the two histories identical without defining which writes win.

A useful failover audit question is:

For every write that the old leader acknowledged, which surviving node is known to contain it durably?

If the answer is “none,” then that write is exposed to loss during a leader failure.

Hazard 3: split brain creates two writers

The most dangerous failure mode is split brain: two nodes both believe they are authorized to be leader and both accept writes.

One common path to split brain is:

1. A network partition prevents some replicas from communicating with the old leader.
2. Those replicas time out and promote a new leader.
3. The old leader was never actually dead; it remains reachable by some clients.
4. Both old and new leaders accept writes independently.

Now two logs are being extended independently. One client may write an order cancellation to the old leader while another writes a shipment confirmation to the new leader. Neither leader has a complete ordering of both operations, and followers may receive incompatible histories.

The operational requirement is therefore stronger than “promote a replica quickly.” A system must ensure that the old leader is fenced off: once a new leader is authorized, the old leader must no longer be able to perform writes, even if it wakes up believing it still owns the role.

Mechanisms for doing this vary by system, but their shared purpose is clear:

• establish one authoritative leader;
• ensure clients route writes only to it;
• prevent the prior leader from continuing to modify shared state;
• require a recovered former leader to rejoin as a follower.

The deeper problem—getting distributed nodes to agree safely on leadership—will return in the later consistency-and-consensus module. For now, retain the practical implication: failover must preserve single-writer authority, not merely start another database process.

A disciplined way to reason about any failover

When reading the rest of Chapter 5 or evaluating an architecture, describe an incident in terms of log positions and acknowledgement rules rather than vague claims that a replica is “up to date.”

For example:

This language separates three conditions that are easy to conflate:

• Committed on the old leader: the old leader accepted the write.
• Replicated to a follower: another node received a copy of the write.
• Preserved after failover: the promoted leader’s history includes the write.

Only the last condition tells you whether a write remains in the active system after leadership changes.

Key takeaways

A single-leader replication system concentrates all writes at one leader, which records an ordered replication log. Followers consume that ordered log and apply it sequentially; when a follower fails, it can usually resume from its last recorded log position and catch up.

A write normally travels from client to leader, is committed and placed in the replication stream, then reaches followers. The leader’s acknowledgement policy determines whether it returns success immediately or waits for a required follower confirmation.

Leader failover is difficult because it must safely replace the authority that accepts writes. Its major hazards are:

• uncertain failure detection, because silence may mean a partition or delay rather than a crash;
• lost acknowledged writes, when asynchronous followers have not received the old leader’s latest committed history;
• split brain, when an old and a new leader both accept writes and create divergent histories.

Next, you will examine the different forms the replication log can take: statement-based replication, write-ahead-log shipping, logical replication, and trigger-based replication.