Welcome back. In our previous lesson, we broke down the core mechanics of Raft: how it elects leaders, replicates logs, and safely manages cluster membership changes. You saw how its design, centered on understandability, creates a robust foundation for consensus. We established that Raft works reliably under normal conditions and planned configuration changes.
Now, we will put Raft to a more rigorous test. The true measure of a consensus algorithm is its resilience during failures, especially network partitions. In this lesson, you will analyze how Raft maintains safety and liveness during network partitions, including leader isolation scenarios. Understanding these failure modes is not just an academic exercise; for a Principal Engineer, it is fundamental to designing systems that can withstand the chaotic reality of production environments and to diagnosing them when they fail.
The Challenge of Network Partitions
A network partition occurs when communication failures split a cluster into two or more groups of servers that cannot communicate with each other. This presents two fundamental challenges to a consensus algorithm:
- Safety: The system must not violate its consistency guarantees. The most dangerous failure is a "split-brain" scenario, where each partition elects its own leader, and the two leaders proceed to accept different client commands, creating two divergent, irreconcilable versions of history.
- Liveness: The system should continue to make progress (i.e., remain available to process client requests) if possible. A partition should not cause the entire system to halt indefinitely.
Raft's design directly addresses the safety challenge through a single, powerful principle: all decisions require a majority.
Safety: The Majority Rule
For a Raft cluster to do anything meaningful, it must achieve a quorum, or a majority vote.
- A candidate can only be elected leader if it receives votes from a majority of servers.
- A leader can only commit a log entry if that entry has been replicated to a majority of servers.
Consider a 5-node cluster that gets partitioned into a minority group of 2 nodes and a majority group of 3 nodes.
- The Minority Partition (Partition A): This group has only 2 out of 5 nodes. It cannot achieve a majority. It cannot elect a new leader, nor can the existing leader (if it's in this partition) commit any new log entries. The system is unavailable in this partition, but it remains safe—it cannot make any progress that would conflict with the other partition.
- The Majority Partition (Partition B): This group has 3 out of 5 nodes. It can achieve a majority. If the original leader was in Partition A, the nodes in Partition B will time out and one will successfully be elected as the new leader for a higher
term. This new leader can then accept and commit new log entries. The system remains live in this partition.
This majority rule makes a split-brain impossible. Since the two partitions are disjoint, they cannot both contain a majority of the cluster's nodes. Therefore, at most one partition can ever elect a leader and commit entries.
The Role of Terms as Fencing Tokens
When the network partition heals, the old leader from the minority partition (in Term: 2 in the diagram) might try to resume its duties. However, its first communication with the majority partition will expose it to nodes that are now in a higher term (Term: 3). When any Raft server receives a message with a higher term number, it immediately updates its own term and reverts to a follower state.
This mechanism, where the term number invalidates stale leaders, is a form of fencing. It ensures that old leaders, partitioned or otherwise delayed, cannot re-emerge and propose conflicting state.
The following video provides a concise explanation of how term numbers act as fencing tokens and reinforces the rule that a candidate must have an up-to-date log to be elected.
Distributed Consensus - Raft Leader Election | Systems Design Interview 0 to 1 with Ex-Google SWE
This video from the "Jordan has no life" channel explains the mechanics of a Raft election. It clearly illustrates how term numbers are used and the critical log up-to-date check.
First, watch the segment explaining the conditions for voting, paying close attention to the up-to-date log check. Then, watch the explanation of fencing tokens, which describes how term numbers prevent old leaders from causing inconsistencies.
Safety Argument in the Raft Paper
The core of Raft's safety lies in the Leader Completeness Property, which we touched on in the last lesson. It guarantees that a newly elected leader has all log entries that were committed in previous terms. This is ensured by the voting process: since a candidate must get votes from a majority, and any committed entry must exist on a majority, there's an overlap. The "up-to-date" log check during voting ensures the candidate selected from that overlap has all the committed entries.
Let's briefly revisit the safety section of the original Raft paper to solidify this crucial concept.
In Search of an Understandable Consensus Algorithm
This section of the paper formally describes the election restriction and provides a proof sketch for the Leader Completeness Property, which is the cornerstone of Raft's safety guarantees during partitions and other failures.
Please review Section 5.4, Safety. Focus on the logic in subsection 5.4.1 ("Election restriction") and the proof sketch in 5.4.3 ("Safety argument"). The key insight is how the voting mechanism ensures a new leader's log is a superset of all previously committed entries.
Liveness and Recovery
While the majority partition remains live, the minority partition is effectively offline. What happens when the partition heals?
The nodes from the defunct minority partition will eventually receive an AppendEntries RPC (even if it's just a heartbeat) from the legitimate leader in the majority partition. This RPC will contain the higher term number.
- The minority nodes will see the higher term and immediately revert to follower state.
- They will then begin the standard log consistency check process. The leader will find the last point of agreement in their logs and force them to match its own log, overwriting any uncommitted, divergent entries they may have accumulated.
This process is identical to how a server recovers after a crash, demonstrating the elegance of Raft's unified approach to restoring consistency.
From Theory to Practice: Partial Partitions and etcd
So far, we've discussed clean partitions. Real-world failures are often messier. A partial partition can occur where connectivity is asymmetric or intermittent. A server might be able to talk to some peers but not others.
This is where many theoretical models meet their match, and where understanding the specifics of a production implementation like etcd becomes critical. A famous 2020 Cloudflare outage provides a fascinating case study.
[PDF] Examining Raft's behaviour during partial network failures
This paper analyzes the Cloudflare outage and reveals crucial differences between the behavior of "textbook" Raft and the etcd implementation during a partial network partition. It highlights key optimizations that have a profound impact on liveness.
Please read the paper carefully. Start with the introduction and background (Sections 1 and 2) to understand the problem setup and the difference between full and partial partitions. Read Section 4, Reproducing Cloudflare's outage, which details the failure scenario and contrasts the expected Raft behavior with etcd's actual behavior. Pay close attention to the explanation of the CheckQuorum optimization. Finally, read the conclusion and the section on the PreVote optimization (Section 5). This is a critical mechanism for improving liveness in the face of flapping or partitioned nodes.
As the paper details, a naive Raft implementation can suffer from liveness problems during certain partitions. A partitioned node, unable to hear the leader's heartbeats, will time out and start an election. This increments the cluster's term, forcing the stable leader to step down, triggering a disruptive and often fruitless election cycle.
The etcd paper highlights two key, practical optimizations:
CheckQuorum: An optimization where a leader periodically confirms it can still contact a majority. More relevantly for this scenario, followers that are receiving heartbeats from a stable leader may ignoreRequestVotemessages from candidates. This prevented the partitioned node in the Cloudflare case from constantly disrupting the cluster, but it led to performance degradation as client requests sent to the partitioned node would time out.PreVote: This is the modern solution, now default inetcd. Before a node can become a candidate and increment the term, it must first test if it could win an election by sendingPreVoterequests. A partitioned node will fail to get a majority of pre-votes and will thus never start a real election. This prevents a single flapping node from harming the liveness of the entire cluster.
Understanding the trade-offs between the Raft specification and the behavior of production systems with optimizations like PreVote is a hallmark of a principal-level thought process. It's about moving beyond "does it work?" to "how exactly does it fail, and how can we make it more resilient?"
Conclusion
In this lesson, we've scrutinized Raft's behavior under the stress of network partitions, separating its safety guarantees from its liveness characteristics.
Here are your key takeaways:
- Safety is guaranteed by the majority rule. Raft prevents split-brain by requiring a majority for both leader elections and log commitment. At most one partition can ever form a quorum.
- Terms provide fencing. Higher term numbers are used to unambiguously depose stale leaders when partitions heal, preventing them from introducing conflicting state.
- Liveness is a trade-off. Raft deliberately sacrifices liveness in minority partitions to guarantee safety. The majority partition remains available.
PreVoteis crucial for production liveness. In real-world scenarios with flaky nodes or partial partitions, thePreVoteoptimization is essential to prevent disruptive election cycles that can severely impact the cluster's availability.
We've now explored Raft in considerable depth, from its basic operation to its behavior in complex failure scenarios. While we've made comparisons to Paxos along the way, our next lesson will be dedicated to a direct, focused analysis of the two protocols. You will compare Paxos and Raft in terms of understandability and practical implementation trade-offs, solidifying your ability to articulate why Raft has become so prominent and when one might still consider Paxos.