Leader-Follower Replication: Failover and Read Scaling

1. The Core Problem: Single Points of Failure and the Need for Replication

At the heart of distributed system design is the elimination of Single Points of Failure (SPOFs). Any component that, upon failure, brings down the entire system (or a part of it) is a SPOF. A single database server—or a single shard from our last lesson—is a classic SPOF.

• Technical Explanation:
  The solution to a SPOF is redundancy. For databases, this means replication: creating and maintaining multiple copies of the data on different servers, called replicas. If one replica fails, the system can continue operating using another.

  The most common architecture to manage replicas is the Leader-Follower pattern (also known as Primary-Secondary or Master-Slave).

  • The Leader (Primary): There is one designated leader node. It is the sole authority for all write operations (INSERT, UPDATE, DELETE). This design elegantly solves the problem of conflicting writes by creating a single, ordered history of all data changes.
  • The Followers (Secondaries/Replicas): There are one or more follower nodes. They receive a stream of data changes from the leader and apply them to their own local copy of the data. Followers are typically used to serve read requests, which allows the system to scale its read capacity.

This diagram shows the fundamental Leader-Follower architecture. Clients send all writes to the Leader. The Leader then replicates these changes to the Followers. Read requests can be served by either the Leader or any of the Followers, enabling read scaling and high availability.

• Metaphor for a 5-Year-Old:
  Imagine a chef (the Leader) who is the only person allowed to write new recipes in the master cookbook. After writing a new recipe, the chef makes photocopies and gives them to several apprentice cooks (the Followers). When someone wants to know how to make a dish (a read request), they can ask any of the apprentices, which frees up the head chef to focus on creating new recipes (handling writes). If an apprentice gets sick, others can still answer questions. If the head chef gets sick, one of the apprentices must be promoted to take over the master cookbook.

• Real-World Context:
  This pattern is ubiquitous. It's the default replication model for most relational databases like PostgreSQL, MySQL, and MS SQL Server. It's also used in NoSQL databases like MongoDB and message queues like Apache Kafka. Any system that needs both high availability and the ability to handle a high volume of reads benefits from this design.

2. Under the Hood: Log-Based Replication

How exactly does the data travel from the leader to the followers? This is not an ad-hoc process; it relies on a simple but powerful data structure: the log.

• Technical Explanation:
  The leader records every data modification (write) as an entry in a durable, append-only replication log. This log is a strictly ordered sequence of operations. Think of it as the definitive, chronological history of every change made to the database.

  Followers connect to the leader and "subscribe" to this log. They read the log entries sequentially and apply the same operations to their local data in the exact same order. Each follower keeps track of the last log entry it has processed, often called an offset or a Log Sequence Number (LSN). If a follower disconnects and reconnects, it simply tells the leader the offset it needs to resume from, ensuring it doesn't miss any updates. This log-centric design guarantees that all followers will eventually converge to the same state as the leader.

• Metaphor for a 5-Year-Old:
  Think of the replication log as a numbered instruction manual. The Leader (chef) writes down every step: 1. Add 1 cup of flour, 2. Mix eggs, 3. Set oven to 350F. The Followers (apprentices) all have a copy of this manual. They read and perform the steps in the exact numerical order. If an apprentice takes a break after step 2, they know to come back and start at step 3. They never skip a step or do them out of order, so all their cakes end up identical to the leader's.

• Python Code Snippet:
  This conceptual code illustrates the log-based mechanism. The leader appends commands to a log, and followers pull changes based on their last known offset.

  import time
  
  class ReplicationLog:
      """A simplified representation of the leader's write-ahead log."""
      def __init__(self):
          self.log = []
          self.offset = 0
  
      def append(self, command):
          """Appends a command to the log."""
          entry = {'offset': self.offset, 'command': command}
          self.log.append(entry)
          print(f"LEADER: Wrote '{command}' to log at offset {self.offset}.")
          self.offset += 1
  
      def get_changes(self, follower_offset):
          """Returns log entries from a given offset."""
          if follower_offset < self.offset:
              return self.log[follower_offset:]
          return []
  
  class Follower:
      """A simplified follower node."""
      def __init__(self, name, leader_log):
          self.name = name
          self.data = {}
          self.leader_log = leader_log
          self.processed_offset = 0
          print(f"FOLLOWER {self.name}: Initialized at offset 0.")
  
      def sync_with_leader(self):
          """Reads the log from the leader and applies changes."""
          print(f"FOLLOWER {self.name}: Requesting changes since offset {self.processed_offset}.")
          changes = self.leader_log.get_changes(self.processed_offset)
          
          if not changes:
              print(f"FOLLOWER {self.name}: Already up to date.")
              return
  
          for entry in changes:
              # In a real system, this would execute the command (e.g., SQL)
              print(f"FOLLOWER {self.name}: Applying offset {entry['offset']}: {entry['command']}")
              key, value = entry['command'].split('=')
              self.data[key.strip()] = value.strip()
              self.processed_offset += 1
          
          print(f"FOLLOWER {self.name}: Sync complete. New offset: {self.processed_offset}. Data: {self.data}")
  
  # --- Simulation ---
  leader_log = ReplicationLog()
  follower1 = Follower("Replica-A", leader_log)
  follower2 = Follower("Replica-B", leader_log)
  
  leader_log.append("user_count=100")
  time.sleep(1)
  follower1.sync_with_leader()
  
  leader_log.append("active_sessions=5")
  leader_log.append("error_rate=0.01")
  time.sleep(1)
  follower1.sync_with_leader()
  follower2.sync_with_leader()
  

3. Critical Trade-offs: Replication Modes

The leader-follower pattern seems straightforward, but a critical design choice determines the system's trade-offs between performance, consistency, and durability: when does the leader acknowledge a write as successful to the client?

To understand this, watch this excellent explanation of the different replication strategies.

Here's a summary of what you've just watched:

1. Asynchronous Replication:
   • How it works: The leader writes the data to its local log and immediately sends an "OK" back to the client. The replication to followers happens in the background.
   • Pros: Very low write latency (fast writes for the user), high availability for writes (the leader doesn't get blocked by slow or failed followers).
   • Cons: Potential for data loss. If the leader crashes after acknowledging the write but before the change is replicated to any follower, the write is lost forever. This also leads to replication lag, where followers are temporarily out of sync with the leader, meaning clients might read stale data.

*This image illustrates replication lag. An update happens on the Leader at t=0, but the Follower only receives it at t=5s. Any read query to the Follower during that 5-second window will see old, stale data.*

2. Synchronous Replication:
* How it works: The leader writes the data to its local log, sends the change to all followers, and waits for a confirmation from every single one before sending an "OK" back to the client.
* Pros: Guaranteed consistency and durability. A write acknowledged by the leader is guaranteed to be on all replicas, so there's no data loss on leader failure. Reads from any follower are always up-to-date.
* Cons: Terrible write performance and low availability. The total write latency is determined by the slowest follower. If any follower is down or slow, all client writes are blocked, making the system unavailable for writes.

3. Semi-Synchronous Replication (The Practical Compromise):
   • How it works: The leader writes to its local log, sends the change to all followers, but only waits for confirmation from at least one follower before acknowledging the write to the client. The other followers catch up asynchronously.
   • Pros: A great balance. It offers much stronger durability than asynchronous replication (data is guaranteed to be on at least two machines) while having much better write performance and availability than fully synchronous replication.
   • Cons: There's still a small chance of data loss if both the leader and the single synchronous follower crash simultaneously. Stale reads on the asynchronous followers are still possible.

The following table from the "Understanding Leader-Follower Replication" article (Resource ID LINK) is a perfect summary:

4. When Things Go Wrong: The Failover Process

Designing for failure is paramount. While handling a follower failure is simple (it just stops serving reads and tries to catch up), handling a leader failure is a complex, multi-step process called failover.

• Technical Explanation:
  The goal of failover is to automatically promote one of the followers to become the new leader, ensuring the system remains available for writes. This involves three critical phases:

  1. Failure Detection: The system must realize the leader is down. This is typically done using timeouts. Followers might periodically "ping" the leader, or the leader might be expected to send regular "heartbeat" messages. If heartbeats stop for a configured period, the leader is presumed dead.
  2. Leader Election: The remaining followers must coordinate to choose a new leader. This is a complex distributed consensus problem. The most up-to-date follower (the one with the highest log offset) is the ideal candidate to minimize data loss. Often, a quorum-based algorithm (like Raft or Paxos, which we will cover later) or an external coordinator service (like ZooKeeper or a Sentinel process) manages this election.
  3. System Reconfiguration: Once a new leader is elected, the system must be reconfigured. The new leader starts accepting writes. All other followers must be reconfigured to start consuming the replication log from the new leader. Clients and load balancers must be updated to route write traffic to the new leader's address.

• Metaphor for a 5-Year-Old:
  The head chef suddenly faints (leader failure).
  1. Detection: The apprentices notice they haven't received a new recipe page in a while. They try calling the chef's name but get no response. They agree the chef is down.
  2. Election: The apprentices huddle up. They compare their stacks of recipe pages. The apprentice who has the most pages (the most up-to-date) is chosen to be the new head chef.
  3. Reconfiguration: The newly promoted chef puts on the big chef's hat and starts writing in the master cookbook. All other apprentices are told to now get their copies from this new chef. The restaurant manager tells all the waiters to bring new recipe ideas to the new chef.

5. Elite Interview Deep Dive: Failover Pitfalls

Explaining the happy path for failover is good; discussing the pitfalls is what separates a senior candidate from a junior one. An elite interview will probe your understanding of what can go disastrously wrong.

• Technical Explanation:

  1. Split-Brain: This is a dangerous condition where, due to a network partition, the followers mistakenly believe the leader is dead and elect a new one. The original leader, however, is still alive and thinks it's in charge. You now have two active leaders accepting writes, leading to data divergence, conflicts, and corruption. A common prevention mechanism is for the old leader to require a "fencing token" or confirmation from a quorum of nodes before it can commit a write. If it can't reach the quorum, it steps down.

  2. Data Loss with Asynchronous Failover: In an asynchronous or semi-synchronous setup, if the leader fails, the newly promoted follower may not have all the writes the old leader acknowledged. For instance, the old leader might have acknowledged write_A to a client and then immediately crashed before replicating it. The new leader will never know about write_A. This acknowledged-but-unreplicated data is lost. The system has sacrificed perfect durability for availability. This is a fundamental trade-off you must be able to articulate.

  3. Cascading Failures: A poorly configured failover timeout can cause issues. If the timeout is too short, a temporary network glitch or a brief leader slowdown (e.g., due to garbage collection) can trigger an unnecessary and costly failover. If the timeout is too long, the system remains unavailable for writes for an extended period, impacting users.

Conclusion

Today you have designed a core component of any highly available system. You've moved beyond simply partitioning data to making those partitions resilient.

Key Takeaways:

• Leader-Follower Replication is the foundational pattern for providing high availability and read scalability, with a single Leader for writes and multiple Followers for reads.
• The Replication Log is the "under the hood" mechanism that ensures followers can reliably reproduce the leader's state.
• Replication Modes (Async, Sync, Semi-Sync) present a fundamental trade-off triangle between Consistency, Performance, and Durability. Semi-synchronous is often the pragmatic choice for many systems.
• Failover is a complex, three-step process (detection, election, reconfiguration) required to automatically recover from a leader failure.
• Real-world systems must defend against dangerous pitfalls like Split-Brain and be designed with a clear understanding of the data loss risks inherent in non-synchronous replication.

Next Lesson Preview

We've now established a system with multiple replicas managed by a single leader. But the leader election process itself is a major challenge. How do a group of distributed servers agree on who the new leader should be, especially in the face of network delays and failures? In our next lesson, we will dive into the distributed consensus problem and explore the Raft algorithm, a groundbreaking and understandable protocol designed to solve exactly this problem.