Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 7

Leader Election with ZooKeeper/etcd

Recommended resources

Welcome to the next lesson in our exploration of consensus and coordination. In our previous session, we did a deep dive into the Raft consensus algorithm, understanding how a cluster of servers can internally elect a leader and replicate a log to maintain a strongly consistent state. This internal leader election is the foundation upon which many distributed systems are built.

Today, we'll build directly on that foundation. We will shift our focus from how a system like etcd or ZooKeeper elects its own leader to how an application can use such a service to manage leadership among its own distributed instances. This is a crucial pattern for any scenario where only one process in a group should be active at a time, such as a job scheduler, a singleton service, or a node responsible for partitioning data.

By the end of this lesson, you will be able to apply common leader election patterns using consensus-based coordination services like ZooKeeper and etcd. This will move your understanding from the theoretical mechanics of consensus to its practical application, a key skill for designing robust systems and a common topic in senior engineering interviews.

1. The Role of a Coordination Service

Before we dive into patterns, it's worth asking: why use a separate service for coordination? Given your experience building distributed backends, you know that implementing consensus correctly is notoriously difficult. Instead of having every distributed application implement its own version of Raft or Paxos (a recipe for subtle, hard-to-debug bugs), it's far more effective to rely on a specialized, battle-tested service.

This is the role of systems like ZooKeeper and etcd. They are not general-purpose databases; they are specialized coordination engines that expose a simple set of primitives to applications. These primitives allow developers to build complex coordination patterns like leader election, distributed locks, and service discovery without needing to manage the underlying consensus protocol.

Let's begin with a video that frames the problem of leader election from an application perspective.

How do computers elect leaders? | Consensus and Leader Election Explained

This short video from "A Dev' Story" introduces the concept of leader election and why it's needed for coordination in distributed systems.

Watch the initial segment that defines leader election and its purpose. Then, skip to the end and watch the brief segment on simpler alternatives using a key-value store. This nicely sets the stage for the patterns we'll explore.

The video mentions solving this problem with a key-value store using TTLs and atomic operations. This is exactly the approach taken by modern coordination services. They abstract away the complexity of the underlying consensus algorithm and provide these powerful primitives as a service.

The following article on ZooKeeper provides an excellent explanation of why these systems are a fundamental building block.

2. Pattern 1: Leader Election with ZooKeeper

ZooKeeper is one of the original and most well-known coordination services. It is built on its own consensus protocol, ZAB (ZooKeeper Atomic Broadcast), which is conceptually similar to Raft in that it's a leader-based protocol for state machine replication.

To implement leader election, ZooKeeper provides a few simple but powerful primitives:

  • Znodes: A hierarchical, filesystem-like namespace of data nodes.
  • Ephemeral znodes: These znodes are tied to a client's session and are automatically deleted if the session terminates (e.g., due to a client crash or network partition). This is the primary mechanism for failure detection.
  • Sequential znodes: When creating a znode, you can request that ZooKeeper append a monotonically increasing sequence number to its name. This provides a global ordering for events.
  • Watches: Clients can set a one-time trigger (a watch) on a znode to be notified of changes, such as its deletion or modification.

The Recipe for Leader Election

Combining these primitives leads to a very elegant and robust leader election algorithm:

  1. A predefined parent znode is used for the election (e.g., /election).
  2. Each process that wants to become a leader (a candidate) creates an ephemeral, sequential znode under the parent path (e.g., /election/candidate-0000000001).
  3. After creating its znode, each candidate gets the list of all children under /election and checks if its znode has the lowest sequence number.
  4. The candidate with the lowest sequence number becomes the leader.
  5. All other candidates become followers. A follower does not watch the leader's znode. Instead, to avoid a "thundering herd" problem where all followers are notified at once upon leader failure, each follower watches the znode with the next-lowest sequence number to its own.
  6. If the leader process fails, its ephemeral znode is automatically deleted by ZooKeeper. The follower that was watching that znode gets a notification, checks if it now has the lowest sequence number, and if so, takes over leadership.

This pattern is highly effective because it serializes the failover process, ensuring only one follower reacts to the leader's failure at a time. The following article describes this pattern in detail.

3. Pattern 2: Leader Election with etcd

etcd is a more modern coordination service that is a core component of Kubernetes. You'll appreciate that it uses the Raft algorithm, which we studied in the last lesson, for consensus.

This diagram shows the high-level architecture of etcd. The Raft module is the heart of the system, providing the consensus that underpins all other features. The server exposes APIs that applications use to access these features.

etcd's approach to leader election is conceptually similar to ZooKeeper's but uses a different set of primitives:

  • Keys: A simple key-value store, analogous to znodes.
  • Leases: An explicit time-based primitive. A client creates a lease with a Time-To-Live (TTL). The client must periodically send "keep-alive" messages to refresh the lease. If the client fails to do so, the lease expires, and all keys attached to it are automatically deleted. This serves the same purpose as ZooKeeper's ephemeral znodes.
  • Transactions: etcd supports atomic transactions that can bundle multiple operations, including a conditional check. This is often used for a Compare-And-Swap (CAS) operation, which is critical for safely acquiring leadership.

The Recipe for Leader Election

The typical leader election pattern in etcd is as follows:

  1. A candidate creates a lease with a specific TTL (e.g., 15 seconds).
  2. The candidate then attempts to create a "leader key" (e.g., /election/leader) using a transaction. The transaction is conditional: it only creates the key if it does not already exist. The key is created with the candidate's unique ID as its value and is attached to the lease from step 1.
  3. If the transaction succeeds, the candidate has acquired leadership. It must now continuously send keep-alive pings to prevent the lease from expiring.
  4. If the transaction fails, it means another candidate has already created the key and is the current leader.
  5. The follower then watches the leader key for a DELETE event.
  6. If the leader fails, it stops sending keep-alives. Its lease expires, and etcd automatically deletes the leader key. The watch is triggered, and all waiting followers can now attempt to acquire leadership by returning to step 2.

The Go client for etcd provides a high-level concurrency package that abstracts this logic away, making it very easy to implement. Given your experience with backend languages, reviewing this implementation is instructive.

How to Implement Leader Election with etcd - OneUptime

This article provides a fantastic, code-driven walkthrough of the etcd leader election pattern.

Start by studying the introductory section and the sequence diagram, which clearly illustrates the flow. Then, review the code in section 1. Basic Leader Election with etcd. You don't need to understand every line of Go, but focus on the key concepts: concurrency.NewSession(client, concurrency.WithTTL(15)): Creating a session backed by a lease with a TTL. election.Campaign(ctx, le.instanceID): The blocking call where the client competes for leadership. election.Resign(ctx): How a leader gracefully steps down. election.WatchLeader(...): How followers can observe leadership changes.

4. Preventing Split-Brain: Fencing

A critical problem in any leader-based system is "split-brain." Imagine a leader experiences a long garbage collection pause or a temporary network partition. The other nodes, seeing it as unresponsive, will elect a new leader. However, the old leader might wake up and, unaware it has been deposed, continue performing operations. This results in two active leaders, potentially corrupting data.

The solution is fencing.

This diagram illustrates the core idea of fencing. A leader is issued a token (like a term number or epoch). When it performs an action on a shared resource, that resource must verify that the token is still the most current one. This prevents stale leaders from causing harm.

A fencing token is a monotonically increasing number associated with each leadership term. This could be:

  • The term number in Raft.
  • The epoch and zxid in ZooKeeper.
  • The revision number of the key in etcd.

When a new leader is elected, it obtains a new, higher fencing token. Before performing any write operation on a shared resource (like a database or a storage service), the leader must present its token. The resource, in turn, must validate that the token is the most recent one it has seen. If it receives a request with a stale (lower) token, it must reject the operation.

This mechanism effectively "fences off" old leaders, ensuring that even if they are alive, they cannot modify the system's state. Articulating the split-brain problem and the fencing solution is a hallmark of a senior engineer's understanding of distributed systems.

Conclusion

In this lesson, we bridged the gap between the theory of consensus and its practical application. We saw how coordination services like ZooKeeper and etcd leverage their underlying consensus protocols to provide simple, powerful primitives for applications.

Key Takeaways:

  • Coordination services (e.g., ZooKeeper, etcd) abstract the complexity of consensus protocols like Raft, providing primitives for building distributed coordination patterns.
  • The ZooKeeper leader election pattern relies on ephemeral-sequential znodes to create an ordered, fault-tolerant list of candidates, elegantly avoiding the "thundering herd" problem.
  • The etcd leader election pattern uses leases for failure detection and atomic transactions (Compare-And-Swap) to safely acquire leadership.
  • Fencing is a critical technique to prevent split-brain scenarios, where a deposed leader continues to operate. It works by issuing monotonically increasing tokens (epochs, revision numbers) to each new leader.

You now have a framework for discussing and applying one of the most fundamental patterns in distributed systems. This knowledge is not only essential for building reliable services but also for demonstrating depth in a system design interview.

In our next lesson, we will explore a closely related topic: distributed locking. You will see how the same primitives used for leader election can be adapted to create distributed mutexes, and we will analyze the trade-offs and failure modes of different implementations, including the famous Redis Redlock algorithm.