Designing Replication Strategies

Core Replication Architectures

At its core, replication involves a leader node that processes writes and followers that receive copies of that data. However, the configuration of these leaders and followers leads to distinct architectural patterns with different trade-offs.

1. Single-Leader (or Leader-Follower) Replication:
   This is the most common architecture. All write operations are directed to a single designated leader node. The leader then logs these changes and broadcasts them to a set of follower nodes. Reads can be served by either the leader or any of the followers. This model naturally fits read-heavy workloads. As shown in the "Read-Heavy System" diagram below, you can easily scale out read capacity by adding more read replicas. Because all writes are serialized through a single leader, strong consistency is relatively straightforward to achieve, and write conflicts are inherently avoided. The primary drawbacks are that the leader represents a single point of failure for writes and can become a performance bottleneck.

2. Multi-Leader Replication:
   In this setup, two or more nodes can accept write operations. Each leader simultaneously acts as a follower to the other leaders, propagating its changes to them. This architecture is particularly useful for systems with high write throughput requirements or those deployed across multiple geographic data centers, as users can write to their nearest leader, reducing latency. However, this flexibility comes at a significant cost: write conflicts. Since writes can occur concurrently on different leaders, there is no single, authoritative order of operations. Deciding what to do when two clients concurrently modify the same data is the central challenge of multi-leader systems.

Let's now turn our attention to the core problem of multi-leader replication: resolving write conflicts. The following video provides an excellent overview of the multi-leader pattern and introduces the conflict resolution strategies we will explore in depth.

Designing a Conflict Resolution Plan

As the video highlighted, if you choose a multi-leader architecture, you must have a plan for handling concurrent writes. This plan can range from simply avoiding conflicts to using sophisticated data structures that merge them automatically.

Strategy 1: Conflict Avoidance

The simplest approach is to design the system such that conflicts rarely or never happen. As mentioned in the video at 00:02:54, this can be achieved by ensuring all writes for a specific record are routed to the same leader. For example, in a system with user data, you could hash the user_id to determine which leader is responsible for that user's writes. This is effectively a form of data partitioning on top of replication. While effective, it may not always be possible, especially if a leader fails and its traffic must be redirected.

Strategy 2: Last-Write-Wins (LWW)

When conflicts are unavoidable, Last-Write-Wins is a common resolution strategy due to its simplicity. Each write is assigned a timestamp, and when a conflict is detected, the write with the latest timestamp is kept, while the others are discarded.

While simple, LWW is fraught with peril, primarily due to two factors:

1. Data Loss: Concurrent updates are silently discarded, which can be catastrophic. Imagine two users editing a document; with LWW, one user's changes would be completely lost without notification.
2. Clock Skew: LWW relies on accurate, synchronized clocks across all nodes, which is a notoriously difficult problem in distributed systems. If one server's clock is even slightly ahead of another's, its writes will always "win," regardless of when they actually occurred in the physical world.

The following resource provides a deep dive into implementing LWW, including the critical nuances of timestamp generation and its suitability for different use cases.

Given your background in financial systems, the data loss inherent in LWW makes it an unsuitable choice for nearly all core transactional workflows. However, for ephemeral, non-critical data like session state or caching, its simplicity can be a reasonable trade-off.

Strategy 3: Preserving all Writes with Version Vectors

Instead of discarding data, a more robust approach is to detect concurrent writes and preserve all conflicting versions. This allows the application (or the user) to make an informed decision about how to merge them. The key technology for detecting concurrency is the version vector.

A version vector is a list of counters, with one counter per replica. When a replica processes a write, it increments its own counter. When a client writes data, it includes the version vector it last saw. A server can then determine if a new write is a direct descendant of a known state or if it is concurrent with another write.

The video we watched earlier contains one of the clearest explanations of version vectors available.

When a system detects concurrent writes (e.g., two different shopping carts with incomparable version vectors), it can store both versions as "siblings." On the next read, it returns both versions to the client, which is then responsible for merging them (e.g., by taking the union of the two shopping carts) and writing back a new, resolved version. This pattern, sometimes called "read-repair" or "on-read conflict resolution," prevents data loss but moves the complexity to the application layer.

Strategy 4: Automatic Resolution with CRDTs

The most advanced strategy is to use data structures that are mathematically designed to merge concurrently, known as Conflict-free Replicated Data Types (CRDTs). CRDTs define operations that are commutative, associative, and idempotent, which guarantees that even if operations arrive in different orders or are applied multiple times, all replicas will eventually converge to the same state without conflict.

This approach automates conflict resolution, making it an "on-write" strategy. Instead of flagging a conflict, the database itself merges the changes.

To make this concrete, let's explore a few practical CRDTs that you can use as design patterns.

The shopping cart example is particularly illuminating. It demonstrates how you can compose simpler CRDTs (PN-Counter, OR-Set principles) to build a complex, application-specific data type that handles concurrency correctly and automatically, a far more robust solution than LWW.

Conclusion

In this lesson, we dissected the critical decision of choosing a replication strategy. We saw how this choice directly impacts a system's scalability, latency, and consistency guarantees.

Key Takeaways:

• Leader-Follower replication is simple, provides strong consistency easily, and is ideal for read-heavy workloads. Its main drawback is a single point of failure and a bottleneck for writes.
• Multi-Leader replication offers high write throughput and low-latency writes in globally distributed systems, but at the cost of significant complexity in handling write conflicts.
• Conflict Resolution is the central design challenge for multi-leader systems. The main strategies are:
  • Last-Write-Wins (LWW): Simple to implement but causes silent data loss and is vulnerable to clock skew. Suitable only for non-critical, idempotent data.
  • Version Vectors: A mechanism to detect concurrent writes, allowing an application to preserve all conflicting versions and merge them intelligently. This prevents data loss but adds complexity to the client.
  • CRDTs: Data structures with mathematically-proven convergence properties. They provide automatic, conflict-free merging for specific data types like counters (PN-Counter) and sets (OR-Set), offering the most robust solution for collaborative applications.

In our next lesson, we will move from data replication to data partitioning. While replication is about making copies of data for availability, partitioning is about splitting data across multiple nodes for scalability. We will begin by exploring one of the most important algorithms for partitioning in dynamic environments: consistent hashing.