Synchronous vs. Asynchronous Replication: Durability-Performance Trade-offs

In our previous lesson, we tackled the challenge of scale by exploring how data partitioning and sharding allow us to distribute a massive dataset across a fleet of servers. This is fundamental to building systems that can handle large volumes of data and requests. However, distributing data introduces a new critical vulnerability: what happens if one of these servers, or shards, fails? Without a strategy to handle such failures, we risk both data loss and service unavailability.

This lesson addresses that very problem through replication. We will explore how creating and maintaining copies of our data provides the fault tolerance and durability that sharded systems need. Our focus will be on the two primary strategies for keeping these copies in sync: synchronous and asynchronous replication. You'll learn to analyze the crucial trade-offs between them—specifically, the balance between performance and data durability—a decision that lies at the heart of distributed system design. This understanding is vital for managing stateful services in AI infrastructure, from model metadata stores to the vector databases you're interested in.

The Need for Replication

A sharded architecture, by itself, is not fault-tolerant. If a shard containing a specific range of user data goes down, that data becomes completely inaccessible, and if the failure is permanent (e.g., disk corruption), the data is lost forever. Replication is the standard solution to this problem.

In a replication scheme, each piece of data is stored on multiple independent machines. If one machine fails, another can take its place, ensuring the system remains available and the data is safe. A common architecture for managing this is the leader-follower model (also known as master-slave).

In this model:

One replica is designated as the leader. It is responsible for handling all write requests.
All other replicas are followers. They receive a stream of data changes from the leader and apply them to their own local copy of the data.
Read requests can often be served by any of the replicas (leader or followers), which allows for scaling read throughput.

The central question in this architecture is: how and when does the leader propagate changes to the followers? The answer defines the system's trade-offs between performance and consistency.

This diagram illustrates the basic leader-follower setup and hints at the core problem we'll be discussing.

This diagram shows a write operation going to a leader, which then propagates the update to two follower replicas. A subsequent search query might hit a follower that hasn't received the update yet, illustrating the concept of replication lag.

Synchronous Replication: Prioritizing Durability

Synchronous replication enforces the strongest guarantees. When a client sends a write request to the leader, the following steps occur:

The leader writes the data to its own local storage.
The leader forwards the data change to its followers.
The leader waits until it receives a confirmation from at least one (and often a quorum of) followers that they have successfully saved the change.
Only after receiving these confirmations does the leader report success back to the client.

This process ensures that by the time the client receives a successful acknowledgment, the data is durably stored on multiple machines.

Trade-offs of Synchronous Replication:

Pro (High Durability & Strong Consistency): The primary advantage is durability. If the leader fails immediately after acknowledging a write, the data is not lost because it is guaranteed to exist on at least one follower. This enables a follower to be promoted to the new leader without any data loss, ensuring the system remains strongly consistent.
Con (Higher Latency & Reduced Availability): The write operation is inherently slower. The total latency now includes not just the write to the leader but also the network round-trip time to the followers. The system's write performance is gated by the response time of the follower(s) in the replication loop. If a required follower is slow or unavailable, the leader cannot process the write, potentially halting all incoming write requests.

The following video offers a concise explanation of the mechanics of synchronous replication.

Distributed Consensus and Data Replication strategies on the server

In this video, Gaurav Sen explains the fundamental concepts of data replication.

Watch the segment that describes synchronous copies. Pay attention to how the master (leader) sends the command to the slave (follower) and how this serial, ordered process ensures consistency.

Asynchronous Replication: Prioritizing Performance

Asynchronous replication prioritizes performance and responsiveness over immediate durability. Here, the sequence is much simpler:

The leader writes the data to its local storage.
The leader immediately sends a success acknowledgment back to the client.
In the background, the leader forwards the data change to its followers.

The key difference is that the leader does not wait for followers to confirm the write.

Trade-offs of Asynchronous Replication:

Pro (Low Latency & High Availability): Writes are extremely fast from the client's perspective. The leader can continue to accept writes even if followers are slow, disconnected, or have failed entirely. This makes the system more available for writes.
Con (Potential for Data Loss & Eventual Consistency): The main drawback is a window of potential data loss. If the leader fails after acknowledging a write but before the change has been sent to any followers, that write is lost forever. This model also introduces replication lag, the delay between a write succeeding on the leader and it being applied on a follower. This means that if a client reads from a follower shortly after a write, they may see stale data. The system is therefore eventually consistent.

This model is often perfectly acceptable for use cases where performance is paramount and the occasional loss of very recent data or temporary inconsistencies are tolerable.

Distributed Consensus and Data Replication strategies on the server

Now, let's watch the corresponding segment on asynchronous replication from the same video.

Focus on the part explaining asynchronous replication. The video clearly articulates the performance benefit (less load on the primary) and the consistency drawback (being out of sync).

A Core AI Infra Use Case: Checkpointing in LLM Training

This trade-off between synchronous and asynchronous operations is not just a database concept; it is a critical consideration in AI infrastructure, particularly in the training of large models.

Training a foundation model can take weeks or months on thousands of GPUs. Given the high probability of hardware or software failure in such a large-scale, long-running job, it's essential to periodically save the model's state (its weights, optimizer state, etc.). This saved state is called a checkpoint. If the training job fails, it can be resumed from the last successful checkpoint instead of starting from scratch.

Saving a checkpoint is fundamentally a large write operation. For a 100-billion parameter model, a single checkpoint can be hundreds of gigabytes or even terabytes. How this write is performed has a massive impact on training efficiency.

The following article from AWS provides an excellent, in-depth analysis of this exact problem.

Architecting scalable checkpoint storage for large-scale ML training on AWS | AWS Storage Blog

This article from the AWS Storage Blog dissects the performance implications of different checkpointing strategies in large-scale ML training. It provides a concrete, real-world example of the synchronous vs. asynchronous trade-off.

Please read the following sections: Synchronous checkpointing: Start reading from the heading Synchronous checkpointing. The key takeaway here is the quantification of wasted GPU time. The calculation shows how a "short" 3-minute pause for a synchronous checkpoint adds up to thousands of lost GPU-hours per day in a large cluster. This is the performance cost. Asynchronous checkpointing: Continue to the next section, Asynchronous checkpointing. This explains the alternative: the training process hands off the checkpoint data to a background process and immediately resumes computation. This directly maps to the asynchronous replication model we discussed.

As the article makes clear:

Synchronous checkpointing is analogous to synchronous replication. The training job (the "client") is blocked until the checkpoint write is confirmed as complete. This provides a strong durability guarantee—if the job resumes, it will be from a verifiably complete checkpoint. However, the performance cost is immense, as thousands of expensive accelerators sit idle.
Asynchronous checkpointing mirrors asynchronous replication. The training job offloads the model state to a buffer and immediately continues with the next training step. This maximizes performance (GPU utilization). The durability trade-off is that if a catastrophic failure occurs while the checkpoint is being written in the background, the progress made since the previous checkpoint is lost. In most training scenarios, this is an acceptable risk when weighed against the huge cost of idle compute time.

The article also introduces multi-level checkpointing, a hybrid strategy that further optimizes this trade-off. It involves frequent, fast asynchronous writes to local storage (low durability, fast recovery from minor errors) and less frequent, slower writes to highly durable object storage like Amazon S3. This layered approach is a sophisticated application of the principles we've discussed.

Choosing the Right Strategy

The decision between synchronous and asynchronous replication boils down to a single question: What is the business or application tolerance for data loss?

Feature	Synchronous Replication	Asynchronous Replication
Performance	Lower (higher write latency)	Higher (lower write latency)
Durability	High (no data loss on leader failure)	Lower (potential for data loss)
Consistency	Strong	Eventual
Availability	Lower (writes can block if a follower is down)	Higher (writes succeed even if followers are down)
Use Cases	Financial systems, user authentication, e-commerce orders—anywhere data integrity is non-negotiable.	Social media feeds, activity logs, analytics pipelines, ML model checkpointing—anywhere performance is key and minimal data loss is acceptable.

As an AI Infra Engineer, you will constantly face this decision. When designing a system to store model artifacts, you might choose synchronous replication for the model registry to ensure no model version is ever lost. Conversely, for a system that ingests real-time user interaction logs for model personalization, you would likely opt for an asynchronous pipeline to handle the high throughput, accepting the small risk of losing a few seconds of data in case of failure.

Conclusion

In this lesson, we've explored the critical role of replication in building robust, fault-tolerant distributed systems. You've learned how the leader-follower model provides a framework for keeping data copies synchronized and how the choice between synchronous and asynchronous replication forces a fundamental trade-off between durability and performance.

Here are the key takeaways:

Replication provides fault tolerance: It protects against data loss and unavailability when a server in a distributed system fails.
Synchronous replication prioritizes durability: It ensures data is copied to a follower before a write is acknowledged, at the cost of higher latency.
Asynchronous replication prioritizes performance: It acknowledges writes immediately and copies data in the background, offering low latency at the risk of losing recent data if the leader fails.
The trade-off is universal: We saw this principle applied directly to a core AI infrastructure problem—checkpointing for large model training—where asynchronous methods are favored to maximize the utilization of expensive GPU resources.

We've established that these replication strategies depend on the leader sending updates to followers. But what are the underlying mechanisms for this communication? In our next lesson, we will delve into the fundamental distributed communication patterns, such as Remote Procedure Calls (RPCs), message queues, and publish-subscribe systems, which form the nervous system of any distributed architecture.