Synchronous vs. Asynchronous SGD: Convergence and Hardware Tradeoffs

Introduction

Welcome to the next lesson in our exploration of large-scale training dynamics. In our previous discussions on data parallelism, particularly in the context of 3D parallelism, we implicitly assumed a synchronous update model. In this model, all workers (GPUs) process their mini-batch, compute gradients, and then participate in a collective AllReduce operation to average these gradients. Only after this global synchronization does every worker apply the exact same update to its copy of the model. This lock-step approach ensures consistency but isn't the only way to perform distributed training.

Today, we will dissect this assumption by comparing the synchronous Stochastic Gradient Descent (SGD) paradigm with its alternative, asynchronous SGD. This lesson will directly address the critical trade-offs between these two approaches, focusing on their impact on model convergence and hardware utilization. Understanding this trade-off is fundamental to designing efficient and robust distributed training systems.

The Fundamental Distinction: Blocking vs. Non-Blocking

At its heart, the difference between synchronous and asynchronous training mirrors the classic computer science concepts of blocking and non-blocking operations.

In the context of distributed training:

• Synchronous: All worker processes are blocked at a synchronization barrier, waiting for every other worker to finish computing gradients before the collective update can proceed.
• Asynchronous: Each worker process operates independently, sending its computed gradients to be applied without waiting for others.

This simple difference has profound consequences for both system efficiency and algorithmic behavior.

Synchronous SGD: Consistency and Stability

Synchronous SGD is the most direct extension of single-GPU training to a distributed setting. It operates in a highly coordinated, lock-step fashion.

For a deeper, more formal understanding of this process, please read the following section from the provided paper.

As you read, the core mechanism of synchronous SGD is a multi-step cycle:

1. Broadcast: All workers start with an identical copy of the model parameters ${\theta }_t$.
2. Local Computation: Each of the $N$ workers computes gradients $g_i({\theta }_t)$ on its own unique mini-batch of data.
3. Synchronization & Aggregation: The workers enter a synchronization barrier. The gradients are aggregated across all workers, typically using an AllReduce operation, and averaged.
4. Update: The global model parameters are updated using the averaged gradient: $${\theta }_{t+1}={\theta }_t-\eta \cdot \frac{1}{N}\sum _{i=1}^N{g}_i({\theta }_t)$$ This new parameter set ${\theta }_{t+1}$ is then broadcast back to all workers for the next iteration.

This is exactly what happens when using libraries like NCCL for AllReduce, which we discussed in the context of data parallelism. The key characteristic is that all workers apply the same update based on information from the entire global batch.

Trade-offs of Synchronous SGD

Advantages:

• Convergence Stability: Since gradients are always calculated on the current model state ${\theta }_t$, there is no "staleness". The algorithm behaves exactly like standard mini-batch SGD on a very large batch (N times the per-worker batch size). This makes convergence behavior much easier to analyze and reproduce.
• Algorithmic Correctness: It is a faithful implementation of the SGD algorithm, which is often critical for research and when precise replication of results is needed.

Disadvantages:

• Hardware Underutilization (The Straggler Problem): The system's throughput is dictated by its slowest worker. If one GPU is slow due to hardware variation, network congestion, or contention from other processes, all other GPUs will sit idle waiting for it at the synchronization barrier. This can dramatically reduce overall hardware utilization.
• High Communication Overhead: The global synchronization barrier can become a significant bottleneck, especially as the number of workers increases.

Asynchronous SGD: Maximizing Throughput

Asynchronous SGD relaxes the strict synchronization requirement to maximize hardware utilization. In this paradigm, workers don't wait for each other.

The dominant architecture for asynchronous training is the parameter server.

The workflow in a parameter server architecture is as follows:

1. Fetch Parameters: A worker fetches the latest model parameters from a central parameter server(s).
2. Local Computation: The worker computes gradients on its mini-batch using the parameters it just fetched.
3. Push Gradients: The worker sends its computed gradients back to the parameter server.
4. Server-Side Update: The parameter server updates its copy of the model using the gradients it just received.

Crucially, each worker does this independently and on its own schedule.

The Stale Gradient Problem

The independence of workers introduces a fundamental challenge: gradient staleness. By the time a worker finishes computing its gradients and sends them to the parameter server, other workers may have already updated the model. Therefore, the gradients being applied are "stale"—they were computed based on an older version of the model parameters.

This staleness introduces noise into the training process. While the stochastic nature of SGD provides some resilience, excessive staleness can hurt convergence.

Trade-offs of Asynchronous SGD

Advantages:

• High Hardware Utilization: Workers are never idle. As soon as a worker finishes its task, it fetches the latest parameters and starts a new one. This is especially beneficial in heterogeneous clusters where workers have varying performance.
• Fault Tolerance: If a worker fails, the training can continue with the remaining workers, making the system more robust for long-running jobs.

Disadvantages:

• Convergence Instability: Stale gradients can degrade statistical efficiency, potentially slowing convergence or causing the model to diverge, especially with a large number of workers. This often requires careful tuning of hyperparameters like the learning rate.
• Parameter Server Bottleneck: The centralized parameter server can become a communication bottleneck, as all workers must communicate with it.

A Head-to-Head Comparison: Theory and Practice

We now have a clear theoretical understanding of the trade-offs. Let's ground this with a direct comparison and empirical evidence.

First, let's watch a concise summary that contrasts the two approaches as implemented in TensorFlow.

The following table, adapted from the "Analysis of Distributed Training Systems" paper, provides an excellent summary of the core differences.

Empirical Evidence

The paper "Revisiting Distributed Synchronous SGD" from Google provides compelling experimental results. The authors found that while asynchronous training could be faster per step, the noise from stale gradients often led to lower final accuracy and required more training epochs to converge.

The key findings from that paper are crucial for system design:

1. Sync-SGD achieves better accuracy: By eliminating stale gradients, synchronous training consistently converged to a better final model.
2. The Straggler Problem is real: As seen in Figure 2, vanilla Sync-SGD has a noticeably slower step time than Async-SGD.
3. Backup Workers offer a solution: By launching a few extra workers and only waiting for a quorum (N out of N+k) to report, the impact of stragglers is largely eliminated. This makes Sync-SGD nearly as fast as Async-SGD per step, while retaining its superior convergence properties.

This has led to a general industry trend: most state-of-the-art training systems today favor synchronous training, leveraging highly optimized communication libraries (like NCCL) and system-level tricks (like backup workers) to make it efficient. The stability and predictability are often worth the engineering effort.

Conclusion

In this lesson, we have contrasted the two dominant paradigms for distributed SGD, analyzing their core mechanisms and trade-offs.

Key Takeaways:

• Synchronous SGD enforces a lock-step update, ensuring all workers apply the same gradients computed on the same model version. This guarantees convergence stability but can suffer from low hardware utilization due to the "straggler problem."
• Asynchronous SGD maximizes hardware utilization by allowing workers to operate independently without waiting. However, this introduces stale gradients, which can destabilize training and harm final model accuracy.
• In practice, the superior stability of synchronous training is highly valued. Modern systems often use Sync-SGD and mitigate its performance drawbacks with optimized collective communication algorithms (e.g., Ring-AllReduce) and clever system design (e.g., backup workers).

Preview of the Next Lesson:

We established that synchronous training is equivalent to mini-batch SGD with a very large effective batch size. While this aids convergence stability in one sense, simply increasing the batch size introduces its own set of optimization challenges. In our next lesson, we will analyze the impact of large-batch training on model convergence and explore the necessary adjustments, such as learning rate scaling and the use of adaptive optimizers like LARS and LAMB.