Comparing Gradient Descent Optimizers

Hello! Welcome to the first lesson of our new module, "Techniques for Training Deep Neural Networks."

Introduction

In our last lesson, you successfully built and trained a complete multi-class classifier in PyTorch. For that project, we used the torch.optim.Adam optimizer without much discussion. You might have wondered: why "Adam"? What other choices are there, and does the choice of optimizer matter?

This lesson answers those exact questions. The optimizer is the engine of your training loop, dictating how your model's weights are updated to minimize loss. The right choice can dramatically speed up training and improve your final model's performance.

Our goal today is to explain and compare the most fundamental optimizers: Stochastic Gradient Descent (SGD), Momentum, RMSProp, and Adam. We will trace the evolution of these algorithms, understanding the problem each one was designed to solve.

By the end of this lesson, you'll have a strong intuition for how these optimizers work, their strengths and weaknesses, and when to use each one.

1. The Starting Point: Stochastic Gradient Descent (SGD)

As you know from your study of machine learning, Gradient Descent is the core idea behind optimization. We calculate the gradient of the loss function with respect to the model's parameters and take a small step in the opposite direction. Stochastic Gradient Descent (SGD) applies this concept to deep learning by computing the gradient not on the entire dataset, but on small batches of data. This makes the process computationally feasible and often leads to better solutions.

The update rule is simple and direct:

$$W_{new}=W_{old}-\eta \cdot \nabla J(W_{old})$$

where $W$ are the model weights, $\eta$ is the learning rate, and $\nabla J(W)$ is the gradient of the loss for the current mini-batch.

To build a solid foundation, let's watch a video that provides an excellent overview of the role of optimizers and recaps the mechanics of SGD.

While simple and effective, SGD has its challenges:

• Slow convergence: It can take a long time to navigate long, narrow valleys in the loss landscape, as it often zig-zags down the steep sides instead of heading directly toward the minimum.
• Getting stuck: It can get trapped in local minima or slow down significantly at saddle points, where the gradient is close to zero.

These limitations motivated the development of more advanced optimizers.

2. Gaining Speed: SGD with Momentum

Imagine a heavy ball rolling down a hill. It doesn't just follow the steepest path at each moment; it builds up momentum, allowing it to roll through small bumps and speed down gentle slopes. This is the core intuition behind the Momentum optimizer.

Momentum helps accelerate SGD in the relevant direction and dampens oscillations. It does this by adding a fraction of the previous update step to the current one. This "velocity" term helps the optimizer maintain a consistent direction, especially across many mini-batches where gradients point in a similar direction.

The update involves a new "velocity" term, $v$:

$$v_t=\beta v_{t-1}+(1-\beta )\nabla J(W_{t-1})$$ $$W_t=W_{t-1}-\eta v_t$$

Here, $\beta$ is the momentum coefficient (typically around 0.9), which controls how much of the past velocity is retained. Note: You may see different but equivalent formulations of this equation.

Let's see how this works visually.

Momentum is a powerful addition that solves many of SGD's speed issues. However, it still uses the same learning rate for all parameters. What if some parameters need to move faster than others?

3. Adaptive Learning Rates: RMSprop

Consider a loss landscape that is very steep in one dimension but very flat in another (like a long, narrow canyon). SGD and Momentum might oscillate wildly across the steep dimension while making slow progress along the flat one.

RMSprop (Root Mean Square Propagation) addresses this by adapting the learning rate for each parameter individually. The main idea is to perform smaller updates for parameters with large gradients and larger updates for parameters with small gradients.

It accomplishes this by keeping a moving average of the squared gradients for each parameter. The update rule then divides the learning rate by the square root of this average.

The update looks like this:

$$s_t=\gamma s_{t-1}+(1-\gamma )(\nabla J(W_{t-1}){)}^2$$ $$W_t=W_{t-1}-\frac{\eta }{\sqrt{s_t}+ϵ}\nabla J(W_{t-1})$$

Here, $s_t$ is the moving average of squared gradients, $\gamma$ is a decay factor (similar to $\beta$ in momentum), and $ϵ$ is a small value to prevent division by zero.

Let's see the intuition behind this.

RMSprop gives us parameter-specific learning rates. Momentum gives us acceleration. The natural next question is: can we have both?

4. The Best of Both Worlds: Adam

Adam (Adaptive Moment Estimation) is arguably the most popular and widely used optimizer today. It combines the ideas of Momentum and RMSprop.

1. It keeps an exponentially decaying average of past gradients (the first moment, like Momentum).
2. It keeps an exponentially decaying average of past squared gradients (the second moment, like RMSprop).

By using both, Adam can compute adaptive learning rates for each parameter while also benefiting from the acceleration provided by momentum. It's like having a smart, heavy ball that can adjust its speed and direction based on both its current momentum and the steepness of the terrain in every direction.

Adam's effectiveness and reliability make it an excellent default choice for most deep learning problems, which is why we used it in our previous lesson.

5. Comparison and Summary

We've covered a lot of ground, from the simple SGD to the sophisticated Adam. Seeing them side-by-side can help solidify your understanding.

This animated GIF provides a classic visualization of how different optimizers navigate the same loss surface.

To summarize the key characteristics, the following table is extremely helpful.

Practical Takeaways:

• Adam is a robust and excellent starting point for most problems. It often works well with its default hyperparameter settings.
• SGD with Momentum is still widely used. Some research suggests it can sometimes find "flatter" minima, which may lead to better generalization than Adam on certain tasks. It often requires more careful tuning of the learning rate.
• RMSprop can be effective, especially in reinforcement learning or for recurrent neural networks, but Adam has often superseded it in popularity.

Ultimately, the best optimizer can be problem-dependent, and experimentation is always a good idea.

Conclusion

In this lesson, we demystified the world of optimizers by tracing their evolution from SGD to Adam.

Key Takeaways:

• Stochastic Gradient Descent (SGD) is the foundational algorithm, updating weights based on mini-batch gradients.
• Momentum adds a "velocity" term to accelerate convergence and dampen oscillations, like a heavy ball rolling downhill.
• RMSprop introduces adaptive learning rates, using a moving average of squared gradients to scale updates for each parameter individually.
• Adam combines the benefits of both Momentum (first moment) and RMSprop (second moment) to create a powerful, adaptive, and generally effective optimizer.

Understanding these optimizers gives you a deeper insight into the training process and empowers you to make more informed decisions when building your models.

In our next lesson, we will explore another critical aspect of training deep networks: weight initialization. How we set the initial values of our model's weights can have a surprising impact on whether our model trains successfully at all.