Momentum for Faster SGD Convergence

1. The Problem with SGD: Oscillations and Slow Progress

Recall the erratic path of SGD from our last lesson. When the loss landscape has narrow valleys (a common feature in neural networks), the gradient points steeply across the valley but shallowly along it. This causes the optimizer to oscillate back and forth between the valley walls, making very slow progress towards the true minimum.

This inefficient movement is sometimes called navigating a "pathological curvature."

To solve this, we need a way to dampen these oscillations while accelerating movement in the consistent, downhill direction.

2. The Intuition: A Ball Rolling Down a Hill

The core idea behind momentum is best understood through a physical analogy.

• Vanilla SGD is like a ball with zero mass or friction. At every point, it instantly moves in the direction of the steepest slope (the gradient). If the slope keeps changing, the ball's direction will change erratically with it.
• SGD with Momentum is like a heavy ball rolling down a hill. The ball doesn't just respond to the gradient at its current position; it also has velocity built up from past movements. This momentum helps it roll past small bumps, smooths its path through ravines, and makes it accelerate when the slope is consistently downhill.

Let's watch a short video that explains this intuition and the resulting behavior.

As the video illustrates, by averaging out the gradients over several steps, the updates that point in opposing directions (across the valley) cancel each other out, while the updates that point in a consistent direction (along the valley) add up.

The result is a much smoother and more direct path to the minimum.

3. The Mathematical Formulation

Now, let's translate this intuition into a concrete algorithm. In vanilla SGD, the weight update is:

where $\theta$ represents the parameters (weights and biases), $\alpha$ is the learning rate, and $\nabla J(\theta )$ is the gradient.

To add momentum, we introduce a velocity term, $v$, which accumulates a moving average of the gradients. The update becomes a two-step process:

1. Update the velocity: $$v_t=\beta v_{t-1}+(1-\beta )\nabla J({\theta }_{t-1})$$
2. Update the parameters: $${\theta }_t={\theta }_{t-1}-\alpha v_t$$

Let's break this down:

• $v_t$ is the velocity at the current time step $t$.
• $\beta$ is the momentum hyperparameter, a value typically between 0 and 1 (a common choice is 0.9). It controls how much of the previous velocity is retained.
• The velocity $v_t$ is an exponentially weighted moving average (EWMA) of the past gradients. When $\beta =0.9$, the velocity roughly averages over the last 10 gradients.

This video walks through the mathematical implementation step by step.

Why Does This Dampen Oscillations?

The magic lies in the exponentially weighted average. To understand exactly how it works, let's read a short, focused explanation.

In short, when gradients alternate (+1, -1, +1, -1...), the velocity term averages them out towards zero. When gradients are consistent (+1, +1, +1...), the velocity term accumulates, leading to larger and larger steps in that direction.

4. Implementing Momentum from Scratch

Now it's time to put theory into practice. Your experience with Python will make this straightforward. We'll implement Gradient Descent with Momentum for a simple one-dimensional function to see the acceleration in action.

The article "Gradient Descent With Momentum from Scratch" provides a very clear implementation.

Let's look at the core Python code from the article.

# A simplified version of the code from the article
import numpy as np
import matplotlib.pyplot as plt

# Objective function and its derivative
def objective(x):
    return x**2.0

def derivative(x):
    return x * 2.0

# Gradient descent algorithm with momentum
def gradient_descent_momentum(objective, derivative, bounds, n_iter, step_size, momentum):
    # Start at a random point
    solution = np.random.uniform(bounds[0,0], bounds[0,1])
    
    # Keep track of the velocity (called 'change' in the article)
    velocity = 0.0
    
    solutions = [solution]
    for i in range(n_iter):
        # Calculate gradient
        gradient = derivative(solution)
        
        # Calculate the new velocity
        new_velocity = step_size * gradient + momentum * velocity
        
        # Take a step using the velocity
        solution = solution - new_velocity
        
        # Store the new velocity for the next iteration
        velocity = new_velocity
        
        # Store solution for plotting
        solutions.append(solution)
        
    return solutions

# --- Run and Plot ---
# Hyperparameters
n_iter = 30
step_size = 0.1
momentum_beta = 0.5 # Try changing this to 0.0, 0.5, and 0.9

# Run the optimization
path_momentum = gradient_descent_momentum(objective, derivative, np.array([[-1.0, 1.0]]), n_iter, step_size, momentum_beta)
path_sgd = gradient_descent_momentum(objective, derivative, np.array([[-1.0, 1.0]]), n_iter, step_size, 0.0) # SGD is just momentum with beta=0

print(f"Steps with momentum (beta={momentum_beta}): {len(path_momentum)}")
print(f"Steps with SGD (beta=0.0): {len(path_sgd)}")

# You can run this code locally to see the plot and experiment!

This implementation demonstrates the core logic. Compared to standard SGD, we just need one extra variable (velocity) and one modified line in the update step. This small change, however, leads to significantly faster convergence, as you saw in the article's plots.

Conclusion

In this lesson, you've moved beyond simple gradient descent and learned a fundamental technique for accelerating neural network training. This is a crucial step from basic understanding to practical application.

Key Takeaways:

• Problem: Vanilla SGD is inefficient on complex loss surfaces, often oscillating in ravines and slowing down convergence.
• Intuition: Momentum acts like a heavy ball rolling downhill, using its accumulated velocity to smooth out its path and power through flat regions.
• Mechanism: Momentum maintains an exponentially weighted moving average of past gradients. This damps oscillations by canceling out opposing gradients and accelerates progress in consistent directions.
• Implementation: Adding momentum to SGD requires introducing a "velocity" term and a momentum hyperparameter, $\beta$, which is typically set to 0.9.

Preview of the Next Lesson:
Momentum successfully addresses the issue of noisy gradient directions. However, we are still using a single, fixed learning rate $\alpha$ for all parameters. What if some parameters need to take large steps while others need smaller, more careful steps?

In our next lesson, we will implement the Adam optimizer. Adam is a state-of-the-art optimizer that combines the idea of momentum with adaptive learning rates, creating an even more powerful and robust algorithm that is the default choice for training most deep learning models today.