Understanding Adam: An Adaptive Learning Rate Method

The Problem: One Size Doesn't Fit All

In complex models, especially deep neural networks, parameters can have vastly different characteristics. Some parameters might be very sensitive, where even a small change in their value causes a large change in the loss. Others might be "stubborn," requiring larger updates to make a difference. Using a single learning rate for all of them is inefficient.

• A learning rate that is too high for the sensitive parameters can cause them to overshoot the minimum and oscillate wildly.
• A learning rate that is too low for the stubborn parameters means they will learn agonizingly slowly.

We need a way to adapt the learning rate for each parameter automatically. Let's watch a video that illustrates this exact problem.

This is the core problem that adaptive learning rate methods are designed to solve.

The Solution: An Adaptive Learning Rate

The key idea behind adaptive learning rates is to scale each parameter's update based on its history of gradients. An early and influential method that did this was Adagrad (Adaptive Gradient Algorithm).

Its logic is brilliant:

• Keep a running sum of the squares of the gradients for each parameter.
• Divide the global learning rate by the square root of this sum.

This means parameters that have had large gradients in the past will have their learning rate automatically reduced. The video you just started continues this story perfectly.

A Fatal Flaw

Did you spot the issue? Adagrad's accumulator of squared gradients only ever grows. Over a long training run, this sum can become enormous, causing the effective learning rate for all parameters to shrink to almost zero. Learning effectively stops.

This "dying learning rate" problem needs a fix. Let's see how the video addresses this.

This idea of using an EWMA of squared gradients instead of a simple sum is the core of another popular optimizer called RMSprop (Root Mean Square Propagation). It gives us the adaptive rate without the dying learning rate problem.

Adam: Combining the Best of Both Worlds

Now we have all the pieces. Adam, which stands for Adaptive Moment Estimation, elegantly combines two ideas:

1. Momentum: It uses an EWMA of the gradients to track the average direction of the updates, helping to accelerate and smooth the path.
2. Adaptive Learning Rate (RMSprop-style): It uses an EWMA of the squared gradients to adapt the learning rate for each parameter based on its historical magnitude.

Adam maintains two "moment" vectors:

• First Moment ($m_t$): The EWMA of the gradients (like momentum's velocity). This is the "Moment Estimation" part.
• Second Moment ($v_t$): The EWMA of the squared gradients (like RMSprop). This is the "Adaptive" part.

Let's see how these components come together.

The Full Adam Algorithm

Let's solidify this by looking at the algorithm in text form. The concepts should now feel familiar. Adam takes the following steps at each iteration $t$:

1. Compute Gradient: $g_t={\nabla }_{\theta }J({\theta }_{t-1})$
2. Update First Moment (Momentum): $m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t$
3. Update Second Moment (Adaptive Rate): $v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)g_t^2$
4. Correct Bias in First Moment: ${\hat{m}}_t=\frac{m_t}{1-{\beta }_1^t}$
5. Correct Bias in Second Moment: ${\hat{v}}_t=\frac{v_t}{1-{\beta }_2^t}$
6. Update Parameters: ${\theta }_t={\theta }_{t-1}-\eta \frac{{\hat{m}}_t}{\sqrt{{\hat{v}}_t}+ϵ}$

The hyperparameters ${\beta }_1$ (typically 0.9), ${\beta }_2$ (typically 0.999), and $ϵ$ (a small number like $10^{-8}$) usually work well out-of-the-box, making Adam relatively easy to use.

To review these steps and the intuition behind them, please read the following article. It provides a concise summary of everything we've just covered.

Conclusion

Congratulations on completing the optimization module! You've progressed from the basic idea of gradient descent to one of the most sophisticated and widely used optimization algorithms in modern machine learning.

Key Takeaways:

• Core Idea: Adam combines the benefits of Momentum (using past gradients to determine direction) with an adaptive learning rate mechanism (using the magnitude of past gradients to scale step size for each parameter).
• First Moment ($m_t$): An exponentially weighted moving average of gradients, which provides momentum and smooths the optimization path.
• Second Moment ($v_t$): An exponentially weighted moving average of squared gradients, which provides a per-parameter scaling factor for the learning rate.
• Bias Correction: Adam includes a correction for the initial bias of its moment estimates, which are initialized at zero, ensuring more effective steps at the start of training.
• Result: A robust optimizer that converges quickly and works well across a wide range of problems with minimal hyperparameter tuning.

Preview of the Next Module:

We have now built a strong foundation in the core mechanics of how machine learning models learn—by minimizing a loss function using an optimizer. In the next module, "Linear Regression and Regularization," we will apply these concepts for the first time. We will formulate our first proper machine learning model, define its loss function (the sum of squared errors), and see how optimizers like gradient descent can be used to find the best-fitting line through a set of data points.