Batch vs. Mini-Batch vs. Stochastic Gradient Descent

The Problem with Batch Gradient Descent

Let's first solidify our understanding of the computational challenge posed by Batch Gradient Descent when dealing with large datasets.

As you can see, performing trillions of calculations for every single step of the optimization process is not feasible. This computational bottleneck is the primary motivation for the variants of gradient descent we will now discuss.

The Three Flavors of Gradient Descent

The core idea behind the different gradient descent variants is simple: instead of using all the data for each update, we can use a subset. The key difference lies in the size of that subset.

Let's watch a video that introduces and compares all three methods.

This gives us a great high-level overview. Now, let's dive deeper and formalize these concepts.

1. Batch Gradient Descent (The "Vanilla" Version)

• How it works: Computes the gradient of the loss function using the entire training dataset for each parameter update.
• Update Rule: $\theta =\theta -\alpha \cdot \nabla J_{\text{all}}(\theta )$
• Pros:
  • Stable Convergence: The descent path is smooth and direct because each step is an accurate estimate of the true gradient.
  • Guaranteed Convergence: For convex loss functions, it's guaranteed to converge to the global minimum.
• Cons:
  • Computationally Expensive: Prohibitively slow and memory-intensive for large datasets.
  • Inefficient: Makes very infrequent updates (one per epoch).

2. Stochastic Gradient Descent (SGD)

• How it works: Computes the gradient and updates the parameters using just one randomly selected training sample at a time. An epoch consists of iterating through all samples one by one.
• Update Rule: $\theta =\theta -\alpha \cdot \nabla J_i(\theta )$, for a random sample $i$.
• Pros:
  • Fast Iterations: Computation per update is extremely cheap.
  • Handles Large Datasets: Can process massive datasets that don't fit in memory.
  • Escapes Local Minima: The noisy updates can help the optimizer jump out of shallow local minima.
• Cons:
  • High Variance/Noisy Convergence: The loss function fluctuates wildly, and the path to the minimum is very erratic. It never truly settles at the minimum but oscillates around it.
  • Inefficient Vectorization: Processing one sample at a time doesn't take advantage of optimized matrix operations in libraries like NumPy or on GPUs.

3. Mini-Batch Gradient Descent (The Practical Choice)

• How it works: This is the compromise. It computes the gradient and updates parameters using a small batch of training samples (e.g., 32, 64, or 256) at a time.
• Update Rule: $\theta =\theta -\alpha \cdot \nabla J_{\text{batch}}(\theta )$, for a mini-batch of samples.
• Pros:
  • Balanced Approach: Combines the stability of Batch GD with the efficiency of SGD.
  • Efficient Computation: Allows for vectorized operations on mini-batches, making it highly efficient on modern hardware like GPUs.
  • Smoother Convergence: The loss fluctuates less than in SGD, leading to a more stable convergence.
• Cons:
  • New Hyperparameter: Introduces the "batch size" which needs to be tuned.

Visualizing the Convergence Paths

A picture is worth a thousand words when it comes to understanding the behavior of these algorithms.

The "zigzag" nature of SGD also shows up when you plot the cost over time. The codebasics video demonstrates this clearly. In SGD, the cost doesn't decrease smoothly; it jumps up and down while trending downwards overall. In contrast, the cost for Batch GD decreases in a much more predictable, smooth curve. Mini-batch would be somewhere in between.

A Comparative Summary

For a final, structured comparison, let's consult a reading that lays out these differences clearly.

Here is a summary of the key trade-offs:

Conclusion

In this lesson, we dissected the family of gradient descent algorithms. You learned that while Batch Gradient Descent is conceptually simple and stable, its computational cost makes it impractical for the large datasets common in modern machine learning.

Key Takeaways:

• Batch Gradient Descent: Uses the entire dataset for each update. It's stable but slow.
• Stochastic Gradient Descent (SGD): Uses a single sample per update. It's fast but has a noisy, erratic convergence.
• Mini-Batch Gradient Descent: Uses a small batch of samples per update. It strikes an optimal balance between computational efficiency (by leveraging vectorization on GPUs) and convergence stability. It is the de facto standard for training neural networks.
• The choice of algorithm involves a fundamental trade-off between the accuracy of the gradient estimate and the speed of computation.

Preview of the Next Lesson:

Regardless of which variant you choose, the learning rate remains a critical hyperparameter that determines the size of each step you take. A poorly chosen learning rate can ruin your training process, causing it to go too slowly, get stuck, or even diverge completely. In our next lesson, we will focus on how to "Diagnose and address issues with learning rates (too high or too low)."