Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 4

Batch vs. Stochastic Gradient Descent: A Comparative Study

Recommended resources

Hello! Welcome to the fourth lesson in our module on Neural Network Foundations.

In our previous lessons, we assembled the core components of a neural network. You've seen how to implement forward propagation, how backpropagation calculates the gradient of the loss function (), and how to choose appropriate activation functions and weight initialization schemes.

So, we know how to find the "direction of steepest descent" for our weights at any given moment. But a real-world dataset can have millions of data points. How do we use this massive dataset to guide our descent and train the model? This is the central question for today's lesson.

Our learning outcome is to apply batch gradient descent and stochastic gradient descent (SGD) and compare their training dynamics. We'll explore the two extreme approaches—using all the data at once versus using one example at a time—and discover the practical, middle-ground solution that powers modern deep learning.

1. Setting the Scene: The Loss Landscape

Before we discuss the strategies for descending, let's form a clear mental image of the terrain we're navigating. For any given neural network and dataset, we can imagine a vast, high-dimensional surface called the loss landscape.

  • The location on this landscape is determined by the specific values of all the weights and biases in our network (our parameters).
  • The elevation at any location is the value of the loss function for those parameters.

Our goal is simple: find the lowest point (or a very low point) in this landscape. The parameters at that location represent our trained model.

To build this intuition, let's watch a short, visually rich explanation.

Gradient Descent vs Evolution | How Neural Networks Learn

This video from Emergent Garden provides a fantastic visualization of the loss landscape, treating the training process as a search problem.

Please watch from 00:30 to 07:16. Focus on how the parameters of the network define a 'parameter space' and how the loss function creates a 'loss landscape' over this space. The key idea is that our optimization algorithm's job is to navigate this landscape.

2. The Two Extremes: Batch vs. Stochastic Gradient Descent

Now that we have the map (the loss landscape), we need a navigation strategy. We'll start by examining two opposing methods: Batch Gradient Descent and Stochastic Gradient Descent.

Batch Gradient Descent (BGD)

This is the most straightforward approach. To perform a single weight update, BGD considers the entire training dataset.

  1. Pass all training examples through the network.
  2. Calculate the loss for each example.
  3. Average the gradients of the loss from all examples.
  4. Update the network's weights using this single, averaged gradient.

This process is repeated for a number of epochs.

Stochastic Gradient Descent vs Batch Gradient Descent vs Mini Batch Gradient Descent |DL Tutorial 14

The video from codebasics clearly explains Batch Gradient Descent and its computational implications.

Watch the segment from 00:39 to 04:04. Pay attention to the diagram showing that all errors are summed up before any weight adjustment happens. Note the discussion on why this is computationally challenging for large datasets.

  • Pros: The gradient is a very accurate estimate of the "true" steepest direction for the entire dataset. This leads to a smooth, stable, and predictable descent down the loss landscape.
  • Cons: It is extremely slow and memory-intensive. For a dataset with millions of images, you would need to perform millions of forward and backward passes just to make one single update to your weights. It also has a higher chance of getting stuck in a sharp local minimum.

Stochastic Gradient Descent (SGD)

SGD takes the opposite approach. It updates the weights after seeing just one training example.

  1. Pick one training example at random.
  2. Pass it through the network and calculate the loss.
  3. Calculate the gradient based on this single example.
  4. Update the network's weights immediately.
  5. Repeat for all examples, one by one.

Stochastic Gradient Descent vs Batch Gradient Descent vs Mini Batch Gradient Descent |DL Tutorial 14

The same codebasics video now explains Stochastic Gradient Descent.

Now, watch from 04:04 to 06:01. The key contrast here is that weights are adjusted after every single sample. Notice the mention of the 'zigzag' convergence pattern.

  • Pros: Updates are very fast and require minimal memory. The noisy updates can help the optimizer "jump out" of sharp, undesirable local minima and find better, flatter ones, which often leads to better generalization.
  • Cons: The path of descent is extremely noisy and erratic. Each update is based on a very rough estimate of the true gradient, so the training process can look like a drunken walk down the hill rather than a smooth slide. Also, processing samples one by one is computationally inefficient as it doesn't take advantage of vectorized operations on modern hardware (like GPUs).

3. Comparing Training Dynamics

The difference between these two methods isn't just academic; it manifests in visually distinct training behaviors.

First, let's look at the path taken through the parameter space (our loss landscape).

Comparison of Gradient Descent Training Dynamics
A visual comparison of the paths taken by Batch (blue), Mini-batch (green), and Stochastic (red) gradient descent. Batch GD moves directly toward the minimum. SGD takes a noisy, erratic path. Mini-batch is a balance between the two.

Next, let's see how the loss function changes over time.

Comparison of Gradient Descent Methods
This chart shows the cost function over training iterations for the three methods. Batch GD is smooth. SGD is very noisy. Mini-Batch GD is a compromise, showing some noise but a clear downward trend.

These two images perfectly summarize the core trade-off: BGD is smooth and stable but slow, while SGD is fast and noisy.

4. The Practical Choice: Mini-Batch Gradient Descent

As you might have guessed, neither extreme is ideal. The solution used in virtually all modern deep learning is Mini-Batch Gradient Descent. It's a compromise that captures the best of both worlds.

Instead of the whole dataset or a single sample, we use a small "mini-batch" of data (e.g., 32, 64, or 256 samples) for each update.

  1. Create mini-batches by shuffling the dataset and splitting it into small chunks.
  2. For each mini-batch:
    a. Calculate the average gradient over all samples in that mini-batch.
    b. Update the weights using this mini-batch gradient.
  3. Once all mini-batches are processed, an epoch is complete. The process repeats.

This approach is so standard that in modern contexts, when people say "SGD," they are almost always referring to Mini-Batch SGD.

To solidify these concepts, let's read a concise summary that compares all three methods.

Stochastic Gradient Descent – Mini-batch and more

The blog post 'Stochastic Gradient Descent – Mini-batch and more' provides a clear textual summary with excellent comparison plots.

Read through the sections reviewing Batch GD, introducing SGD, comparing their performance, and finally introducing Mini-batch GD as the optimal trade-off. Pay close attention to the plots comparing the cost functions.

5. Applying the Concepts: Implementation from Scratch

Your background in Python and DL projects makes you well-equipped to appreciate how these theoretical differences translate into code. We'll now walk through a from-scratch implementation of BGD and SGD. This will make the concepts concrete and connect directly to the "apply" part of our learning outcome.

Stochastic Gradient Descent vs Batch Gradient Descent vs Mini Batch Gradient Descent |DL Tutorial 14

We'll return to the codebasics video, which provides an excellent, well-explained Python implementation of these algorithms for a simple regression problem.

This is the most hands-on part of the lesson. First, watch the implementation of Batch Gradient Descent (from 07:30 to 20:30). You don't need to memorize every line, but focus on the main training loop and how the gradient calculation and weight update happen outside the loop over the samples. Next, watch the implementation of Stochastic Gradient Descent (from 30:07 to 33:24). Notice how small the code change is: the weight update logic is simply moved inside the loop that iterates over the training samples.

Test your understanding!

Based on the code for Batch GD and Stochastic GD from the video, outline the main changes you would need to make to implement Mini-Batch Gradient Descent.

Show answer

To implement Mini-Batch Gradient Descent, you would need to make the following changes:

  1. Data Shuffling and Batching: Before the main epoch loop starts, you would shuffle the entire dataset (both X and y together to maintain correspondence). Then, you would split the shuffled dataset into smaller mini-batches of a specified batch_size.

  2. Outer Loop for Batches: Inside the main epoch loop, you would add a new loop that iterates through each mini-batch.

  3. Gradient Calculation per Batch: The gradient calculation (batch_gradient in the video's example) would now be performed on the current mini-batch (X_batch, y_batch), not the entire dataset X.

  4. Weight Update per Batch: The weight update step would occur inside this new mini-batch loop, meaning the weights are updated once per mini-batch, rather than once per epoch (like BGD) or once per sample (like SGD).

Conclusion

Today, you've learned about the fundamental strategies for driving neural network training. You now understand the critical trade-offs between computation, stability, and update frequency that every AI engineer must consider.

Key Takeaways:

  • Batch Gradient Descent (BGD): Uses the entire dataset for one update. It's stable and converges smoothly but is computationally infeasible for large datasets.
  • Stochastic Gradient Descent (SGD): Uses a single sample for one update. It's fast and the noise can help escape local minima, but its convergence is erratic.
  • Mini-Batch Gradient Descent: Uses a small batch of samples (e.g., 32-256) for each update. It's the standard in practice, balancing the stability of BGD with the efficiency and regularizing noise of SGD.
  • The choice of batch size is a key hyperparameter that affects both training speed and model performance.

Preview of the Next Lesson:
The noisy path of SGD and Mini-Batch GD can sometimes cause the optimizer to bounce around in a flat region of the loss landscape or oscillate back and forth in a narrow ravine, slowing down convergence. In our next lesson, we will implement momentum-based optimization, a simple but powerful technique that helps smooth out these updates and accelerate the journey to the minimum.