Batch Gradient Descent for Function Minimization

1. The Gradient Descent Algorithm

The core idea of gradient descent is remarkably simple. We start at a random point on our loss surface and take a series of small steps downhill until we reach a minimum.

The "magic" is all in the update rule, which is applied at each iteration:

Let's break this down:

• current_parameters: This is our current position in the landscape (e.g., a vector [x, y]).
• ∇f(...): This is the gradient of the function f at our current position. It's the vector pointing uphill, which you learned to compute in the last lesson.
• η (eta): This is the learning rate, a small positive number (e.g., 0.01). It's a hyperparameter that controls how big of a step we take. Its value is critical:
  • If η is too small, the algorithm will take a very long time to converge.
  • If η is too large, it might overshoot the minimum and bounce around unstably, or even diverge entirely.

The following article provides a clear, concise summary of these core components.

2. Implementing Gradient Descent from Scratch

Given your preference for building things from the ground up, let's implement this. We'll start with a simple 1D function, f(x) = x², and then generalize. The video below provides a fantastic visual walkthrough of this exact process.

Inspired by the video and the ML Mastery article, we can write a clean, reusable function to perform gradient descent. Here is a from-scratch implementation that captures the core logic.

import numpy as np
import time
import matplotlib.pyplot as plt

# Let's define a simple function and its derivative
def objective_func(x):
    return x**2

def derivative_func(x):
    return 2*x

def gradient_descent_1d(objective, derivative, start_x, learning_rate, n_iter):
    """
    Performs gradient descent on a 1D function.
    
    Returns:
        tuple: A tuple containing the list of positions and the list of scores.
    """
    positions = [start_x]
    scores = [objective(start_x)]
    
    current_x = start_x
    
    print(f"Starting at x={current_x:.4f}, Score={scores[0]:.4f}")
    
    for i in range(n_iter):
        # Calculate the gradient (which is just the derivative in 1D)
        gradient = derivative(current_x)
        
        # Apply the update rule
        current_x = current_x - learning_rate * gradient
        
        # Store the new position and its score
        current_score = objective(current_x)
        positions.append(current_x)
        scores.append(current_score)
        
        if (i+1) % 10 == 0:
            print(f"> Iteration {i+1}: x={current_x:.4f}, Score={current_score:.4f}")
            
    return positions, scores

# --- Hyperparameters ---
LEARNING_RATE = 0.1
ITERATIONS = 50
STARTING_POINT = 8.0

# --- Run the algorithm ---
solutions, scores = gradient_descent_1d(objective_func, derivative_func, STARTING_POINT, LEARNING_RATE, ITERATIONS)

print(f"\nDone! Final solution: x={solutions[-1]:.4f}, Score={scores[-1]:.4f}")

3. What Does "Batch" Mean?

So far, we've minimized a simple mathematical function. In machine learning, the function we want to minimize is a loss function (like Mean Squared Error). A loss function's value depends on the entire training dataset.

Batch Gradient Descent gets its name because for every single update step, it calculates the gradient using the entire batch of training data.

When the loss is an average over all N data points (e.g., MSE), the gradient of the loss is also an average of the gradients contributed by each data point:

This makes each step accurate and reliable, but it can be computationally expensive if you have millions of data points.

The following resource explains this concept in the context of a real model. Don't worry about the specific linear regression formulas for now; focus on the high-level idea of using the entire dataset for each update.

4. Generalizing to Multiple Variables

Now, let's combine everything. We'll create a single batch_gradient_descent function that works for any number of variables. It will use the estimate_gradient function you built in the previous lesson. This is where your from-scratch toolkit starts to come together!

The logic is identical to the 1D case, but now we are operating on vectors using NumPy.

import numpy as np

# --- Paste your functions from the previous lesson here ---
def partial_difference_quotient(f, v, i, h):
    w = v.copy().astype(float) # Ensure w is float for addition
    w[i] = w[i] + h
    return (f(w) - f(v)) / h

def estimate_gradient(f, v, h=0.00001):
    v_float = np.array(v, dtype=float) # Ensure input vector is float
    return np.array([partial_difference_quotient(f, v_float, i, h) for i in range(len(v_float))])
# --------------------------------------------------------

def batch_gradient_descent(objective_func, start_point, learning_rate, n_iter):
    """
    Performs batch gradient descent on a multivariable function.
    
    Returns:
        np.array: The final optimized parameters.
    """
    current_point = np.array(start_point, dtype=float)
    
    for i in range(n_iter):
        # Calculate the gradient vector using our numerical method
        gradient = estimate_gradient(objective_func, current_point)
        
        # Apply the update rule to the vector
        current_point = current_point - learning_rate * gradient
        
        if (i+1) % 50 == 0:
            current_score = objective_func(current_point)
            print(f"> Iteration {i+1}: Point={current_point}, Score={current_score:.5f}")
            
    return current_point

# --- Example with a 2D function: f(x,y) = x^2 + y^2 ---
# The minimum is at (0, 0)
def bowl_function(v):
    return v[0]**2 + v[1]**2

# --- Hyperparameters ---
LEARNING_RATE_2D = 0.1
ITERATIONS_2D = 300
STARTING_POINT_2D = [8.0, 6.0]

# --- Run the algorithm ---
final_solution = batch_gradient_descent(bowl_function, STARTING_POINT_2D, LEARNING_RATE_2D, ITERATIONS_2D)

print(f"\nDone! Final solution: {final_solution}")
print(f"Final score: {bowl_function(final_solution):.5f}")

Notice how the core logic inside the loop is the same. The beauty of using NumPy is that the update rule current_point = current_point - learning_rate * gradient works seamlessly whether current_point is a single number or a vector with thousands of dimensions.

import numpy as np

# (Helper functions from the lesson should be here)
def partial_difference_quotient(f, v, i, h):
    w = v.copy().astype(float)
    w[i] = w[i] + h
    return (f(w) - f(v)) / h

def estimate_gradient(f, v, h=0.00001):
    v_float = np.array(v, dtype=float)
    return np.array([partial_difference_quotient(f, v_float, i, h) for i in range(len(v_float))])

def batch_gradient_descent(objective_func, start_point, learning_rate, n_iter):
    current_point = np.array(start_point, dtype=float)
    for i in range(n_iter):
        gradient = estimate_gradient(objective_func, current_point)
        current_point = current_point - learning_rate * gradient
        if (i+1) % 100 == 0:
            print(f"> Iter {i+1}: Point={current_point}, Score={objective_func(current_point):.5f}")
    return current_point

# 1. Define the function
def my_complex_func(v):
    x, y = v[0], v[1]
    return x**2 + 3*x*y + y**3

# 2. Set hyperparameters
START_POINT = [-2.0, 2.0]
# A smaller learning rate is needed for this more complex surface
LEARNING_RATE = 0.01 
ITERATIONS = 500

# 3. Run the optimization
final_solution = batch_gradient_descent(my_complex_func, START_POINT, LEARNING_RATE, ITERATIONS)

print(f"\nDone! Final solution found at: {final_solution}")
print(f"Final score: {my_complex_func(final_solution):.5f}")

# The algorithm should converge near the point [2.25, -1.5], where the score is -5.0625.
# This is the analytical minimum for this function.

Conclusion

Congratulations! You have now implemented the complete batch gradient descent algorithm from scratch. You've combined your gradient calculation tool with an iterative update rule to create a general-purpose optimizer that can find the minimum of complex, multivariable functions.

Key Takeaways:

• The gradient descent update rule is new_params = old_params - learning_rate * gradient.
• The learning rate is a critical hyperparameter that controls convergence speed and stability.
• Batch Gradient Descent computes the gradient using the entire dataset for each parameter update, making it stable but computationally intensive.
• Your numerical estimate_gradient function is a powerful, reusable component in your optimization toolkit.

Preview of the Next Lesson:
Batch gradient descent is great, but what if your dataset has 50 million images? Calculating the gradient across all of them for every single step would be incredibly slow. In the next lesson, we will implement Stochastic Gradient Descent (SGD), a faster variant that approximates the gradient using just one data point at a time, and see how its convergence behavior compares.