Implementing Gradient Descent in Python

1. Implementing Gradient Descent for a Simple Function

The best way to start coding gradient descent is to apply it to a problem where we already know the answer. Let's use the simple quadratic function $f(x)=x^2$. We know from basic calculus that its minimum is at $x=0$. Our goal is to write a Python script that finds this minimum iteratively.

To do this, we need three key components:

1. The function itself: f(x) = x**2
2. The derivative of the function: f'(x) = 2*x
3. The gradient descent update rule: x_new = x_old - learning_rate * f'(x_old)

The following video provides an excellent visual walkthrough of implementing this from scratch, using NumPy for calculations and Matplotlib for visualization.

As you saw in the video, the core of the implementation is a loop. Inside this loop, for a given number of iterations, we repeatedly:

1. Calculate the gradient (the derivative, 2*x) at the current position.
2. Use the update rule to compute the new position.
3. Update our current position to this new value.

This simple example shows the algorithm in its purest form.

2. The Learning Rate in Practice

In the last lesson, we discussed theoretically how a learning rate that is too high can cause the algorithm to overshoot the minimum, while one that is too low can lead to painfully slow convergence. Now you can see this happen live.

The next segment of the same video demonstrates exactly this.

This visualization makes it very clear why tuning the learning rate is so important. It's a hyperparameter that you, the machine learning practitioner, must choose carefully to ensure your model trains effectively.

3. Applying Gradient Descent to Linear Regression

Now that we've implemented gradient descent for a toy problem, let's use it for a real machine learning task: linear regression.

The goal of simple linear regression is to find the best-fit line through a set of data points. This line is defined by the equation $y=mx+b$, where:

• m is the slope
• b is the y-intercept

In this context, m and b are the parameters of our model. Our goal is to use gradient descent to find the optimal values of m and b that make our line fit the data as closely as possible.

To do this, we need a cost function to minimize. For linear regression, the standard cost function is the Mean Squared Error (MSE). It measures the average of the squared differences between the actual y values and the predicted y values (from our line).

The video below introduces this setup and walks through the Python implementation.

The process is very similar to our simple x^2 example, but now we have two parameters to update in each iteration. This requires calculating the partial derivatives of the MSE cost function with respect to m and b. These two partial derivatives form our gradient.

The update rules become:

Both updates are performed within the same loop, using the values of m and b from the previous iteration.

4. Hands-On Implementation: Your Turn!

Now it's time to run the code yourself. Below is a complete Python script that implements gradient descent for linear regression. It's based on the examples you've just seen.

I've included detailed comments to explain each part of the code.

import numpy as np
import matplotlib.pyplot as plt

# Step 1: Generate some synthetic data for linear regression
# We create data that loosely follows the line y = 2x + 3
X = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
y = np.array([5.1, 6.9, 9.2, 11.0, 13.1, 15.3, 17.2, 18.8, 21.3, 23.5])

# Step 2: Define the gradient descent function
def gradient_descent(X, y, learning_rate, iterations):
    # Initialize parameters m (slope) and b (intercept) to 0
    m_current = 0
    b_current = 0
    
    n = len(X)  # Number of data points
    
    cost_history = [] # To store the cost at each iteration for plotting
    
    # The main iterative loop
    for i in range(iterations):
        # Calculate predicted y values: y = mx + b
        y_predicted = m_current * X + b_current
        
        # Calculate the cost (Mean Squared Error)
        cost = (1/n) * sum([val**2 for val in (y - y_predicted)])
        cost_history.append(cost)
        
        # Calculate the partial derivatives (the gradients)
        # Derivative of cost w.r.t. m
        md = -(2/n) * sum(X * (y - y_predicted))
        # Derivative of cost w.r.t. b
        bd = -(2/n) * sum(y - y_predicted)
        
        # Update the parameters using the update rule
        m_current = m_current - learning_rate * md
        b_current = b_current - learning_rate * bd
        
        # Print progress (optional)
        if (i % 100 == 0):
            print(f"Iteration {i}: m = {m_current:.4f}, b = {b_current:.4f}, Cost = {cost:.4f}")
            
    return m_current, b_current, cost_history

# Step 3: Run the gradient descent algorithm
learning_rate = 0.01
iterations = 1000

m_final, b_final, cost_history = gradient_descent(X, y, learning_rate, iterations)

print("\n--- Final Results ---")
print(f"Final m: {m_final:.4f}")
print(f"Final b: {b_final:.4f}")

# Step 4: Visualize the results
plt.figure(figsize=(12, 5))

# Plot 1: The regression line
plt.subplot(1, 2, 1)
plt.scatter(X, y, color='blue', label='Actual Data')
plt.plot(X, m_final * X + b_final, color='red', label='Fitted Line')
plt.title('Linear Regression Fit')
plt.xlabel('X')
plt.ylabel('y')
plt.legend()

# Plot 2: Cost reduction over iterations
plt.subplot(1, 2, 2)
plt.plot(range(iterations), cost_history, color='green')
plt.title('Cost Function Convergence')
plt.xlabel('Iteration')
plt.ylabel('Cost (MSE)')

plt.tight_layout()
plt.show()

For a clean, text-based guide that recaps the code and concepts for linear regression with gradient descent, you can refer to the following article.

Conclusion

Congratulations! You have now implemented one of the most important algorithms in machine learning from scratch. By translating the mathematical formulas into Python code, you've gained a much deeper understanding of how a model can learn from data.

Key Takeaways:

• Implementing gradient descent in Python requires an iterative loop that updates model parameters.
• The core of the loop consists of calculating the predicted values, computing the cost, finding the gradient (partial derivatives), and applying the update rule.
• The same gradient descent logic applies whether you are minimizing a simple mathematical function or a complex cost function for a machine learning model like linear regression.
• Experimenting with hyperparameters like the learning rate and number of iterations is a key part of the machine learning process.

Preview of the Next Module:
This lesson concludes our module on Calculus for Machine Learning. You now have the mathematical tools to understand how models are optimized. Our next module, Probability and Statistics Essentials, will equip you with the tools to evaluate these models. You'll learn how to measure uncertainty, describe data, and formally test hypotheses about your model's performance—essential skills for any machine learning practitioner.