Gradient Descent for Single-Variable Functions in Python

1. From Math to Code: The Building Blocks

To implement gradient descent, we need to translate our mathematical components into Python. This involves three key parts:

1. The function we want to minimize (the objective function).
2. The function that gives us the slope (the derivative function).
3. The algorithm that performs the iterative updates.

We will use two essential Python libraries:

• NumPy: For efficient numerical operations.
• Matplotlib: For plotting our results and visualizing the process.

Let's start by looking at a clean, well-structured way to define our functions and the gradient descent algorithm itself.

From the article, we can establish our core Python structure. Let's say we want to find the minimum of the function $f(x)=x^2$.

First, we define the objective and derivative functions in Python:

# The function we want to minimize
def objective_function(x):
    return x**2




# The derivative of our function, which gives the gradient
def derivative_function(x):
    return 2 * x

Next, we create the main gradient descent function. This function will orchestrate the process by taking our objective and derivative functions as inputs and running the optimization loop.

# The gradient descent algorithm
def gradient_descent(derivative, start_point, learning_rate, n_iterations):



    # Our current position, starting at the specified point
    x = start_point
    



    # Run the process for a fixed number of iterations
    for i in range(n_iterations):



        # 1. Calculate the gradient at the current point
        gradient = derivative(x)
        



        # 2. Apply the update rule to take a step downhill
        x = x - learning_rate * gradient
        



        # (Optional) Print progress
        print(f"Iteration {i+1}: x = {x:.4f}")
        
    return x

This Python code is a direct translation of the algorithm you performed on paper. The for loop automates the repetitive calculations you did by hand.

2. Live Implementation and Visualization

Reading the code is one thing; seeing it built and run is another. The following video provides an excellent, dynamic walkthrough of implementing this exact logic from scratch, complete with a visualization that shows the point "rolling" down the curve to the minimum.

After watching the video, you should have a clear understanding of how these components come together in a script:

1. Import numpy and matplotlib.pyplot.
2. Define the objective and derivative functions.
3. Set hyperparameters: an initial starting position, the learning rate, and the number of iterations.
4. Create a loop that updates the position according to the gradient descent rule.
5. Plot the results to see the algorithm in action.

3. Experimenting with the Learning Rate

In the last lesson, we discussed how a poorly chosen learning rate ($\eta$) can cause the algorithm to converge too slowly, overshoot the minimum, or even diverge entirely. Now, with a working Python implementation, you can see these effects for yourself.

The next segment of the video you just watched demonstrates this perfectly.

This practical demonstration reinforces a critical lesson in machine learning: hyperparameters matter. The learning rate is perhaps the most important hyperparameter you will tune during model training. Getting it right is key to efficient and successful optimization.

Let's summarize the key scenarios:

• Small Learning Rate: The algorithm takes tiny steps. It will eventually reach the minimum, but it may require a very large number of iterations.
• Good Learning Rate: The algorithm converges to the minimum efficiently and smoothly.
• Large Learning Rate: The algorithm overshoots the minimum, oscillating back and forth across the valley. It may still converge, but it will be inefficient.
• Very Large Learning Rate: The steps are so large that each update moves the point further away from the minimum. The values explode, and the algorithm diverges.

Conclusion

Congratulations! You have successfully bridged the gap between mathematical theory and practical code. You took the gradient descent algorithm, which you previously solved by hand, and implemented it in Python—the standard language for AI development.

Key Takeaways:

• The gradient descent update rule can be directly translated into a Python loop.
• We can represent the objective function and its derivative as Python functions, which makes our code modular and easy to read.
• Libraries like NumPy and Matplotlib are fundamental tools for implementing and visualizing machine learning algorithms.
• Experimenting with the learning_rate in code provides a powerful, practical understanding of its critical role in optimization.

Preview of the Next Lesson:
You have now completed the "Calculus for Machine Learning" module, finishing with a concrete programming achievement. We will now shift gears to another essential pillar of AI: probability and statistics. In the next lesson, we will begin by learning how to apply the fundamental rules of probability (addition, multiplication, and complement). This will lay the groundwork for understanding uncertainty and making data-driven inferences, which are at the heart of machine learning.