Numerical Differentiation in Python

Recommended resources

The freely available resources this lesson will direct you to.

Welcome to the next step in our journey to build a mathematical toolkit for AI. In our last lesson, we learned how to use Matplotlib to visualize functions and data. This skill of turning abstract numbers into insightful graphs is fundamental, and we are about to see just how powerful it can be.

This lesson marks our first formal entry into calculus, a cornerstone of the mathematics behind machine learning. We will demystify one of its central concepts: the derivative. You will learn what a derivative represents, why it's so important, and, following our "theory-to-code" philosophy, how to define a Python function and compute its numerical derivative from scratch. This will bridge the gap between abstract mathematical theory and practical, hands-on computation.

What is a Derivative?

At its heart, a derivative measures the rate of change. If you have a function that describes the position of a car over time, its derivative would describe the car's velocity. Geometrically, the derivative of a function at a specific point is the slope of the line tangent to the function's graph at that point.

This image illustrates the core concept of a derivative. It is the slope of the tangent line at a point on the curve, found by taking the limit of the slope of secant lines as the interval `Δx` shrinks to zero.

The formal definition of the derivative of a function $f (x)$ , denoted $f^{'} (x)$ , is expressed as a limit:

f^{'} (x) = Δ x \to 0 lim \frac{f ( x + Δ x ) - f ( x )}{Δ x}

This equation calculates the slope of a line between two points on the curve that are infinitesimally close to each other.

From Analytical to Numerical Differentiation

In calculus classes, you learn rules to find the analytical derivative. For example, the derivative of $f (x) = x^{2}$ is exactly $f^{'} (x) = 2 x$ . While we will revisit these rules later, in the world of AI and machine learning, we often don't have a neat, simple function. Instead, we might have a set of data points from an experiment, or a function so complex that finding its analytical derivative is impractical.

This is where numerical differentiation comes in. The idea is simple: we take the limit definition of the derivative and, instead of finding the limit as $Δ x$ (often written as h) goes to zero, we just use a very small, finite value for h. This gives us an approximation of the derivative.

The following video provides an excellent conceptual overview of this idea, introducing the "finite differences" method as a way to approximate derivatives.

Finite Differences Method for Differentiation | Numerical Computing with Python

This video from mechtutor com clearly explains the intuition behind numerical differentiation.

Please watch the first three sections: The introduction explaining the basic idea (introduction). The graphical explanation showing how a secant line approximates a tangent line (graphical explanation). The derivation, which connects the approximation back to the Taylor series and explains why some methods are more accurate than others (derivation).

The Finite Difference Formulas

As the video explained, there are several ways to approximate the derivative. The most common formulas are derived directly from the limit definition.

Finite Difference Approximating Derivatives

This document from the Python Numerical Methods course at Berkeley provides the formal mathematical definitions and derives them using Taylor Series, explaining their accuracy.

Please read the first part of the document, from the beginning down to the end of the section on the central difference formula. Focus on understanding the three main formulas presented: The forward difference, which looks ahead from the point x_j. The backward difference, which looks behind from the point x_j. The central difference, which uses points on both sides of x_j. Pay special attention to the discussion of the "order" of the approximation, like O(h) and O(h^2). This tells you how quickly the approximation error decreases as you make your step size h smaller. A method with O(h^2) accuracy is generally much better than one with O(h) accuracy.

For our purposes, the simplest and most intuitive method is the forward difference formula:

f^{'} (x) \approx \frac{f ( x + h ) - f ( x )}{h}

We will use this to build our first derivative-calculating function from scratch.

Implementing a Numerical Derivative Function

Let's translate the forward difference formula directly into Python code. Our goal is to create a function that can calculate the numerical derivative of any given function at a specific point.

Calculus Derivatives with python

This article from 4Geeks provides a very clear, "from scratch" implementation of a numerical derivative.

Please read the section titled Derivative as Function. The key part is the df(x) function, which directly implements the forward difference formula we just discussed. Notice how it defines a very small h and then applies the formula.

As you saw in the reading, we can define a function for our derivative:




# Let's define the function we want to differentiate
def my_function(x):
    return x**2




# Now, let's define a function to compute its numerical derivative
def numerical_derivative(f, x, h=0.00001):
    """
    Computes the numerical derivative of a function f at point x using the forward difference method.
    """
    return (f(x + h) - f(x)) / h




# Let's test it!
# The analytical derivative of x^2 is 2x. At x=3, the slope should be 2*3 = 6.
x_point = 3
derivative_at_x = numerical_derivative(my_function, x_point)

print(f"The analytical derivative at x={x_point} is: {2 * x_point}")
print(f"The numerical derivative at x={x_point} is: {derivative_at_x}")

If you run this code, you'll see the numerical result is extremely close to the true analytical value. By choosing a small h, we get a good approximation.

Differentiating an Entire Array

Calculating the derivative at a single point is useful, but we often want to find the derivative at every point along our function's domain. This is where your NumPy skills become invaluable. We can use the np.diff() function to efficiently calculate the differences between all adjacent elements in an array.

The following video provides a practical guide on how to build a robust numerical derivative function that operates on entire NumPy arrays.

How to: Numerical Derivative in Python

This video from HagesLab walks through the process of writing a Python function to compute the derivative of a set of data points.

Watch how the presenter builds the derivative function. Pay close attention to two key details: Using np.diff() (delta y / delta x): The core of the calculation is np.diff(y_list) / np.diff(x_list). This is a vectorized and efficient way to compute the forward difference for all points at once. Handling the x-values (x-prime list): This is a crucial and often overlooked point. Since np.diff() returns an array that is one element shorter, where do these new derivative values "live" on the x-axis? The video explains that each derivative value corresponds to the midpoint between the original x-values.

Let's solidify this with a complete example. We will take a function, generate a set of points, and then compute its numerical derivative across all those points. The resource from Berkeley we looked at earlier has a perfect code example for this.

Finite Difference Approximating Derivatives

Let's revisit the Python Numerical Methods document.

Find the first code example, which starts with the comment # step size. This code computes the numerical derivative of y = cos(x) and compares it to the exact analytical solution, y' = -sin(x). Study how np.diff(y)/h is used to calculate the forward difference and how x_diff = x[:-1] is used to adjust the x-array for plotting.

This "from-scratch" approach, using np.diff(), is exactly what we need. It's computationally efficient and gives us a full picture of how the function's rate of change behaves across its domain.

Conclusion

In this lesson, you have taken a significant step from pure mathematics into computational calculus. You have learned not just the theoretical definition of a derivative, but also how to approximate it and implement that approximation in Python from scratch.

Key Takeaways:

The derivative represents the instantaneous rate of change, or the slope of the tangent line to a function.
Numerical differentiation allows us to approximate derivatives using finite step sizes, which is essential when working with data or complex functions.
The forward difference formula, $f^{'} (x) \approx \frac{f ( x + h ) - f ( x )}{h}$ , is a direct and intuitive way to compute a numerical derivative.
By combining a Python function definition with NumPy's np.diff(), you can efficiently calculate the numerical derivative for an entire array of data points.

In our next lesson, we will combine the skills from this lesson and the previous one. You will learn to visualize a function and its derivative on the same plot, which will provide a powerful, intuitive understanding of the deep relationship between a function and its rate of change.