Second-Order Taylor Approximation

Recommended resources

The freely available resources this lesson will direct you to.

Hello! Let's continue our exploration of multivariate calculus for machine learning.

Introduction

In our last lesson, we defined the Hessian matrix as the multivariable generalization of the second derivative for a scalar-valued function. We saw that it is formally the Jacobian of the gradient and that it geometrically describes the local curvature of a function's surface.

Today, we will put the gradient and the Hessian to work together. This lesson's goal is to construct the second-order Taylor approximation of a multivariable function. Just as a single-variable function can be locally approximated by a parabola, a multivariable function can be locally approximated by a quadratic surface (a "paraboloid"). This approximation is one of the most important tools in continuous optimization, forming the basis for powerful algorithms and the analysis of a model's loss landscape.

From Single to Multivariable Taylor Approximations

Recall the second-order Taylor approximation for a single-variable function $f (x)$ around a point $a$ :

f (x) \approx f (a) + f^{'} (a) (x - a) + \frac{1}{2} f^{''} (a) (x - a)^{2}

This formula approximates $f (x)$ near $a$ using a simple quadratic polynomial whose value, first derivative, and second derivative all match those of $f (x)$ at point $a$ .

Our goal is to find the equivalent formula for a multivariable function $f (x) : R^{n} \to R$ around a point $a \in R^{n}$ . The structure is analogous, but we replace the scalar derivatives with their multivariable counterparts:

The first derivative $f^{'} (a)$ becomes the gradient $\nabla f (a)$ .
The second derivative $f^{''} (a)$ becomes the Hessian matrix $H_{f} (a)$ .

The resulting approximation involves a constant term, a linear term based on the gradient, and a quadratic term based on the Hessian.

Building the Approximation: Intuition and Formulation

Let's begin by developing a visual and conceptual understanding of how these pieces fit together.

M4ML - Multivariate Calculus - 4.8 Multivariate Taylor

This video from Imperial College London provides an excellent visual demonstration of how a function's Taylor approximation is built up layer by layer, from a constant value (0th order), to a tilted plane (1st order), to a quadratic surface (2nd order).

Please watch from 01:29 to 06:25. As you watch, focus on two key aspects: How the zeroth, first, and second-order approximations visually relate to the original function's surface. How the familiar concepts of gradient (as part of the Jacobian) and Hessian are assembled into a compact and elegant matrix expression for the approximation.

As the video demonstrates, the second-order Taylor approximation of $f (x)$ around a point $a$ can be written very compactly using vector and matrix notation. If we define the displacement vector as $δ = x - a$ , the approximation is:

f (x) \approx f (a) + \nabla f (a)^{T} δ + \frac{1}{2} δ^{T} H_{f} (a) δ

This formula is fundamental. Let's break it down:

$f (a)$ : The zeroth-order term. This is simply the value of the function at the point of expansion. It provides the "height" of our approximating surface.
$\nabla f (a)^{T} δ$ : The first-order (linear) term. This is the dot product of the gradient at $a$ and the displacement vector $δ$ . It defines a tangent plane (or hyperplane in higher dimensions) that matches the function's slope at $a$ .
$\frac{1}{2} δ^{T} H_{f} (a) δ$ : The second-order (quadratic) term. This is a quadratic form involving the Hessian matrix. It adds the curvature to our approximation, bending the tangent plane into a paraboloid that matches the function's curvature at $a$ .

Second Order Approximation Graph — This image illustrates how a complex function surface (green) can be locally approximated by a simpler quadratic surface (yellow) using the second-order Taylor expansion. The formula shows the roles of the gradient (g) and Hessian (H).

For a more formal statement of this approximation, let's turn to the MIT lecture notes we used previously.

Second Derivatives, Bilinear Maps, and Hessian Matrices

The video gave us the intuition and the final formula. Now, let's look at a formal definition from the MIT lecture notes.

Please read the paragraph on page 88 that starts with 'Instead of thinking of H...'. This section directly presents the second-order Taylor approximation using the compact vector/matrix notation that is standard in machine learning. The term o(∥δx∥²) denotes higher-order terms that become negligible as δx approaches zero.

Derivation of the Multivariable Taylor Expansion

Given your interest in the mathematical underpinnings, it's valuable to see where this formula comes from. A classic and elegant way to derive the multivariable Taylor expansion is to reduce the problem to the single-variable case, for which we already know the answer.

The core idea is to consider the function's behavior along a straight line. Let's define a path from our point $a$ to the point $x$ . We can parameterize this line segment using a scalar $t \in [0, 1]$ :

p (t) = a + t (x - a) = a + t δ

Now, we can create a new single-variable function $g (t) = f (p (t))$ . Notice that $g (0) = f (a)$ and $g (1) = f (x)$ .

We can now write the single-variable Taylor expansion for $g (t)$ around $t = 0$ :

g (t) \approx g (0) + g^{'} (0) t + \frac{1}{2} g^{''} (0) t^{2}

Setting $t = 1$ gives us an approximation for $g (1) = f (x)$ :

f (x) \approx g (0) + g^{'} (0) + \frac{1}{2} g^{''} (0)

The final step is to calculate $g^{'} (0)$ and $g^{''} (0)$ using the multivariable chain rule, which will introduce the gradient and Hessian of $f$ .

The following video walks through this exact derivation.

Taylor Expansions for Functions of Several Variables

This video provides a formal proof for the multivariable Taylor expansion using the technique just described: reducing the problem to a single variable along a line and applying the chain rule.

Watch the section from 05:21 to 09:02. The presenter defines the function G(t), which is equivalent to our g(t), and then repeatedly applies the chain rule to find its first and second derivatives at t=0. This process beautifully reveals how the gradient and Hessian of the original function f emerge as the coefficients of the expansion.

Worked Example: Constructing the Approximation

Let's solidify this by constructing a second-order Taylor approximation for the function $f (x, y) = e^{x} cos (y)$ around the point $a = (0, 0)$ .

Step 1: Evaluate the function at $a$ .

f (0, 0) = e^{0} cos (0) = 1 \times 1 = 1

Step 2: Compute the gradient and evaluate it at $a$ .
The gradient is:

\nabla f (x, y) = [\frac{\partial f}{\partial x} \frac{\partial f}{\partial y}] = [e^{x} cos (y) - e^{x} sin (y)]

At $a = (0, 0)$ :

\nabla f (0, 0) = [e^{0} cos (0) - e^{0} sin (0)] = [10]

Step 3: Compute the Hessian and evaluate it at $a$ .
The Hessian is the matrix of second partial derivatives:

H_{f} (x, y) = [\frac{\partial ^{2} f}{\partial x ^{2}} \frac{\partial ^{2} f}{\partial y \partial x} \frac{\partial ^{2} f}{\partial x \partial y} \frac{\partial ^{2} f}{\partial y ^{2}}] = [e^{x} cos (y) - e^{x} sin (y) - e^{x} sin (y) - e^{x} cos (y)]

At $a = (0, 0)$ :

H_{f} (0, 0) = [e^{0} cos (0) - e^{0} sin (0) - e^{0} sin (0) - e^{0} cos (0)] = [10 0 - 1]

Step 4: Assemble the approximation.
The displacement vector is $δ = x - a = [x - 0 y - 0] = [x y]$ .
Plugging everything into the formula:

f (x, y) \approx f (0, 0) + \nabla f (0, 0)^{T} δ + \frac{1}{2} δ^{T} H_{f} (0, 0) δ

f (x, y) \approx 1 + [10] [x y] + \frac{1}{2} [x y] [10 0 - 1] [x y]

f (x, y) \approx 1 + x + \frac{1}{2} (x^{2} - y^{2})

This quadratic function $1 + x + \frac{1}{2} x^{2} - \frac{1}{2} y^{2}$ is the second-order Taylor approximation of $e^{x} cos (y)$ near the origin.

Test your understanding!

In your past projects, you've likely encountered objective functions. Let's consider a simple one: $f (x, y) = x^{2} y + y^{3}$ . Construct the second-order Taylor approximation for this function around the point $a = (1, 2)$ .

Show answer

1. Function value:
$f (1, 2) = (1)^{2} (2) + (2)^{3} = 2 + 8 = 10$

2. Gradient:
$\nabla f (x, y) = [2 x y x^{2} + 3 y^{2}]$
$\nabla f (1, 2) = [2 (1) (2) (1)^{2} + 3 (2)^{2}] = [413]$

3. Hessian:
$H_{f} (x, y) = [2 y 2 x 2 x 6 y]$
$H_{f} (1, 2) = [2 (2) 2 (1) 2 (1) 6 (2)] = [42212]$

4. Assembly:
The displacement vector is $δ = [x - 1 y - 2]$ .
The approximation is:
$f (x, y) \approx 10 + [413] [x - 1 y - 2] + \frac{1}{2} [x - 1 y - 2] [42212] [x - 1 y - 2]$
Expanding this gives the full quadratic polynomial, but this compact form is often more useful. The linear term is $4 (x - 1) + 13 (y - 2)$ and the quadratic term is $\frac{1}{2} (4 (x - 1)^{2} + 4 (x - 1) (y - 2) + 12 (y - 2)^{2})$ .

Conclusion

In this lesson, we successfully combined the gradient and the Hessian to build the second-order Taylor approximation, a local quadratic model of a multivariable function.

Key Takeaways:

The second-order Taylor approximation provides a quadratic model of a function $f (x)$ near a point $a$ .
It is constructed using the function's value, gradient, and Hessian, all evaluated at $a$ .
The standard formula in matrix notation is: $f (x) \approx f (a) + \nabla f (a)^{T} (x - a) + \frac{1}{2} (x - a)^{T} H_{f} (a) (x - a)$

Preview of the Next Lesson:
The power of this approximation truly shines when we analyze a function's critical points—points where the gradient is zero. At a critical point, the linear term in the Taylor expansion vanishes. The local behavior of the function (whether it's a minimum, maximum, or saddle point) is therefore determined entirely by the quadratic term, which is governed by the Hessian. In our next lesson, we will use this insight to classify critical points using the properties of the Hessian matrix, a cornerstone of optimization theory in machine learning.