Identifying Dominant Patterns via SVD

1. Decomposing Transformations: The Intuition Behind SVD

At its heart, any matrix $M$ represents a linear transformation—it takes vectors from an input space and maps them to an output space. This transformation can involve rotation, scaling, and shearing in complex ways. SVD provides a way to break down any such transformation into three simpler, fundamental operations:

1. A rotation (and possibly a reflection).
2. A scaling along the new, rotated axes.
3. Another rotation.

The decomposition is written as:

Here, $U$ and $V$ are orthogonal matrices, which represent the rotations. Since you've studied linear algebra, you'll recall that orthogonal matrices preserve the lengths of vectors and the angles between them. $\Sigma$ is a diagonal matrix containing the singular values (${\sigma }_i$), which represent the scaling factors.

This decomposition is incredibly powerful because it applies to any rectangular matrix, making it a universal tool. The following video provides a fantastic visual explanation of this process.

2. Identifying Dominant Directions

The real magic of SVD for data analysis comes from the properties of the $\Sigma$ and $V$ matrices.

• The singular values in $\Sigma$ are, by convention, sorted in descending order: ${\sigma }_1\ge {\sigma }_2\ge \cdots \ge 0$.
• The columns of the matrix $V$, called the right singular vectors, form an orthogonal basis for the input space.
• The columns of the matrix $U$, called the left singular vectors, form an orthogonal basis for the output space.

The sorted singular values act as an importance ranking. The largest singular value, ${\sigma }_1$, corresponds to the most significant component of the transformation. The associated right singular vector, $v_1$ (the first column of $V$), represents the primary input direction. An input aligned with $v_1$ will be scaled by the largest factor, ${\sigma }_1$, and mapped to the output direction $u_1$.

This is the essence of identifying dominant directions. The first few right singular vectors ($v_1,v_2,\dots$) correspond to the directions along which the matrix $M$ has the most "action" or influence.

This idea is central to a technique you may have encountered in statistics called Principal Component Analysis (PCA). PCA aims to find the directions of maximum variance in a dataset. It turns out that the right singular vectors of a (centered) data matrix are precisely the principal components.

The following video provides a clear and practical overview of SVD in the context of a data matrix, explaining how the components reveal a hierarchy of importance.

3. SVD in Mechanistic Interpretability

Now, let's connect this back to our goal of understanding neural networks. When we pass a batch of different inputs through a network, we can collect the activation vectors from a specific layer. If we stack these vectors together, we form an activation matrix. This matrix is our data.

By applying SVD to this activation matrix, we can answer the question: "In which directions in the activation space does the network's state vary the most?"

The dominant singular vectors we find may correspond to semantically meaningful features. For example, in a language model, one direction might correspond to "is a plural noun," while another might relate to "is part of a person's name." Identifying these directions is a key activity in interpretability research.

4. Hands-On: Finding Directions in PyTorch

Let's put this into practice. We'll create a synthetic activation matrix with a known dominant direction, then use torch.linalg.svd to recover it. This exercise mirrors the process of analyzing real activations from a model.

First, we create a 2D dataset. Most of the data will be random noise, but we will add a strong signal along a specific direction, [0.5, -0.866].

import torch
import matplotlib.pyplot as plt




# 1. Create a synthetic activation matrix
num_samples = 500
d_activation = 2




# The direction we want to be dominant
signal_direction = torch.tensor([0.5, -0.866])




# Create a strong signal along this direction
signal = torch.randn(num_samples, 1) * signal_direction




# Create random noise
noise = torch.randn(num_samples, d_activation) * 0.3




# Our "activations" are the sum of the signal and noise
activations = signal + noise

print(f"Activation matrix shape: {activations.shape}")

For PCA, the data must be centered (have a mean of zero).

# 2. Center the data
mean = activations.mean(dim=0)
activations_centered = activations - mean

Now, we perform SVD. torch.linalg.svd returns three tensors: U, S, and Vh. S contains the singular values, and Vh is the transpose of the right singular vectors matrix, $V^T$.

# 3. Perform SVD
U, S, Vh = torch.linalg.svd(activations_centered)

print(f"Singular values (S): {S}")

You should see that the first singular value is much larger than the second, confirming that one direction is dominant. Let's extract this dominant direction. The right singular vectors are the rows of Vh (or columns of V). So, the first principal component is the first row of Vh.

# 4. Extract the dominant direction (first principal component)
pc1 = Vh[0, :]

print(f"First principal component: {pc1}")
print(f"Original signal direction: {signal_direction}")




# 5. Verify the result using cosine similarity
# We can use abs() because the direction could be flipped (v1 or -v1)
cosine_sim = torch.abs(torch.dot(pc1, signal_direction))
print(f"Cosine similarity with original direction: {cosine_sim:.4f}")

The cosine similarity should be very close to 1.0, showing that SVD successfully identified the hidden direction of maximum variance in our data. The principal component pc1 might be [-0.5, 0.866] instead of [0.5, -0.866]. This is perfectly fine, as a direction and its negative represent the same axis of variation.

This process—collecting activations, centering, and applying SVD to find the principal components—is a fundamental workflow you will use to probe the internal representations of neural networks.

Conclusion

In this lesson, we moved from manipulating tensors to interpreting them. We've seen that Singular Value Decomposition is more than just a mathematical curiosity; it's a powerful tool for finding structure in data.

Here are the key takeaways:

• SVD decomposes any matrix $M$ into $U\Sigma V^T$, representing a sequence of rotation, scaling, and another rotation.
• The singular values (${\sigma }_i$) in $\Sigma$ are ordered by magnitude and indicate the "importance" of different components of the transformation.
• The right singular vectors (columns of $V$, or rows of $V^T$) represent the principal directions of variation in the data. The first vector, $v_1$, corresponding to the largest singular value ${\sigma }_1$, is the most dominant direction.
• In interpretability, we apply SVD to activation matrices to discover the most salient feature directions in a model's representation space.

So far, we've discussed analyzing an activation matrix that we assumed we already have. But how do we get these activations from a model as it's running? In our next lesson, we will learn how to use PyTorch hooks, a mechanism that lets us tap into a model's forward pass and record (or even modify) its internal states.